K8s core arguments

core/config.go

@@ -129,6 +129,8 @@ func setDefaults() error {
 	viper.SetDefault("enableKafka", true)
 	viper.SetDefault("logAllIL", false)
 	viper.SetDefault("metricsEndpoint", "8088/ecsmetrics")
+	viper.SetDefault("kubeNamespace", "alice-tasks")
+	viper.SetDefault("kubeConfig", "/home/aliecs/.kube/config")
 	return nil
 }
 
@@ -200,6 +202,8 @@ func setFlags() error {
 	pflag.Bool("enableKafka", viper.GetBool("enableKafka"), "Turn on the kafka messaging")
 	pflag.Bool("logAllIL", viper.GetBool("logAllIL"), "Send all the logs into IL, including Debug and Trace messages")
 	pflag.String("metricsEndpoint", viper.GetString("metricsEndpoint"), "Http endpoint from which metrics can be scraped: [port/endpoint]")
+	pflag.String("kubeNamespace", viper.GetString("kubeNamespace"), "Kubernetes namespace in which ECS creates Task and Environment objects (empty disables Kubernetes support)")
+	pflag.String("kubeConfig", viper.GetString("kubeConfig"), "Path to the kubeconfig used to access the Kubernetes cluster (empty uses in-cluster config)")
 
 	pflag.Parse()
 	return viper.BindPFlags(pflag.CommandLine)

core/task/manager.go

@@ -152,6 +152,8 @@ func NewManager(shutdown func(), internalEventCh chan<- event.Event) (taskman *M
 	taskman.k8sEnvs = newK8sEnvRegistry()
 	if k8sClient, k8sErr := newK8sClientFromViper(); k8sErr != nil {
 		log.WithField("level", infologger.IL_Devel).WithError(k8sErr).Warn("K8s client init failed, K8s tasks disabled")
+	} else if k8sClient == nil {
+		log.Info("kubeNamespace not set, K8s tasks disabled")
 	} else {
 		log.Info("K8s client initiated")
 		taskman.k8sClient = k8sClient

core/task/managerk8s.go

@@ -90,7 +90,7 @@ func newK8sClientFromViper() (*k8sclient.Client, error) {
 	if namespace == "" {
 		return nil, nil
 	}
-	kubeconfig := viper.GetString("kubeconfig")
+	kubeconfig := viper.GetString("kubeConfig")
 	return k8sclient.New(kubeconfig, namespace)
 }
 

docs/kubernetes_ecs.md

@@ -47,9 +47,28 @@ The bridge works as follows:
 This design lets ECS reuse its existing environment lifecycle, configuration generation, and monitoring logic while outsourcing Pod scheduling and OCC communication to the in-cluster operators. The relevant code can be found in `task/manager.go` and `task/managerk8s.go`. 
 The manager.go implements the split between Mesos and K8s tasks while managerk8s.go implements actual creation, deployment, transitioning and updating ECS structure. 
 
-To deploy a Kubernetes task you can in principle reuse existing control-workflow task YAML manifests by changing the control mode to `kubernetes_direct` or `kubernetes_fairmq`. But in practice there can be issues
-with reusing existing manifests, for example bad quotation of arguments: Mesos runs tasks via shell, which interpreted and stripped quotation from commands. For this reason you can find ready-to-use manifests with all necessary changes
-inside `control-operator/ecs-manifests/control-workflows/*kube-direct*`.
+To deploy a Kubernetes task you can in principle reuse existing control-workflow task YAML manifests by changing the control mode to `kubernetes_direct` or `kubernetes_fairmq`. But in practice there can be issues reusing existing manifests, for example issues with argument quotation: Mesos runs tasks via shell, which interpreted and stripped quotation from commands. But K8s uses arguments as they are passed. For this reason you can find ready-to-use manifests with all necessary changes inside `control-operator/ecs-manifests/control-workflows/*kube-direct*` and you can just copy these files to be used instead of Workflow yaml templates.
+
+### Core configuration
+
+The Kubernetes client in ECS core is configured with the following parameters. Like any other core parameter they can be passed as CLI arguments
+| Parameter       | Default                     | Description |
+|-----------------|-----------------------------|-------------|
+| `kubeNamespace` | `alice-tasks`               | Namespace in which ECS creates custom Task and Environment objects. Setting it to an empty string disables Kubernetes support in core. |
+| `kubeConfig`    | `/home/aliecs/.kube/config` | Path to the kubeconfig used to access the cluster. If empty, core tries to use the in-cluster config, which works only when core itself runs inside the cluster. |
+
+If the Kubernetes client cannot be created (e.g. the kubeconfig file doesn't exist), core logs a warning `K8s client init failed, K8s tasks disabled`
+and continues running with Mesos only. Deploying a task with a Kubernetes control mode then fails.
+
+### TaskTemplates
+
+Since some tasks are deployed with known values ahead of time (image, env vars, ...), we added the `TaskTemplate` CR, which stores this data in the cluster so the user doesn't have to define everything every time. You can find all the `TaskTemplate`s in the `ecs-manifests/task-templates` folder. Currently we have templates for DPL, Readout, and StfBuilder/Sender. These templates must be applied to the cluster for the ECS bridge to work.
+
+### DPL and JIT
+
+DPL tasks work similarly to those mentioned before: they use `kubernetes_fairmq` in their workflow YAML manifests. However, there is a problem: JIT tasks' manifests are created the moment they are run. So for now you need to first run the workflow to generate those tasks, then change the control mode after they've been generated. This will be changed in the future.
+
+### Worflow templates
 
 ## Running tasks (`KubectlTask`)