diff --git a/.github/workflows/helm-template-check.yml b/.github/workflows/helm-template-check.yml index 04d4928..bc5b0b1 100644 --- a/.github/workflows/helm-template-check.yml +++ b/.github/workflows/helm-template-check.yml @@ -113,6 +113,27 @@ jobs: --set imageCredentials.username=test \ --set imageCredentials.password=test \ --set gateway=null + # Rollout strategy: the default, what --reuse-values from a chart + # before 0.2.4 leaves (no deploymentStrategy), and a Recreate override. + ai_strategy() { + helm template test ./charts/ai \ + --set-file ctaiPropertiesFile=./charts/ai/ctai-configuration.properties.template \ + --set service.fqdn=https://example.codetogether.io \ + --set imageCredentials.username=test \ + --set imageCredentials.password=test \ + --show-only templates/deployment.yaml "$@" \ + | sed -n '/^ strategy:$/,/^ selector:$/p' + } + default_strategy="$(ai_strategy)" + grep -q 'type: RollingUpdate' <<<"$default_strategy" + grep -q 'maxSurge: 0' <<<"$default_strategy" + grep -q 'maxUnavailable: 1' <<<"$default_strategy" + [ "$(ai_strategy --set deploymentStrategy=null)" = "$default_strategy" ] + recreate_strategy="$(ai_strategy --set deploymentStrategy.type=Recreate --set deploymentStrategy.rollingUpdate=null)" + grep -q 'type: Recreate' <<<"$recreate_strategy" + if grep -q 'rollingUpdate' <<<"$recreate_strategy"; then + echo "Recreate rendered rollingUpdate" >&2; exit 1 + fi - name: Validate Live Chart run: | diff --git a/charts/ai/Chart.yaml b/charts/ai/Chart.yaml index aa3b426..12db068 100644 --- a/charts/ai/Chart.yaml +++ b/charts/ai/Chart.yaml @@ -3,7 +3,7 @@ name: codetogether-ai description: CodeTogether AI on-premises Kubernetes deployment type: application -version: 0.2.3 +version: 0.2.4 appVersion: "1.2.8616" icon: https://www.codetogether.com/wp-content/uploads/2020/02/codetogether-circle-128.png diff --git a/charts/ai/README.md b/charts/ai/README.md index 92685be..1e1ebed 100644 --- a/charts/ai/README.md +++ b/charts/ai/README.md @@ -374,6 +374,58 @@ If login fails after returning from Google or another identity provider with and that the registered OAuth redirect URI exactly matches `{service_fqdn}/api/v1/auth/sso/provider/callback`. +### What the container image contains + +The image is built on [Wolfi](https://github.com/wolfi-dev), a minimal Linux +distribution, and runs as the non-root user `ctai` (uid and gid `2200`). + +- Shell and tools: `bash`, BusyBox `sh` (`/bin/sh`), `curl`, `ps`, `top`, + `ping`, `netstat`, `vi`, `less`, and the GNU core utilities. `ss`, `ip`, + `openssl` and `dig` are not included. +- Java: OpenJDK 25 with its tools (`jcmd`, `jstack`, `jmap`, `jfr`, + `keytool`). `JAVA_HOME` is `/usr/lib/jvm/default-jvm`, and the JDK's trust + store is `/etc/ssl/certs/java/cacerts`. +- Packages: the package manager is `apk`. It needs root, so it cannot install + anything in the running container. An image built `FROM` this one installs + packages with `apk add` as root before switching back to `USER 2200`. + +### Inspect a running pod + +`kubectl exec` is enough for the tools above: + +```bash +kubectl exec -it -n codetogether-ai deploy/codetogether-ai -c codetogether-ai -- bash +``` + +For a tool the image does not have, attach an ephemeral debug container. It +shares the application's process namespace and network, so it sees the Java +process, its files under `/proc//root` and `localhost`, and nothing +restarts. Use kubectl 1.32 or later, and any image your cluster can pull: + +```bash +POD=$(kubectl get pod -n codetogether-ai -l app.kubernetes.io/name=codetogether-ai -o name | head -1) +kubectl debug -it -n codetogether-ai "$POD" \ + --image= \ + --target=codetogether-ai --profile=general -c debug -- sh +``` + +If the namespace enforces the Pod Security `restricted` profile, which does not +allow a root debug container, use the CodeTogether AI image itself as the debug +image and run it as the application's user. The image names its user (`ctai`), +so the numeric uid has to be given: + +```bash +echo '{"securityContext":{"runAsUser":2200,"runAsGroup":2200}}' > uid2200.json +IMAGE=$(kubectl get "$POD" -n codetogether-ai -o jsonpath='{.spec.containers[0].image}') +kubectl debug -it -n codetogether-ai "$POD" --image="$IMAGE" \ + --target=codetogether-ai --profile=restricted --custom=uid2200.json \ + -c debug-app -- bash +``` + +An ephemeral container stays in the pod's spec until the pod is replaced, and +each one needs a new name (`-c`). The process environment and the mounted +properties file contain secrets: do not paste them into tickets or chat. + ## Upgrade ```bash @@ -399,6 +451,51 @@ helm upgrade codetogether-ai codetogether/codetogether-ai \ The Deployment rolls when the chart-managed properties Secret changes. +### Rollout strategy + +From chart 0.2.4 the Deployment stops the old pod before it starts the new one +(`deploymentStrategy`: `RollingUpdate` with `maxSurge: 0`, `maxUnavailable: 1`). +On a single replica an upgrade therefore has a short outage while the new pod +starts. Earlier charts used the Kubernetes default, which starts the new pod +first. + +The pods of one release share an embedded Hazelcast cluster, and a pod only +joins a cluster that runs its own Hazelcast minor version. When a release +changes that version (the release notes say so), any old pod still running +makes the new pod fail to join: its container exits and restarts in a loop, and +the rollout never completes while the old pods keep serving. `helm upgrade` +without `--wait` still prints `deployed`; `kubectl rollout status` shows the +stall. + +- **One replica:** the default strategy handles it, as long as the release was + upgraded to chart 0.2.4 or later. Upgrade the chart and the image in the same + `helm upgrade`, as above. +- **More than one replica:** upgrade across a Hazelcast version with + `Recreate`, which stops every old pod first (a short outage): + + ```bash + helm upgrade codetogether-ai codetogether/codetogether-ai \ + --namespace codetogether-ai \ + --reset-then-reuse-values \ + --set deploymentStrategy.type=Recreate \ + --set deploymentStrategy.rollingUpdate=null + ``` + + `rollingUpdate=null` is required: Helm merges overrides into the existing + values, and Kubernetes rejects `Recreate` while `rollingUpdate` is set. The + same command completes a rollout that has already stalled. + `--reset-then-reuse-values` keeps `Recreate` for later upgrades; to return to + the default on a later upgrade, add (Helm 3.10 or later) + `--set-json 'deploymentStrategy={"type":"RollingUpdate","rollingUpdate":{"maxSurge":0,"maxUnavailable":1}}'`. + +If a rollout has already stalled on a chart older than 0.2.4, switch the +Deployment to `Recreate` directly: + +```bash +kubectl -n codetogether-ai patch deployment codetogether-ai \ + -p '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}' +``` + ## Uninstall ```bash diff --git a/charts/ai/templates/deployment.yaml b/charts/ai/templates/deployment.yaml index dd87f27..464e5a6 100644 --- a/charts/ai/templates/deployment.yaml +++ b/charts/ai/templates/deployment.yaml @@ -7,6 +7,10 @@ metadata: {{- include "ctai.labels" . | nindent 4 }} spec: replicas: {{ .Values.replicaCount }} + {{- /* Read through default: --reuse-values from a chart before 0.2.4 has no deploymentStrategy. */}} + {{- $strategy := .Values.deploymentStrategy | default (dict "type" "RollingUpdate" "rollingUpdate" (dict "maxSurge" 0 "maxUnavailable" 1)) }} + strategy: + {{- toYaml $strategy | nindent 4 }} selector: matchLabels: {{- include "ctai.selectorLabels" . | nindent 6 }} diff --git a/charts/ai/values.yaml b/charts/ai/values.yaml index 3500d75..dbca263 100644 --- a/charts/ai/values.yaml +++ b/charts/ai/values.yaml @@ -202,6 +202,21 @@ tolerations: [] affinity: {} replicaCount: 1 +# Rollout strategy. The default stops the old pod before it starts the new one +# (a short outage on a single replica), instead of the Kubernetes default +# (maxSurge 25%, which rounds up to one extra pod). Pods of one release share an +# embedded Hazelcast cluster, and a pod only joins a cluster of its own +# Hazelcast minor version: when a release changes that version, an old pod still +# running makes the new pod fail to join and restart in a loop, so a surge +# rollout never completes. With more than one replica, upgrade across a +# Hazelcast version with Recreate (see "Upgrade" in README.md): +# --set deploymentStrategy.type=Recreate --set deploymentStrategy.rollingUpdate=null +deploymentStrategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 0 + maxUnavailable: 1 + extraEnvVars: [] extraVolumes: [] extraVolumeMounts: []