From 420fa056c4b9b0166ced6b9d1270bef8e7912632 Mon Sep 17 00:00:00 2001 From: Daniel Garcia Rico Date: Mon, 9 Nov 2026 21:02:35 +0000 Subject: [PATCH 1/2] AITRAX-2921: codetogether-ai chart 0.2.4 - deploymentStrategy, old pod stops first The chart rendered no strategy, so Kubernetes' default (maxSurge 25%, which rounds up to one extra pod) started the new pod while the old one ran. ct-ai's next release moves Hazelcast from 5.5 to 5.7, and a pod only joins a cluster of its own minor version: on the dev cluster, upgrading a single replica with chart 0.2.3 left the new pod crash-looping while the old one kept serving, and the rollout never completed. deploymentStrategy now defaults to RollingUpdate with maxSurge 0 and maxUnavailable 1, as in ct-ai's internal chart, and can be set to Recreate for multi-replica installs. The template reads it through default, so an upgrade with --reuse-values from an earlier chart gets the same strategy. README documents the upgrade, and CI checks the three renders. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/helm-template-check.yml | 21 +++++++++++ charts/ai/Chart.yaml | 2 +- charts/ai/README.md | 45 +++++++++++++++++++++++ charts/ai/templates/deployment.yaml | 4 ++ charts/ai/values.yaml | 15 ++++++++ 5 files changed, 86 insertions(+), 1 deletion(-) diff --git a/.github/workflows/helm-template-check.yml b/.github/workflows/helm-template-check.yml index 04d4928..bc5b0b1 100644 --- a/.github/workflows/helm-template-check.yml +++ b/.github/workflows/helm-template-check.yml @@ -113,6 +113,27 @@ jobs: --set imageCredentials.username=test \ --set imageCredentials.password=test \ --set gateway=null + # Rollout strategy: the default, what --reuse-values from a chart + # before 0.2.4 leaves (no deploymentStrategy), and a Recreate override. + ai_strategy() { + helm template test ./charts/ai \ + --set-file ctaiPropertiesFile=./charts/ai/ctai-configuration.properties.template \ + --set service.fqdn=https://example.codetogether.io \ + --set imageCredentials.username=test \ + --set imageCredentials.password=test \ + --show-only templates/deployment.yaml "$@" \ + | sed -n '/^ strategy:$/,/^ selector:$/p' + } + default_strategy="$(ai_strategy)" + grep -q 'type: RollingUpdate' <<<"$default_strategy" + grep -q 'maxSurge: 0' <<<"$default_strategy" + grep -q 'maxUnavailable: 1' <<<"$default_strategy" + [ "$(ai_strategy --set deploymentStrategy=null)" = "$default_strategy" ] + recreate_strategy="$(ai_strategy --set deploymentStrategy.type=Recreate --set deploymentStrategy.rollingUpdate=null)" + grep -q 'type: Recreate' <<<"$recreate_strategy" + if grep -q 'rollingUpdate' <<<"$recreate_strategy"; then + echo "Recreate rendered rollingUpdate" >&2; exit 1 + fi - name: Validate Live Chart run: | diff --git a/charts/ai/Chart.yaml b/charts/ai/Chart.yaml index aa3b426..12db068 100644 --- a/charts/ai/Chart.yaml +++ b/charts/ai/Chart.yaml @@ -3,7 +3,7 @@ name: codetogether-ai description: CodeTogether AI on-premises Kubernetes deployment type: application -version: 0.2.3 +version: 0.2.4 appVersion: "1.2.8616" icon: https://www.codetogether.com/wp-content/uploads/2020/02/codetogether-circle-128.png diff --git a/charts/ai/README.md b/charts/ai/README.md index 92685be..f2eefe1 100644 --- a/charts/ai/README.md +++ b/charts/ai/README.md @@ -399,6 +399,51 @@ helm upgrade codetogether-ai codetogether/codetogether-ai \ The Deployment rolls when the chart-managed properties Secret changes. +### Rollout strategy + +From chart 0.2.4 the Deployment stops the old pod before it starts the new one +(`deploymentStrategy`: `RollingUpdate` with `maxSurge: 0`, `maxUnavailable: 1`). +On a single replica an upgrade therefore has a short outage while the new pod +starts. Earlier charts used the Kubernetes default, which starts the new pod +first. + +The pods of one release share an embedded Hazelcast cluster, and a pod only +joins a cluster that runs its own Hazelcast minor version. When a release +changes that version (the release notes say so), any old pod still running +makes the new pod fail to join: its container exits and restarts in a loop, and +the rollout never completes while the old pods keep serving. `helm upgrade` +without `--wait` still prints `deployed`; `kubectl rollout status` shows the +stall. + +- **One replica:** the default strategy handles it, as long as the release was + upgraded to chart 0.2.4 or later. Upgrade the chart and the image in the same + `helm upgrade`, as above. +- **More than one replica:** upgrade across a Hazelcast version with + `Recreate`, which stops every old pod first (a short outage): + + ```bash + helm upgrade codetogether-ai codetogether/codetogether-ai \ + --namespace codetogether-ai \ + --reset-then-reuse-values \ + --set deploymentStrategy.type=Recreate \ + --set deploymentStrategy.rollingUpdate=null + ``` + + `rollingUpdate=null` is required: Helm merges overrides into the existing + values, and Kubernetes rejects `Recreate` while `rollingUpdate` is set. The + same command completes a rollout that has already stalled. + `--reset-then-reuse-values` keeps `Recreate` for later upgrades; to return to + the default on a later upgrade, add (Helm 3.10 or later) + `--set-json 'deploymentStrategy={"type":"RollingUpdate","rollingUpdate":{"maxSurge":0,"maxUnavailable":1}}'`. + +If a rollout has already stalled on a chart older than 0.2.4, switch the +Deployment to `Recreate` directly: + +```bash +kubectl -n codetogether-ai patch deployment codetogether-ai \ + -p '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}' +``` + ## Uninstall ```bash diff --git a/charts/ai/templates/deployment.yaml b/charts/ai/templates/deployment.yaml index dd87f27..464e5a6 100644 --- a/charts/ai/templates/deployment.yaml +++ b/charts/ai/templates/deployment.yaml @@ -7,6 +7,10 @@ metadata: {{- include "ctai.labels" . | nindent 4 }} spec: replicas: {{ .Values.replicaCount }} + {{- /* Read through default: --reuse-values from a chart before 0.2.4 has no deploymentStrategy. */}} + {{- $strategy := .Values.deploymentStrategy | default (dict "type" "RollingUpdate" "rollingUpdate" (dict "maxSurge" 0 "maxUnavailable" 1)) }} + strategy: + {{- toYaml $strategy | nindent 4 }} selector: matchLabels: {{- include "ctai.selectorLabels" . | nindent 6 }} diff --git a/charts/ai/values.yaml b/charts/ai/values.yaml index 3500d75..dbca263 100644 --- a/charts/ai/values.yaml +++ b/charts/ai/values.yaml @@ -202,6 +202,21 @@ tolerations: [] affinity: {} replicaCount: 1 +# Rollout strategy. The default stops the old pod before it starts the new one +# (a short outage on a single replica), instead of the Kubernetes default +# (maxSurge 25%, which rounds up to one extra pod). Pods of one release share an +# embedded Hazelcast cluster, and a pod only joins a cluster of its own +# Hazelcast minor version: when a release changes that version, an old pod still +# running makes the new pod fail to join and restart in a loop, so a surge +# rollout never completes. With more than one replica, upgrade across a +# Hazelcast version with Recreate (see "Upgrade" in README.md): +# --set deploymentStrategy.type=Recreate --set deploymentStrategy.rollingUpdate=null +deploymentStrategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 0 + maxUnavailable: 1 + extraEnvVars: [] extraVolumes: [] extraVolumeMounts: [] From 260cf61e4ed3c50f2f551dd917156d6c7ee0d391 Mon Sep 17 00:00:00 2001 From: Daniel Garcia Rico Date: Thu, 12 Nov 2026 04:26:54 +0000 Subject: [PATCH 2/2] AITRAX-2921: chart README - what the image contains, inspecting a running pod The image is built on Wolfi: list the tools it has and lacks, JAVA_HOME and the JDK trust store path, and that apk needs root. Add how to inspect a pod with kubectl exec and, for other tools, an ephemeral debug container (including namespaces that enforce Pod Security restricted). Co-Authored-By: Claude Opus 5.5 --- charts/ai/README.md | 52 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/charts/ai/README.md b/charts/ai/README.md index f2eefe1..1e1ebed 100644 --- a/charts/ai/README.md +++ b/charts/ai/README.md @@ -374,6 +374,58 @@ If login fails after returning from Google or another identity provider with and that the registered OAuth redirect URI exactly matches `{service_fqdn}/api/v1/auth/sso/provider/callback`. +### What the container image contains + +The image is built on [Wolfi](https://github.com/wolfi-dev), a minimal Linux +distribution, and runs as the non-root user `ctai` (uid and gid `2200`). + +- Shell and tools: `bash`, BusyBox `sh` (`/bin/sh`), `curl`, `ps`, `top`, + `ping`, `netstat`, `vi`, `less`, and the GNU core utilities. `ss`, `ip`, + `openssl` and `dig` are not included. +- Java: OpenJDK 25 with its tools (`jcmd`, `jstack`, `jmap`, `jfr`, + `keytool`). `JAVA_HOME` is `/usr/lib/jvm/default-jvm`, and the JDK's trust + store is `/etc/ssl/certs/java/cacerts`. +- Packages: the package manager is `apk`. It needs root, so it cannot install + anything in the running container. An image built `FROM` this one installs + packages with `apk add` as root before switching back to `USER 2200`. + +### Inspect a running pod + +`kubectl exec` is enough for the tools above: + +```bash +kubectl exec -it -n codetogether-ai deploy/codetogether-ai -c codetogether-ai -- bash +``` + +For a tool the image does not have, attach an ephemeral debug container. It +shares the application's process namespace and network, so it sees the Java +process, its files under `/proc//root` and `localhost`, and nothing +restarts. Use kubectl 1.32 or later, and any image your cluster can pull: + +```bash +POD=$(kubectl get pod -n codetogether-ai -l app.kubernetes.io/name=codetogether-ai -o name | head -1) +kubectl debug -it -n codetogether-ai "$POD" \ + --image= \ + --target=codetogether-ai --profile=general -c debug -- sh +``` + +If the namespace enforces the Pod Security `restricted` profile, which does not +allow a root debug container, use the CodeTogether AI image itself as the debug +image and run it as the application's user. The image names its user (`ctai`), +so the numeric uid has to be given: + +```bash +echo '{"securityContext":{"runAsUser":2200,"runAsGroup":2200}}' > uid2200.json +IMAGE=$(kubectl get "$POD" -n codetogether-ai -o jsonpath='{.spec.containers[0].image}') +kubectl debug -it -n codetogether-ai "$POD" --image="$IMAGE" \ + --target=codetogether-ai --profile=restricted --custom=uid2200.json \ + -c debug-app -- bash +``` + +An ephemeral container stays in the pod's spec until the pod is replaced, and +each one needs a new name (`-c`). The process environment and the mounted +properties file contain secrets: do not paste them into tickets or chat. + ## Upgrade ```bash