Skip to content

AITRAX-2921: codetogether-ai chart 0.2.4 - deploymentStrategy, old pod stops first - #232

Merged
danc094codetogether merged 2 commits into
devfrom
AITRAX-2921-ai-chart-strategy
Nov 12, 2026
Merged

danc094codetogether merged 2 commits into
devfrom
AITRAX-2921-ai-chart-strategy

Conversation

@CodeTogetherBot

@CodeTogetherBot CodeTogetherBot commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

codetogether-ai chart 0.2.4 (AITRAX-2921). The chart rendered no strategy, so Kubernetes' default applied: RollingUpdate with maxSurge: 25%, which rounds up to one extra pod, so the new pod starts while the old one still runs.

That breaks the next ct-ai release (CodeTogether-Inc/ct-ai#3131), which moves the embedded Hazelcast from 5.5 to 5.7. An open-source Hazelcast member only joins a cluster of its own minor version. The new pod finds the running old pod through hazelcast_cluster_nodes, its join is rejected (Joining node's version 5.7.0 is not compatible with cluster version 5.5), the application context fails and the pod crash-loops. The old pod is never removed because the new one never becomes ready: the upgrade never completes, even with one replica. The old pod keeps serving, and helm upgrade still prints deployed.

Change

  • deploymentStrategy value, default RollingUpdate with maxSurge: 0, maxUnavailable: 1 (same default as ct-ai's internal chart docker/ctai/helm). The old pod stops before the new one starts, so a single-replica upgrade has a short outage instead of a surge.
  • The template reads it through default, so helm upgrade --reuse-values from 0.2.3 (no deploymentStrategy in the stored values) gets the same strategy.
  • Multi-replica installs set Recreate for an upgrade across a Hazelcast version: --set deploymentStrategy.type=Recreate --set deploymentStrategy.rollingUpdate=null.
  • README "Upgrade" → "Rollout strategy": the behaviour change, the multi-replica command, how to return to the default with --set-json, and a kubectl patch for a rollout already stalled on 0.2.3.
  • CI renders the default, deploymentStrategy=null (the --reuse-values case) and a Recreate override.

README: the Wolfi image

The same ct-ai release also moves the image from Ubuntu 24.04 to Wolfi (CodeTogether-Inc/ct-ai#3149). Two subsections are added under "Validation and Troubleshooting":

  • What the container image contains:
    • the non-root user ctai (2200);
    • the tools it has (bash, curl, ps, ping, netstat, the JDK tools) and the ones it lacks (ss, ip, openssl, dig);
    • JAVA_HOME (/usr/lib/jvm/default-jvm) and the JDK trust store (/etc/ssl/certs/java/cacerts), which is the path a mounted trust store has to use;
    • apk, which needs root, so nothing can be installed in the running container.
  • Inspect a running pod: kubectl exec for those tools, and an ephemeral kubectl debug container for anything else, including the case of a namespace that enforces Pod Security restricted (the app image run as uid 2200).

These commands are the ones tested for ct-ai's internal chart on the dev cluster (Kubernetes 1.35, image b8695, all three cases in ct-ai's docs/runbooks/wolfi-images.md), with this chart's names. The names were checked with helm template: Deployment, app.kubernetes.io/name and container are all codetogether-ai, and the container runs as 2200. The tool list was checked on the Ubuntu image b8679 and the Wolfi image b8695; ports, /opt/ctai, locale and time zone are the same in both.

Test plan

Dev cluster (do-nyc3-saas-dev-us), throwaway namespace with its own Postgres, 1 replica, images b8650 (Hazelcast 5.5) → b8687 (5.7, feature build of ct-ai#3131). The old pod ran past its first registry heartbeat before each upgrade, as a long-running production pod would:

  • Published 0.2.3, 5.5 → 5.7: new pod crash-looping (5 restarts in 9 min), old pod serving, rollout never completes.
  • That stalled release, helm upgrade --reuse-values with this chart: strategy applied, old pod stopped, new pod ready, rollout complete.
  • This chart from the start, 5.5 → 5.7: rollout complete in ~70 s, 0 restarts, /actuator/health 200. The rollback 5.7 → 5.5 also completes.
  • helm lint and every charts/ai render step in helm-template-check.yml; the new strategy checks fail against 0.2.3 and pass here.

Multi-replica (3 replicas, ct-ai internal chart with the same strategy): Recreate completes and also unsticks a stalled rolling update; details in https://github.com/CodeTogether-Inc/ct-ai/pull/3131#issuecomment-6749651876.

Notes

  • This PR goes to dev. The release PR to main (which publishes) should ship before or with the ct-ai release that contains Hazelcast 5.7.
  • Allow Helm to omit Deployment replicas #214 (open, manageReplicas) edits the same lines of deployment.yaml; whichever merges second needs a small rebase.

🤖 Generated with Claude Code

…d stops first

The chart rendered no strategy, so Kubernetes' default (maxSurge 25%,
which rounds up to one extra pod) started the new pod while the old one
ran. ct-ai's next release moves Hazelcast from 5.5 to 5.7, and a pod only
joins a cluster of its own minor version: on the dev cluster, upgrading a
single replica with chart 0.2.3 left the new pod crash-looping while the
old one kept serving, and the rollout never completed.

deploymentStrategy now defaults to RollingUpdate with maxSurge 0 and
maxUnavailable 1, as in ct-ai's internal chart, and can be set to Recreate
for multi-replica installs. The template reads it through default, so an
upgrade with --reuse-values from an earlier chart gets the same strategy.
README documents the upgrade, and CI checks the three renders.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ning pod

The image is built on Wolfi: list the tools it has and lacks, JAVA_HOME and
the JDK trust store path, and that apk needs root. Add how to inspect a pod
with kubectl exec and, for other tools, an ephemeral debug container
(including namespaces that enforce Pod Security restricted).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@danc094codetogether
danc094codetogether merged commit 1aa810a into dev Nov 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants