Repository navigation
AITRAX-2921: codetogether-ai chart 0.2.4 - deploymentStrategy, old pod stops first - #232
Merged
Merged
Conversation
…d stops first The chart rendered no strategy, so Kubernetes' default (maxSurge 25%, which rounds up to one extra pod) started the new pod while the old one ran. ct-ai's next release moves Hazelcast from 5.5 to 5.7, and a pod only joins a cluster of its own minor version: on the dev cluster, upgrading a single replica with chart 0.2.3 left the new pod crash-looping while the old one kept serving, and the rollout never completed. deploymentStrategy now defaults to RollingUpdate with maxSurge 0 and maxUnavailable 1, as in ct-ai's internal chart, and can be set to Recreate for multi-replica installs. The template reads it through default, so an upgrade with --reuse-values from an earlier chart gets the same strategy. README documents the upgrade, and CI checks the three renders. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ning pod The image is built on Wolfi: list the tools it has and lacks, JAVA_HOME and the JDK trust store path, and that apk needs root. Add how to inspect a pod with kubectl exec and, for other tools, an ephemeral debug container (including namespaces that enforce Pod Security restricted). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
codetogether-aichart 0.2.4 (AITRAX-2921). The chart rendered nostrategy, so Kubernetes' default applied:RollingUpdatewithmaxSurge: 25%, which rounds up to one extra pod, so the new pod starts while the old one still runs.That breaks the next ct-ai release (CodeTogether-Inc/ct-ai#3131), which moves the embedded Hazelcast from 5.5 to 5.7. An open-source Hazelcast member only joins a cluster of its own minor version. The new pod finds the running old pod through
hazelcast_cluster_nodes, its join is rejected (Joining node's version 5.7.0 is not compatible with cluster version 5.5), the application context fails and the pod crash-loops. The old pod is never removed because the new one never becomes ready: the upgrade never completes, even with one replica. The old pod keeps serving, andhelm upgradestill printsdeployed.Change
deploymentStrategyvalue, defaultRollingUpdatewithmaxSurge: 0,maxUnavailable: 1(same default as ct-ai's internal chartdocker/ctai/helm). The old pod stops before the new one starts, so a single-replica upgrade has a short outage instead of a surge.default, sohelm upgrade --reuse-valuesfrom 0.2.3 (nodeploymentStrategyin the stored values) gets the same strategy.Recreatefor an upgrade across a Hazelcast version:--set deploymentStrategy.type=Recreate --set deploymentStrategy.rollingUpdate=null.--set-json, and akubectl patchfor a rollout already stalled on 0.2.3.deploymentStrategy=null(the--reuse-valuescase) and aRecreateoverride.README: the Wolfi image
The same ct-ai release also moves the image from Ubuntu 24.04 to Wolfi (CodeTogether-Inc/ct-ai#3149). Two subsections are added under "Validation and Troubleshooting":
ctai(2200);bash,curl,ps,ping,netstat, the JDK tools) and the ones it lacks (ss,ip,openssl,dig);JAVA_HOME(/usr/lib/jvm/default-jvm) and the JDK trust store (/etc/ssl/certs/java/cacerts), which is the path a mounted trust store has to use;apk, which needs root, so nothing can be installed in the running container.kubectl execfor those tools, and an ephemeralkubectl debugcontainer for anything else, including the case of a namespace that enforces Pod Securityrestricted(the app image run as uid 2200).These commands are the ones tested for ct-ai's internal chart on the dev cluster (Kubernetes 1.35, image
b8695, all three cases in ct-ai'sdocs/runbooks/wolfi-images.md), with this chart's names. The names were checked withhelm template: Deployment,app.kubernetes.io/nameand container are allcodetogether-ai, and the container runs as 2200. The tool list was checked on the Ubuntu imageb8679and the Wolfi imageb8695; ports,/opt/ctai, locale and time zone are the same in both.Test plan
Dev cluster (
do-nyc3-saas-dev-us), throwaway namespace with its own Postgres, 1 replica, imagesb8650(Hazelcast 5.5) →b8687(5.7, feature build of ct-ai#3131). The old pod ran past its first registry heartbeat before each upgrade, as a long-running production pod would:helm upgrade --reuse-valueswith this chart: strategy applied, old pod stopped, new pod ready, rollout complete./actuator/health200. The rollback 5.7 → 5.5 also completes.helm lintand everycharts/airender step inhelm-template-check.yml; the new strategy checks fail against 0.2.3 and pass here.Multi-replica (3 replicas, ct-ai internal chart with the same strategy):
Recreatecompletes and also unsticks a stalled rolling update; details in https://github.com/CodeTogether-Inc/ct-ai/pull/3131#issuecomment-6749651876.Notes
dev. The release PR tomain(which publishes) should ship before or with the ct-ai release that contains Hazelcast 5.7.manageReplicas) edits the same lines ofdeployment.yaml; whichever merges second needs a small rebase.🤖 Generated with Claude Code