Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions .github/workflows/helm-template-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,27 @@ jobs:
--set imageCredentials.username=test \
--set imageCredentials.password=test \
--set gateway=null
# Rollout strategy: the default, what --reuse-values from a chart
# before 0.2.4 leaves (no deploymentStrategy), and a Recreate override.
ai_strategy() {
helm template test ./charts/ai \
--set-file ctaiPropertiesFile=./charts/ai/ctai-configuration.properties.template \
--set service.fqdn=https://example.codetogether.io \
--set imageCredentials.username=test \
--set imageCredentials.password=test \
--show-only templates/deployment.yaml "$@" \
| sed -n '/^ strategy:$/,/^ selector:$/p'
}
default_strategy="$(ai_strategy)"
grep -q 'type: RollingUpdate' <<<"$default_strategy"
grep -q 'maxSurge: 0' <<<"$default_strategy"
grep -q 'maxUnavailable: 1' <<<"$default_strategy"
[ "$(ai_strategy --set deploymentStrategy=null)" = "$default_strategy" ]
recreate_strategy="$(ai_strategy --set deploymentStrategy.type=Recreate --set deploymentStrategy.rollingUpdate=null)"
grep -q 'type: Recreate' <<<"$recreate_strategy"
if grep -q 'rollingUpdate' <<<"$recreate_strategy"; then
echo "Recreate rendered rollingUpdate" >&2; exit 1
fi

- name: Validate Live Chart
run: |
Expand Down
2 changes: 1 addition & 1 deletion charts/ai/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: codetogether-ai
description: CodeTogether AI on-premises Kubernetes deployment

type: application
version: 0.2.3
version: 0.2.4
appVersion: "1.2.8616"

icon: https://www.codetogether.com/wp-content/uploads/2020/02/codetogether-circle-128.png
Expand Down
97 changes: 97 additions & 0 deletions charts/ai/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -374,6 +374,58 @@ If login fails after returning from Google or another identity provider with
and that the registered OAuth redirect URI exactly matches
`{service_fqdn}/api/v1/auth/sso/provider/callback`.

### What the container image contains

The image is built on [Wolfi](https://github.com/wolfi-dev), a minimal Linux
distribution, and runs as the non-root user `ctai` (uid and gid `2200`).

- Shell and tools: `bash`, BusyBox `sh` (`/bin/sh`), `curl`, `ps`, `top`,
`ping`, `netstat`, `vi`, `less`, and the GNU core utilities. `ss`, `ip`,
`openssl` and `dig` are not included.
- Java: OpenJDK 25 with its tools (`jcmd`, `jstack`, `jmap`, `jfr`,
`keytool`). `JAVA_HOME` is `/usr/lib/jvm/default-jvm`, and the JDK's trust
store is `/etc/ssl/certs/java/cacerts`.
- Packages: the package manager is `apk`. It needs root, so it cannot install
anything in the running container. An image built `FROM` this one installs
packages with `apk add` as root before switching back to `USER 2200`.

### Inspect a running pod

`kubectl exec` is enough for the tools above:

```bash
kubectl exec -it -n codetogether-ai deploy/codetogether-ai -c codetogether-ai -- bash
```

For a tool the image does not have, attach an ephemeral debug container. It
shares the application's process namespace and network, so it sees the Java
process, its files under `/proc/<pid>/root` and `localhost`, and nothing
restarts. Use kubectl 1.32 or later, and any image your cluster can pull:

```bash
POD=$(kubectl get pod -n codetogether-ai -l app.kubernetes.io/name=codetogether-ai -o name | head -1)
kubectl debug -it -n codetogether-ai "$POD" \
--image=<a debugging image> \
--target=codetogether-ai --profile=general -c debug -- sh
```

If the namespace enforces the Pod Security `restricted` profile, which does not
allow a root debug container, use the CodeTogether AI image itself as the debug
image and run it as the application's user. The image names its user (`ctai`),
so the numeric uid has to be given:

```bash
echo '{"securityContext":{"runAsUser":2200,"runAsGroup":2200}}' > uid2200.json
IMAGE=$(kubectl get "$POD" -n codetogether-ai -o jsonpath='{.spec.containers[0].image}')
kubectl debug -it -n codetogether-ai "$POD" --image="$IMAGE" \
--target=codetogether-ai --profile=restricted --custom=uid2200.json \
-c debug-app -- bash
```

An ephemeral container stays in the pod's spec until the pod is replaced, and
each one needs a new name (`-c`). The process environment and the mounted
properties file contain secrets: do not paste them into tickets or chat.

## Upgrade

```bash
Expand All @@ -399,6 +451,51 @@ helm upgrade codetogether-ai codetogether/codetogether-ai \

The Deployment rolls when the chart-managed properties Secret changes.

### Rollout strategy

From chart 0.2.4 the Deployment stops the old pod before it starts the new one
(`deploymentStrategy`: `RollingUpdate` with `maxSurge: 0`, `maxUnavailable: 1`).
On a single replica an upgrade therefore has a short outage while the new pod
starts. Earlier charts used the Kubernetes default, which starts the new pod
first.

The pods of one release share an embedded Hazelcast cluster, and a pod only
joins a cluster that runs its own Hazelcast minor version. When a release
changes that version (the release notes say so), any old pod still running
makes the new pod fail to join: its container exits and restarts in a loop, and
the rollout never completes while the old pods keep serving. `helm upgrade`
without `--wait` still prints `deployed`; `kubectl rollout status` shows the
stall.

- **One replica:** the default strategy handles it, as long as the release was
upgraded to chart 0.2.4 or later. Upgrade the chart and the image in the same
`helm upgrade`, as above.
- **More than one replica:** upgrade across a Hazelcast version with
`Recreate`, which stops every old pod first (a short outage):

```bash
helm upgrade codetogether-ai codetogether/codetogether-ai \
--namespace codetogether-ai \
--reset-then-reuse-values \
--set deploymentStrategy.type=Recreate \
--set deploymentStrategy.rollingUpdate=null
```

`rollingUpdate=null` is required: Helm merges overrides into the existing
values, and Kubernetes rejects `Recreate` while `rollingUpdate` is set. The
same command completes a rollout that has already stalled.
`--reset-then-reuse-values` keeps `Recreate` for later upgrades; to return to
the default on a later upgrade, add (Helm 3.10 or later)
`--set-json 'deploymentStrategy={"type":"RollingUpdate","rollingUpdate":{"maxSurge":0,"maxUnavailable":1}}'`.

If a rollout has already stalled on a chart older than 0.2.4, switch the
Deployment to `Recreate` directly:

```bash
kubectl -n codetogether-ai patch deployment codetogether-ai \
-p '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}'
```

## Uninstall

```bash
Expand Down
4 changes: 4 additions & 0 deletions charts/ai/templates/deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ metadata:
{{- include "ctai.labels" . | nindent 4 }}
spec:
replicas: {{ .Values.replicaCount }}
{{- /* Read through default: --reuse-values from a chart before 0.2.4 has no deploymentStrategy. */}}
{{- $strategy := .Values.deploymentStrategy | default (dict "type" "RollingUpdate" "rollingUpdate" (dict "maxSurge" 0 "maxUnavailable" 1)) }}
strategy:
{{- toYaml $strategy | nindent 4 }}
selector:
matchLabels:
{{- include "ctai.selectorLabels" . | nindent 6 }}
Expand Down
15 changes: 15 additions & 0 deletions charts/ai/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,21 @@ tolerations: []
affinity: {}
replicaCount: 1

# Rollout strategy. The default stops the old pod before it starts the new one
# (a short outage on a single replica), instead of the Kubernetes default
# (maxSurge 25%, which rounds up to one extra pod). Pods of one release share an
# embedded Hazelcast cluster, and a pod only joins a cluster of its own
# Hazelcast minor version: when a release changes that version, an old pod still
# running makes the new pod fail to join and restart in a loop, so a surge
# rollout never completes. With more than one replica, upgrade across a
# Hazelcast version with Recreate (see "Upgrade" in README.md):
# --set deploymentStrategy.type=Recreate --set deploymentStrategy.rollingUpdate=null
deploymentStrategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 0
maxUnavailable: 1

extraEnvVars: []
extraVolumes: []
extraVolumeMounts: []
Expand Down
Loading