(WIP)CKS: reconcile node capacity after live resize

[andrijapanicsb]

Why this follow-up is needed

This draft follows #13226, which removed the CKS restriction that prevented changing node compute offerings while a KVM cluster was running. That work builds on #12975, the KVM live-resize support for fixed service offerings.

The VM resize itself can complete successfully: CloudStack records the new offering and libvirt exposes the additional CPU or RAM. Kubernetes may still be advertising the old values, though. In practice, the missing operator step was to restart kubelet on the resized node and check that the Node object's capacity and allocatable had caught up.

#14213 tracks that gap. This PR moves that reconciliation into the CKS offering-change workflow, so a successful scale operation means CloudStack, the guest, and Kubernetes agree on the node resources.

What this PR does

For a CPU-count or RAM increase on a running, Cloud-managed KVM worker or control-plane node, CKS now handles one node at a time:

1. Record the node's current schedulability and Kubernetes resource values.
2. Cordon the node when it was previously schedulable. Existing Pods are left alone.
3. Resize the VM using the existing CloudStack workflow.
4. Check the guest's online CPU count and MemTotal.
5. Restart kubelet, wait for it to become active, then wait for the Kubernetes Node to be Ready again.
6. Read the Node as JSON and verify its capacity and allocatable reflect the resized guest.
7. Uncordon only a node that CKS cordoned itself, then move on to the next node.

The target-node SSH connection is resolved from its actual port-forwarding rule for isolated/VPC networks, rather than assuming that every node is reachable on port 2222. Direct-access networks use the VM's private address on port 22.

Deliberately not done here

• This is a cordon-only workflow. A kubelet restart does not require an unconditional drain, so resident workloads are not evicted.
• Created, Stopped, cap-only, external-node, and non-KVM paths keep their existing behaviour.
• Dedicated etcd live resize is rejected for now. It needs explicit etcd health/quorum handling rather than Kubernetes-node logic.

Draft / QA notes

The focused reconciler and scale-worker tests pass locally. Early KVM QA found two follow-up cases that were easy to miss in unit tests: the scale worker is created through ComponentContext, so the small stateless reconciliation helper must be available directly to that worker; and a retry after CloudStack has already reached the target offering still needs to finish a CKS resize that had previously cordoned the node. This revision addresses both, uses value comparison for offering IDs, and keeps the observed KVM memory-hotplug overhead within the verification tolerance.

A fresh, isolated QA environment is now available for the final package-based test run. That run will exercise CPU-only, RAM-only and combined changes, a pre-cordoned node, retry/failure cleanup, and the final UI scale flow with guest and Kubernetes capacity evidence.

[codecov]

Codecov Report

❌ Patch coverage is 35.45455% with 142 lines in your changes missing coverage. Please review.
✅ Project coverage is 20.45%. Comparing base (af9764b) to head (7e5a95a).

Files with missing lines	Patch %	Lines
...er/actionworkers/KubernetesClusterScaleWorker.java	9.30%	73 Missing and 5 partials ⚠️
...s/KubernetesClusterNodeCapacityReconcilerImpl.java	49.13%	35 Missing and 24 partials ⚠️
...utils/KubernetesClusterNodeCapacityReconciler.java	72.22%	4 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##               main   #14223    +/-   ##
==========================================
  Coverage     20.45%   20.45%            
- Complexity    21415    21449    +34     
==========================================
  Files          6487     6489     +2     
  Lines        589303   589518   +215     
  Branches      72245    72303    +58     
==========================================
+ Hits         120525   120594    +69     
- Misses       455643   455759   +116     
- Partials      13135    13165    +30     

Flag	Coverage Δ
uitests	3.70% <ø> (ø)
unittests	21.75% <35.45%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[andrijapanicsb]

@ACSHomeBot package G

[ACSHomeBot]

Packaging results:

Result	Artifact	Platform
PASS	RPM	EL (EL8/9/10)
PASS	DEB	Ubuntu, Debian

Test packages are available at:

• EL9/RPM: https://f003.backblazeb2.com/file/andrijapanicsb-cloudstack-pr-builds/pr/14223/el9/
• Debian/APT: https://f003.backblazeb2.com/file/andrijapanicsb-cloudstack-pr-builds/pr/14223/debian/
• OpenPGP public key

[github-actions]

🔴 Test Coverage Grade: D — Marginal

Metric	Value
Line coverage	25.50%
Branch coverage	19.82%

Grade Scale

Grade	Line Coverage	Meaning
🟢 A	≥ 80%	Excellent - this code sleeps well at night 😴
🟡 B	60-79%	Good - almost there, don't stop now 😉
🟠 C	40-59%	Acceptable - your code is wearing a seatbelt, but no airbags 😬
🔴 D	20-39%	Marginal - boldly shipping where no test has gone before 🖖
⛔ F	< 20%	Failing - tests? what tests? 🔥

Branch coverage is shown as a secondary signal. Grade is determined by line coverage.
View full Actions run

[sonarqubecloud]

Quality Gate failed

Failed conditions
31.0% Coverage on New Code (required ≥ 40%)

See analysis details on SonarQube Cloud