Skip to content

workloads: Migrate Kafka source, tests and profile to open source - #826

Draft
AlexWFMS wants to merge 2 commits into
mainfrom
agents/migrate-kafka-workload-to-open-source
Draft

AlexWFMS wants to merge 2 commits into
mainfrom
agents/migrate-kafka-workload-to-open-source

Conversation

@AlexWFMS

Copy link
Copy Markdown
Contributor

Summary

Migrate the Kafka workload into the public Virtual Client projects while preserving
the existing command, profile, exception, warmup, concurrency, and telemetry behavior.

  • Add base/client/server executors and producer/consumer parsers.
  • Migrate unit tests, functional tests, output examples, and PERF-KAFKA.json.
  • Add dependency-installation coverage for all four advertised platforms and
    direct profile-schema validation.
  • Add workload/profile documentation describing actual metric capture, parameters,
    fixed ports, topology limitations, Windows scripts, and troubleshooting.

Validation

Check Result
Original Kafka unit tests 29 passed
Original Kafka functional tests 19 passed
Public Kafka unit tests 29 passed
Public Kafka functional tests 24 passed
Direct Kafka profile schema validation Included in passing functional tests
Production project and normal analyzers Passed
Patch whitespace Passed
Claude Opus 5.5 implementation critique No runtime/test-parity regressions found; documentation findings corrected
Website build Blocked: dependency restore failed (proxy 404 and TLS handshake failure)
Live old/new parity Not run

Tests use the existing in-memory fixtures; platform test cases are not evidence of
live ARM64/Windows/Linux parity.

Draft / release blockers

This PR is not ready to merge:

  • The exact kafka_2.13-3.8.0-2.zip archive is not yet in the public package store.
    Resolve the modified Windows scripts' redistribution/notice question before
    copying the archive. Do not silently substitute a different archive.
  • The JDK archive is already public with matching name, byte length, MD5, and
    content type; all four platform binaries were verified.
  • Complete the website build when dependency access is available.
  • Select a unique next public patch version and update VERSION; package-version
    lookup was blocked by TLS failures. No prerelease packages have been published.
  • Complete live A/B parity for linux-x64, linux-arm64, win-x64, and win-arm64,
    in both single-system and client/server topologies.
  • Obtain coworker verification and explicit owner direction before merging.

Coordinated rollout

Merge the validated public change first, then publish all four runtime packages
and the Framework/TestFramework prereleases from merged code. Only after remote
package availability is verified should the paired internal cleanup update all
six pins, remove duplicate implementation/profile/documentation, and validate
its assembled artifacts. No internal removal or merge has occurred.

Intentional behavior preservation

Producer output has a parser, but the current executor emits benchmark metrics
only for consumer commands. This migration does not enable producer telemetry,
rename metrics, harden parsing, upgrade Kafka, or repair inherited runtime behavior.

Preserve the internal workload behavior and metric schema while adapting namespaces and public test helpers. Add dependency and schema coverage and public workload documentation. Package publication, live parity and coordinated internal cleanup remain release gates.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Build and verify all six local prerelease packages. Publication remains gated on the reviewed public merge.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant