Skip to content

feat(system_test): SVT discovery cleanup, PowerVault and PowerScale CSI inputs, per-cluster inputs file - #5814

Merged
Rajeshkumar-s2 merged 5 commits into
dell:stagingfrom
balajikumaran-c-s:feat/svt-discovery-cleanup-powervault-powerscale
Nov 16, 2026
Merged

Rajeshkumar-s2 merged 5 commits into
dell:stagingfrom
balajikumaran-c-s:feat/svt-discovery-cleanup-powervault-powerscale

Conversation

@balajikumaran-c-s

@balajikumaran-c-s balajikumaran-c-s commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

PR Title

feat(system_test): SVT discovery cleanup, PowerVault and PowerScale CSI inputs, per-cluster inputs file

Description of the Solution

The SVT (test/system_test) could not clean up discovery, build a PowerVault, place the PowerScale CSI files, or let a cluster pick its inputs file. This PR adds all four: a discovery cleanup step, a powervault_config generated from three user values, a files setup step that stages the PowerScale secret.yaml and values.yaml only when the CSI is enabled, and a per-cluster <CLUSTER>_SVT_USER_INPUTS variable. It changes test code and documentation only; nothing under src/ is touched.

Related Issue

Changes

Discovery cleanup (test/system_test/library/vars/theme_vars.py, build_stream_test_config.yml)

  • New cleanup step fvt_discovery cleanup test, after the orchestrator and before image_build_manager. It runs only when discovery holds deployed state (a file under <data>/discovery/output or log), like telemetry; otherwise it is skipped and the log says why.
  • cleanup.discovery has only enabled: discovery's credential and log options are playbook variables set by its own cleanup tests.
  • Discovery is added to the report suites and to the test requirements installed in the Omnia venv.

PowerVault (test/system_test/dataset_generator/dataset_merge.py, user_inputs.yml)

  • orchestrator.storage_config.powervault_config asks for ip (a list or comma separated), iscsi_initiator and volume_id; the generator holds the other defaults (port 3260, /mnt/slurm, xfs, per-host bind directories, slurm_control_node). A key added to the item overrides its default; an unknown key is an error.
  • Without an ip there is no PowerVault: the item is ignored and a block from an earlier run is removed. With an ip, the other two are required.
  • The shipped storage_config.yml has the block only as a comment, so the generator creates the key and merges the item like any list item. The comment in user_inputs.yml explains how the volume is mounted on the nodes.
  • A rerun of the generator with an empty ib_addr_mode no longer fails once the key was removed.

PowerScale CSI (test/system_test/library/functions/setup_func.py, _setup.py, setup_env.sh, pipeline/*)

  • New setup step files (./setup_env.sh --stage-files; last step of a run with no action; run by the pipeline's initialization job after the credentials). It copies secret.yaml and values.yaml to the paths in omnia_config.yml only when enable_powerscale_csi is true, after the install because the clean reinstall deletes that folder.
  • When it is true, both files must exist, have content, be valid YAML and differ from the reference examples, and the paths must be absolute, different and in an existing folder; otherwise the step fails and names the file and the fix.
  • The two path keys may use $NAME / ${NAME} of omnia.env (default $ORCHESTRATOR_DATA_PATH/input/$OMNIA_PROJECT_NAME/..., the orchestrator project input folder); an unknown variable stops the run and is named.
  • setup_svt_project.py --create seeds each cluster folder of the GitLab project with the product's reference examples (src/orchestrator/examples/powerscale_reference_files/CSI_driver/). They are added only when missing and --update-inputs never replaces them. The checkout job copies the project's files next to user_inputs.yml (mode 0600, content on stdin only).
  • The files stay plaintext: the orchestrator encrypts its own copy in place with a key that a clean reinstall deletes. The array user and password come from the orchestrator credentials, not from secret.yaml.

Inputs file per cluster (test/system_test/pipeline/svt_ci.py, svt_pipeline_config.yml)

  • svt.user_inputs per cluster (default user_inputs.yml) becomes the CI/CD variable <CLUSTER>_SVT_USER_INPUTS. The checkout job copies the chosen file to the OIM as user_inputs.yml. The name must be a plain .yml/.yaml file name; a bad name or a file that is not in the project fails before anything on the OIM is touched.

Test hygiene (test/system_test/requirements.txt, test/build_stream/nft/unit/..., dataset_edit.py, pipeline/setup_svt_project.py)

  • requirements.txt uses exact versions (pytest 9.2.0, pytest-order 1.5.0, PyYAML 6.0.3, ruamel.yaml 0.19.1), the ones the other suites pin and the SVT venv already runs.
  • Two build_stream tests no longer write fixed /tmp paths into their config (they use tmp_path / the test's temporary directory).
  • The CSV writer in dataset_edit.py is marked # nosec (PXE inventory read by the orchestrator).
  • The constant PASSWORD_VAR is renamed OIM_LOGIN_VAR (it holds a CI variable name; output and API calls are unchanged), so CodeQL's name-based rule no longer reads it as a password.

Documentation (test/system_test/README.md, docs/*, pipeline/README.md)

  • Updated for the discovery step, PowerVault, the PowerScale files and the files step, and the per-cluster inputs file.

Files Changed

File Change Type Description
test/system_test/library/vars/theme_vars.py Modified Discovery cleanup step, flags, report suite, requirements, deployed-only rule
test/system_test/build_stream_test_config.yml Modified cleanup.discovery block and cleanup order
test/system_test/dataset_generator/dataset_merge.py Modified PowerVault defaults and gating, $NAME path expansion, empty ib_addr_mode rerun fix
test/system_test/dataset_generator/dataset_publish.py Modified Passes the generated omnia.env to the path expansion
test/system_test/dataset_generator/dataset_edit.py Modified # nosec on the CSV writer
test/system_test/library/functions/setup_func.py Modified stage_files step and its checks
test/system_test/library/messages/svt_msgs.py Modified Messages for the files step
test/system_test/library/vars/common_vars.py Modified files action and step, PowerScale constants
test/system_test/_setup.py Modified Runs the files step
test/system_test/setup_env.sh Modified --stage-files option
test/system_test/pipeline/svt_ci.py Modified Inputs file choice, PowerScale file copy, --stage-files after the credentials
test/system_test/pipeline/setup_svt_project.py Modified svt.user_inputs variable, seeding of the reference examples, constant rename
test/system_test/pipeline/svt_pipeline_config.yml Modified svt.user_inputs key per cluster
test/system_test/user_inputs.yml Modified PowerVault item (three values), PowerScale defaults and comments
test/system_test/requirements.txt Modified Exact version pins
test/build_stream/nft/unit/api/parse_catalog/conftest.py Modified Config paths from tmp_path
test/build_stream/nft/unit/infra/artifact_store/test_file_artifact_store.py Modified Config path from the temporary directory
test/system_test/README.md Modified Optional inputs, directory structure
test/system_test/pipeline/README.md Modified Inputs of a cluster section, variables
test/system_test/docs/SETUP.md Modified PowerVault, PowerScale CSI, --stage-files
test/system_test/docs/PIPELINE.md Modified Cluster folder files, initialization flow
test/system_test/docs/PIPELINE_CONFIG.md Modified user_inputs setting and variable
test/system_test/docs/RUNNER.md Modified Cleanup list includes discovery
test/system_test/docs/TEST_CONFIG.md Modified Cleanup order and discovery

Testing

Run:

  • Generator on a clean upstream/staging tree: PowerVault with a list and with a comma separated ip, no ip (ignored), rerun unchanged, an earlier block removed, an ip without iscsi_initiator/volume_id, an unknown key, an override; the output passes the orchestrator's storage_config.json schema and matches the live multisubnet PowerVault entry field by field. PowerScale paths with the default, a custom OMNIA_DATA_PATH/OMNIA_PROJECT_NAME, rerun unchanged, and an unknown variable (error).
  • files step with temporary folders: disabled and no cluster (skipped), both files edited (copied, mode 0600, an existing encrypted copy replaced), and each failure (missing file, unedited reference example, invalid YAML, empty/relative/.. path, same path, missing folder), with the error text checked. --create seeding returns the two reference examples per cluster byte for byte.
  • svt_ci.py with a mocked remote: inputs file default, empty, named, missing, ../, absolute and non-YAML names (all failures happen before any remote command); secret content only on stdin; the initialization order is configure, install, credentials, --stage-files, stage.
  • Cleanup step selection with and without discovery state.
  • The two changed build_stream test areas: 29 passed, 2 skipped (named skips, unrelated).
  • Local run of the repo's secure_code_review.py over test/system_test and test/build_stream; remaining hits were checked and are false positives or the explicit SVT_SSH_SKIP_HOST_KEY opt-out.

Not run:

  • Nothing ran on an OIM or in GitLab after merge: the real fvt_discovery cleanup test, the PowerVault mount on nodes and the PowerScale deployment are not exercised.
  • No new unit tests (this part of the repo has none); bandit not run locally.
  • CI on the current head (e171cf7cb): all 15 checks pass, including "Validate linked issue and IP policy", Bandit, Ansible Lint, Secure Code Review, Unit Tests & Coverage and Scan for secrets.

Backward Compatibility

  • No breaking changes. A cluster's existing project files keep working: a build_stream_test_config.yml without cleanup.discovery defaults to enabled (the step still runs only with discovery state), an empty <CLUSTER>_SVT_USER_INPUTS means user_inputs.yml, and without PowerScale files, or with enable_powerscale_csi: false, the new step is skipped.
  • A user_inputs.yml without powervault_config, or with an empty ip, leaves storage_config.yml untouched.
  • A pipeline project that has the new svt_ci.py needs an OIM checkout that has --stage-files (this PR); an older checkout fails at that step with Unknown option: --stage-files.
  • No lab values: the template has an empty PowerVault item, enable_powerscale_csi: false and $-based default paths.

Suggested Reviewers

Reviewer Review Focus
@abhishek-sa1 SVT cleanup order, setup step, CI and repo conventions
@Rajeshkumar-s2 SVT pipeline (svt_ci.py, GitLab variables), PowerScale and PowerVault inputs
@snarthan Test hygiene changes and orchestrator test impact

…SI inputs, per-cluster inputs file

- Cleanup: add the discovery suite (after the orchestrator, only when discovery holds
  deployed state) to the SVT cleanup steps, settings, report suites and test requirements.
- PowerVault: user_inputs.yml gives only ip (a list or comma separated), iscsi_initiator and
  volume_id; the generator holds the defaults and builds powervault_config in
  storage_config.yml. Without an ip there is no PowerVault (ignored, an earlier block is
  removed); with an ip the other two are required.
- PowerScale CSI: new setup step "files" (./setup_env.sh --stage-files, last step of a run, and
  run by the pipeline after the credentials) copies secret.yaml and values.yaml to the paths of
  omnia_config.yml only when enable_powerscale_csi is true; both must exist, have content and
  be valid YAML. The paths may use $NAME of omnia.env and default to the orchestrator project
  input folder. Header-only secret.yaml and values.yaml ship next to user_inputs.yml and are
  seeded (never replaced) in each cluster folder of the GitLab project.
- Pipeline: each cluster picks its inputs file with <CLUSTER>_SVT_USER_INPUTS (svt.user_inputs,
  default user_inputs.yml).
- Fix: a rerun with an empty ib_addr_mode no longer fails once the key was removed.

Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com>
@balajikumaran-c-s
balajikumaran-c-s force-pushed the feat/svt-discovery-cleanup-powervault-powerscale branch from b256702 to 6a8dc23 Compare November 16, 2026 10:07
…eference examples

- --create adds secret.yaml and values.yaml to each cluster folder of the GitLab project by
  copying src/orchestrator/examples/powerscale_reference_files/CSI_driver/ (only when missing,
  never replaced, not by --update-inputs); the user edits them for the array there.
- The "files" step fails, naming the file and what to edit, when a staged file is still identical
  to its reference example: an unedited example would only fail later, at deployment.
- Drop the header-only secret.yaml and values.yaml from test/system_test: the examples are the
  single source, so they cannot drift.

Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com>
…d_stream tests

- system_test/requirements.txt: exact versions (pytest 9.2.0, pytest-order 1.5.0, PyYAML 6.0.3,
  ruamel.yaml 0.19.1), the ones the other suites pin and the SVT .venv already runs.
- build_stream nft: the parse_catalog conftest and the invalid-config artifact store test no
  longer write fixed /tmp paths into their config; they use pytest's tmp_path / the test's
  temporary directory.
- system_test dataset_edit.py: mark the CSV writer as not spreadsheet output (read by the
  orchestrator).

Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com>
setup_svt_project.py printed the CI variable NAME (for example MULTISUBNET_SVT_TARGET_PASS), never
its value, but the name came from a constant called PASSWORD_VAR, which CodeQL's name-based rule
reads as a password flowing into print(). Rename the constant to OIM_LOGIN_VAR; the messages,
the prompt and the API calls are unchanged.

Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com>
@abhishek-sa1

Copy link
Copy Markdown
Collaborator

LGTM. Well-implemented features with comprehensive validation, excellent backward compatibility, and good test hygiene improvements.

Minor suggestion: Consider adding a comment in the header-only and files indicating they are templates and should be edited before use (e.g., "# This is a template file for PowerScale CSI driver secrets. Edit this file with your actual PowerScale credentials before deployment.").

@abhishek-sa1 abhishek-sa1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. This PR adds well-implemented SVT capabilities (discovery cleanup, PowerVault config, PowerScale CSI file staging, per-cluster inputs) with comprehensive validation, excellent backward compatibility, and good test hygiene. No production code changes, no security concerns.

@Rajeshkumar-s2
Rajeshkumar-s2 merged commit cb85fd2 into dell:staging Nov 16, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants