[jiatolentino]

Summary

dm works with DataMasque projects. Every project-scoped command takes --project, list commands show each resource's project, the ruleset bundle commands export and import through the v2 endpoints, which file each item under its project, and new dm projects and dm resources move commands manage projects and move resources between them.

Note

Draft until a released datamasque-python has projects. See Dependency.

Why

DataMasque v3.26.21.0 puts every connection, ruleset, library, discovery config, seed, file and table reference in a project, and libraries use projects in place of namespaces. Without --project the CLI cannot choose where a resource goes or which of two same-named resources it means.

What changed

• --project (a project name in any case, or its numeric ID) on connections, rulesets, libraries, discovery configs, discovery config libraries, seeds, files, table references, runs and discovery runs.
  • List commands show a Project column, and --project lists one project's resources (dm seeds list also keeps the built-in seeds).
  • Create, upload and validate commands use the --project project, else the instance's only project. With several projects and no --project they fail and ask for one.
  • Get, update and delete commands find a name in any project, and ask for --project when it exists in more than one.
  • dm table-references create and update look up --connection in the reference's own project first, then in every project.
• --namespace is removed from dm libraries and dm discover libraries; use --project.
• dm rulesets export-bundle and import-bundle go through the client's export_bundle and import_bundle, which use the v2 endpoints.
  • export-bundle --include-seed-files also bundles the custom seed files the rulesets and libraries use.
  • import-bundle --fallback-project names the project for items a bundle files under no project (omitted: the server's default), and the CLI says which project took them. The client warns about an older-format bundle.
• dm projects: list (--include-deleted adds deleted projects), get, create, update (rename or change the description) and delete.
  • delete takes --to, the project that receives the deleted project's resources. When a resource cannot move, the server deletes nothing and the CLI lists each conflict and exits with the conflict code. --check lists the conflicts without deleting.
• dm resources move --to <project> moves resources selected by --project, --type, --search or exact names, with their files and runs.
  • It needs at least one selector, and leaves out built-in seeds and resources already in the destination.
  • A name that matches none of the selected resources aborts with the not_found code and lists each such name, so nothing moves.
  • A resource that clashes with the destination stays where it is and the others move. The CLI lists each one that stayed and exits with the conflict code. --check lists the selection and its conflicts without moving anything.
• src/datamasque_cli/projects.py: the shared --project resolution and output helpers.
• README Projects section, and the bundled skills, updated for --project, dm projects and dm resources move.

Breaking: --namespace is gone, and the CLI needs a DataMasque version with projects. The --project and bundle commits carry a BREAKING CHANGE: footer.

Testing

• make check: ruff, format, mypy and deptry clean; 486 unit tests pass. install-smoke fails as described under Dependency.
• Against a DataMasque development build with projects, with two projects:
  • every command group: create, list, get, update and delete in each project, the "name in more than one project" and "project not found" paths, runs and discovery runs;
  • export-bundle writes a v2 bundle with each item under its project, and re-importing it skips each item by ID;
  • export-bundle --include-seed-files adds each seed file under its project, and without it the bundle has none;
  • a v1 bundle with --fallback-project lands in that project and names it, without it lands in the server's default project, and an unknown project exits 3 before uploading;
  • dm resources move with a mistyped name, or a name --type leaves out, exits 3 naming it and moves nothing.
• dm projects and dm resources move against a development build with project rename, on a new instance: create, a name clash, get, rename, description change, list --include-deleted, delete with and without a conflict, --check, declining the prompt, moving one ruleset, and moving a project with one conflict (the rest moved, exit code conflict).
• Integration tests: the two rewritten for projects pass. The rest create without --project and expect a single-project instance; they are still to run on one.

Dependency

• Needs a released datamasque-python with projects. The first commit (build: install datamasque-python from TestPyPI until a release has projects) pins >=1.7.0.dev11 from TestPyPI and is dropped before merging, in favour of the released version.
• Until then install-smoke fails: it installs the wheel from PyPI, where the newest datamasque-python is 1.5.0. Lint, mypy and pytest resolve the TestPyPI build from uv.lock.
• Needs DataMasque v3.26.21.0, which adds projects. The bundle commands also need its v2 export and import endpoints, which are not final yet; the bundle commit follows any change to them.
• The changelog entry and version bump come with the switch to the released client.

[andrewchester-dm]

PROJECT_OPTION_HELP defined here; the text "Project name or id" is repeated inline in --project help strings (e.g. connections.py, rulesets.py, files.py) could you build them from this?

[jiatolentino]

Done, every --project help now builds from PROJECT_OPTION_HELP, the repeated "two projects share a name" ones through get_same_name_project_help.