Skip to content

test: replace the _tests profile with a unit test of profile scripts - #118

Open
cadenmyers13 wants to merge 1 commit into
diffpy:mainfrom
cadenmyers13:replace-tests-profile
Open

cadenmyers13 wants to merge 1 commit into
diffpy:mainfrom
cadenmyers13:replace-tests-profile

Conversation

@cadenmyers13

Copy link
Copy Markdown
Contributor

What problem does this PR address?

closes #106

CI on every PR has been failing at the last step, after the tests themselves pass:

======================= 139 passed, 9 warnings in 55.98s =======================
ERROR: Script failed (exit 1): _pytest.sh tar_url.txt
ERROR: Profile '_tests' installation failed.

cmi install _tests ran _pytest.sh. That script downloaded pinned release tarballs of srreal, srfit, pyobjcryst, structure and utils, listed in tar_url.txt, and ran each package's own test suite against whatever versions conda-forge had installed. In the latest run, srfit 3.3.0's test setup does import six. six isn't installed any more, so srfit's tests never start (ModuleNotFoundError: No module named 'six'). The other four suites pass, but the script exits 1 overall, so the whole profile is marked failed.

This breaks whenever a pinned tag and the conda-forge release drift apart. Someone also has to update tar_url.txt by hand after every dependency release. On top of that, these tests belong to other packages, and each of those runs its own CI.

The only diffpy.cmi behaviour this exercised was that a profile's extras scripts run when the profile is installed. This PR tests that directly:

  • Add test_install_with_script_extras to tests/test_profilesmanager.py. It builds a temporary profile whose extras list a small .sh and .bat script, installs it, and checks that:
    • the script ran with its argument
    • the install returns 0 when the script succeeds (C1) and 1 when it fails (C2)
      It needs no network or conda, and it runs on Linux, macOS and Windows.
  • Remove the _tests profile and its scripts (_pytest.sh, _pytest.bat, tar_url.txt).
  • In tests-on-pr.yml and matrix-and-codecov-on-merge-to-main.yml, replace cmi install _tests with cmi install pdf, so CI still checks that a real pack installs.
  • Remove the _tests section from docs/source/available-profiles.rst and add a news item.

Local results: tests/test_profilesmanager.py passes (3 passed). In my local env, the only other failure is test_version.py, because the editable install reports version 0.0.0. It is unrelated to this PR.

🤖 Generated with Claude Code

The _tests profile downloaded pinned release tarballs of srreal, srfit,
pyobjcryst, structure and utils and ran their test suites against the
conda-forge versions. It broke whenever those drifted apart (srfit 3.3.0
tests need six, which is no longer installed) and the URLs had to be
updated by hand. Remove the profile and its scripts, install the pdf
pack in CI instead, and cover script extras with a test in
tests/test_profilesmanager.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cadenmyers13

Copy link
Copy Markdown
Contributor Author

@sbillinge ready for review

@codecov

codecov Bot commented Nov 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.04%. Comparing base (d72e765) to head (1f7f21e).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #118      +/-   ##
==========================================
+ Coverage   93.63%   94.04%   +0.40%     
==========================================
  Files           4        4              
  Lines         220      235      +15     
==========================================
+ Hits          206      221      +15     
  Misses         14       14              
Files with missing lines Coverage Δ
tests/test_profilesmanager.py 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: update how cmi is tested so it doesn't use the .tar.gz files

1 participant