Repository navigation
Build and test conda packages and wheels - #265
jharlow-intel wants to merge 4 commits into
Conversation
There was a problem hiding this comment.
🔵 Needs a closer look
The native-package pipeline requires end-to-end Linux/Windows CI validation of conda solving and runtime loading for 3.14t/3.15t, which remain unverified by standard-library-only checks.
No open issues 🎉
What changed in this PR
Adds GitHub Actions build and package testing for Intel Python CI, preparing mkl-service for Jenkins integration. Jenkins is not yet enabled for this repository.
Changes:
- Derives Linux and Windows build matrices from the conda-forge recipe, selecting four builds for pull requests and all 16 for master pushes and tags.
- Adds build and isolated package-test scripts, with pytest results written as JUnit reports.
- Documents the integration and its configuration.
| File | Description |
|---|---|
conda-recipe-cf/conda_build_config.yaml |
Declares Python versions and ABI variants for matrix generation. |
ci/test.py |
Checks the interpreter and runs installed-package tests with JUnit output. |
ci/README.md |
Documents CI operation and local commands. |
ci/matrix.py |
Generates platform and Python matrices from build declarations. |
ci/hooks.py |
Installs artifacts into clean environments and runs package tests. |
ci/config.yml |
Defines the workflow, artifact naming, and test requirements. |
ci/build.py |
Builds packages for a selected Python variant and collects artifacts. |
AGENTS.md |
Adds a pointer to Intel Python CI documentation. |
.github/workflows/intel-ci.yml |
Orchestrates matrix generation, builds, artifact uploads, and tests. |
.github/AGENTS.md |
Documents the workflow and its recipe-derived matrix exception. |
Review effort: Balanced
ee0bbbf to
eaaedc1
Compare
0db81d3 to
f4c2344
Compare
926e627 to
c484738
Compare
vchamarthi
left a comment
vchamarthi
left a comment
There was a problem hiding this comment.
Gh-ci side looks good to me. Scripts match the other three repos, artifact names line up with ci/config.yml, actions are pinned.
The PR description is out of date though. It says the workflow builds conda-recipe-cf (deleted now), Linux and Windows only (macOS and wheels are built too), and that Jenkins isn't on for mkl-service (it's enabled in config/gh-ci.json, dry run). It also doesn't say conda-package.yml and the Intel-channel build are gone, which is the part reviewers need to see. Rest are inline.
jharlow-intel
changed the title
|
Thanks @vchamarthi, I've rewritten the description and the title to match what the PR does now. |
jharlow-intel
changed the title
7bba6b0 to
94ceba3
Compare
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
A few points inline, none of them blocking. One more for the description: with conda-package.yml and the Intel-channel recipe removed, no public workflow builds or tests against the Intel channel any more (intel-ci.yml uses conda-forge only too). Worth saying that outright.
Adds a ci/ directory and .github/workflows/intel-ci.yml. The workflow builds conda-recipe/ with conda-build and the wheels with cibuildwheel, for every platform and Python version the recipe and pyproject.toml declare, and tests each package in a clean environment with ci/test.py. The test jobs run for every package that built, even when another build failed. conda-build is pinned to 26.9.1. conda-recipe/ is now the only recipe, the conda-forge one that was conda-recipe-cf/; the Intel-channel recipe and conda-package.yml, which copied wheels out through WHEELS_OUTPUT_FOLDER, are removed. conda-package-cf.yml reports the same check names, so the required checks still run. The conda packages and the wheels both start at Python 3.11. The wheels' run-time requirement on mkl comes from [tool.gh-ci] wheel-requires in pyproject.toml, which ci/build.py adds to each wheel: in [project] it would fail the recipe's pip check, as conda-forge's mkl has no pip metadata. The wheels build against mkl-devel >=2026.2,<2027, the range they require at run time, and the Linux wheels are tagged manylinux_2_28 only, like the mkl wheel. ci/build.py keeps only the $ORIGIN entries of each ELF file's RPATH and RUNPATH in a wheel, as meson-python keeps the isolated build env's absolute lib directory, and refuses a file that would be left with none. ci/hooks.py tests each wheel in a venv made from conda-forge's Python, as pip users install it. Every python: entry in conda_build_config.yaml names its ABI, which ci/matrix.py requires, and every push and tag run gets a concurrency group of its own.
0d201fc to
30f391e
Compare
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
Re-reviewed the latest push: everything from my last round except the dev upload is addressed, and I resolved those threads. A few small points inline, none of them blocking.
conda-build reads x86 as any x86 CPU and has no osx64 selector, while rattler-build has x86 for 32-bit only and osx64 for Intel macOS, so ci/matrix.py now gives conda-build its own values for both and refuses a conda-build skip that uses osx64. ci/build.py keeps a search path entry only when it is $ORIGIN or a directory under it, names RPATH and RUNPATH entries apart, and rewrites the wheels in scratch, so a refused wheel never reaches --output.
[build-system] requires plain mkl-devel again. [tool.gh-ci] pin-compatible maps mkl to mkl-devel, and ci/build.py requires mkl >= the mkl-devel version python -m build installed for the build, < its next major, the way conda's run_exports pin it, so nothing needs a bump for a new MKL release. The version comes from the list python -m build logs, and a build that logs none fails. The repair commands exclude libmkl_rt.so.* and mkl_rt.*.dll, so a newer MKL library name is still left out of the wheels.
| # Some Windows build backends write the metadata with CRLF. | ||
| eol = "\r\n" if "\r\n" in text else "\n" |
There was a problem hiding this comment.
Nit: eol comes from the whole file, so a METADATA whose headers use LF but whose description has CRLF, such as a README checked out with CRLF and copied as it is, gets eol = "\r\n". Then partition("\r\n\r\n") finds no blank line, and the Requires-Dist line goes at the end of the description:
>>> with_requirements(b"Metadata-Version: 2.1\nName: mkl-service\n\n# README\r\ntext\r\n", ["mkl >=2026.2.0,<2027"])
b'Metadata-Version: 2.1\nName: mkl-service\n\n# README\r\ntext\r\nRequires-Dist: mkl >=2026.2.0,<2027\r\n'The build still logs build: added Requires-Dist ..., and pip installs that wheel without mkl, so the first sign would be the wheel test failing to load MKL. meson-python writes the description with LF today (the Windows wheels' METADATA has no CRLF), so nothing hits it now.
Taking the line ending from the headers alone fixes it, and gives the same output as now for LF, for CRLF, and for CRLF headers with an LF description, with or without one:
| # Some Windows build backends write the metadata with CRLF. | |
| eol = "\r\n" if "\r\n" in text else "\n" | |
| # Some Windows build backends write the metadata with CRLF. Take the line | |
| # ending from the headers, as the description can keep the README's. | |
| headers = re.split(r"\r?\n\r?\n", text, maxsplit=1)[0] | |
| eol = "\r\n" if "\r\n" in headers else "\n" |
|
|
||
| The scripts need nothing beyond the Python standard library and the tools each one's docstring names. macOS packages build and test on Intel runners, as MKL has no Apple silicon build. | ||
|
|
||
| **A pull request into `master`.** Actions builds the oldest and the newest regular Python versions on Linux, Windows and macOS, as a conda package and, where `pyproject.toml` declares the platform, as a wheel, and tests each package. |
There was a problem hiding this comment.
Nit: intel-ci.yml runs on every pull request, as its pull_request: trigger has no branches: filter, like the other workflows here. So a pull request into another branch gets the same reduced matrix.
| **A pull request into `master`.** Actions builds the oldest and the newest regular Python versions on Linux, Windows and macOS, as a conda package and, where `pyproject.toml` declares the platform, as a wheel, and tests each package. | |
| **A pull request.** Actions builds the oldest and the newest regular Python versions on Linux, Windows and macOS, as a conda package and, where `pyproject.toml` declares the platform, as a wheel, and tests each package. |
| name: mkl-service ${{ matrix.name }} Python ${{ matrix.python }} | ||
| path: ${{ runner.temp }}/output/*.conda | ||
| if-no-files-found: error | ||
| # Pull requests keep them 3 days, tags 30. |
There was a problem hiding this comment.
Nit: the expression gives 3 days to every run that isn't a tag, so packages from pushes to master are kept 3 days too, not only the pull requests' ones. The wheel upload below uses the same expression.
| # Pull requests keep them 3 days, tags 30. | |
| # Tags keep them 30 days, pull requests and pushes to master 3. |
The Linux wheels build with the hardening flags conda-forge's compilers set for the conda packages: -fstack-protector-strong, -D_FORTIFY_SOURCE=2 and -Wl,-z,relro -Wl,-z,now. ci/build.py refuses a wheel whose ELF files lack full RELRO, so the flags can't silently stop applying. conda-build gets --extra-meta remote_url=... sha=... when GitHub Actions sets GITHUB_SERVER_URL, GITHUB_REPOSITORY and GITHUB_SHA, so each conda package records the commit it was built from in info/about.json. A pull request also builds and tests the newest free-threaded version, so the free-threaded wheels build before a merge.
Adds
ci/and.github/workflows/intel-ci.yml, which build the conda packages for Linux, Windows and Intel macOS and the wheels for Linux and Windows, on every declared Python, and test each in a clean environment.conda-recipe/is now the only recipe: the conda-forge one that wasconda-recipe-cf/. The Intel-channel recipe andconda-package.ymlare removed;conda-package-cf.ymlstays and keeps its check names. No workflow in this repository builds or tests against the Intel channel any more:conda-package-cf.ymlandintel-ci.ymlboth use conda-forge only.pyproject.tomlclassifiers, which drop 3.10 here.requires-pythonstill allows 3.10.mklfrom themkl-develversion the build used up to its next major, through[tool.gh-ci] pin-compatible, whichci/build.pystamps into each wheel.ci/build.pyalso keeps only the$ORIGINRPATH and RUNPATH entries of the wheels' ELF files, dropping the isolated build env's absolute path. The Linux wheels are taggedmanylinux_2_28, like themklwheel.meson.buildaddsLibraryto CMake's search path on Windows, where mkl-devel puts its CMake config.ci/test.pyrunspytest --pyargs mklon the installed package: a conda package in a conda-forge environment, a wheel in a venv made from conda-forge's Python, somkl/_init_helper.pyis what loads MKL on Windows.Validation: run 42277684124 built and tested the conda packages for 3.11 and 3.15 on Linux, Windows and macOS, and the wheels for 3.11 and 3.15 on Linux and Windows. Not run yet: the full matrix, which only
masterand tags build.