Skip to content

hermetic: no_refetch ignores local_path_override modules and local = True repository rules (part of #11443) - #12703

Closed
TrapsterDK wants to merge 6 commits into
mainfrom
fix-11443-no-refetch-local
Closed

TrapsterDK wants to merge 6 commits into
mainfrom
fix-11443-no-refetch-local

Conversation

@TrapsterDK

@TrapsterDK TrapsterDK commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #12277.

Root cause (windows_arm64 pkl lint): the runner hands lsp_check verbatim scratch paths (\\?\C:\...). file_uri turned them into file:////%3F/C%3A/..., pkl-lsp threw in textDocument/didOpen (SettingsManager.handleTextDocumentEvent, shown by run 43659879059), never published diagnostics, and the session timed out. file_uri now drops the verbatim prefix through dx_path::without_verbatim and spells UNC paths as file://host/share/....

Also in this PR:

  • lsp_check default session timeout 45s, so it fails before the 60s small-test limit and shows the server stderr.
  • matrix_harness writes the full failure to $TEST_UNDECLARED_OUTPUTS_DIR/matrix_failure.txt and prints it with control characters escaped.

Evidence:

Local (linux_x86_64): //core/quality/lsp_check:lsp_check_test, matrix_pkl_lint_pass, matrix_pkl_lint_fail pass; tools/dx check //core/quality/lsp_check/... clean. No nested-Bazel or consumer tests run locally.

Remaining:

  • windows_arm64 //core/quality/testdata:matrix_pkl_lint_{pass,fail,module,syntax_error}: pkl-lsp throws at SettingsManager.kt:53 during the first didOpen. Check which settings/workspace-folder or root URI that line reads on Windows (likely initialize rootUri/workspaceFolders or the workspace/configuration reply from lsp_check), and keep the exception header line of pkl-lsp stderr in the lsp_check error (it is currently cut off).
  • Then check whether preflight_smoke_test stops timing out on windows_arm64.
  • Rerun tools/dx check --affected with output saved (previous run reported 2 failing tests, names cut off).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xut2hvuiHK9vSNJZWiqKfX

write_root_patch_overrides copies the root MODULE.bazel single_version_override of protobuf (and its //patches/protobuf files) into the composed consumer, like rules_py/rules_flutter; patches/protobuf:module_srcs joins //:module_srcs. When #12723 drops the protobuf override, nothing is copied.

Tests:

  • //tools/testing/hermetic:hermetic_unit_test passes locally (new unit tests for the filter, show_repo parsing, label mapping, the local-attribute parser and mod argv).
  • //tools/testing/hermetic:cpp_rust builds locally; it is a nested test and was not run locally.
  • //tools/testing/nested:dx_nested_test and //tools/testing/hermetic:hermetic_unit_test pass locally (linux_x86_64), also on the combined branch fix-11418-julia-package-name.
  • Evidence: windows_x86_64 //tools/testing/hermetic:cpp_rust on the combined branch: run 43660885365 green (dx_generate_check ok, Kotlin coverage green, qualified). Run 43658126513 passed refetch_rules_* and coverage but failed dx_generate_check: the cpp_rust consumer java/kotlin test rules lacked size = "small", now added in tools/testing/hermetic/consumers/cpp_rust/apps/. Run 43656923559 timed out (default 600 s test timeout).
  • the_consumer_materializes_from_dx_init writes the rules_julia patches main's write_rules_py_patches copies.
  • Earlier evidence: linux_x86_64 //tools/testing/hermetic:cpp_rust run 43558347963 failed before no_refetch ran: the fetch probe hit Unable to determine host libc! in rules_py's uv host repo (hermetic: consumer fetch fails in rules_py uv host repo: Unable to determine host libc #12482, fixed by rules_py: uv host repo reports glibc 2.28 without running ldd (part of #12482) #12485). Re-run after rules_py: uv host repo reports glibc 2.28 without running ldd (part of #12482) #12485 merges.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xut2hws3L7Tg4oUvpSvuYD

@TrapsterDK

Copy link
Copy Markdown
Collaborator Author

Evidence run 43560359806 (linux_x86_64, //tools/testing/hermetic:cpp_rust, test_timeout=1800, combo of this PR + #12485 + #12641 on main): fetch and fetch_coverage now pass, and the ldd failure is gone (#12485 merged). The run fails in this PR's new refetch_rules probe: bazel mod show_repo exits 37 because it evaluates rules_python's pip extension (twine/cryptography publish deps), which the build never evaluated, and pypi.org is unreachable in the isolation. mod rejects --incompatible_strict_action_env, so the flags are not the cause. The probe needs a repo-rule source that does not run bazel mod, for example reading the rule from the output base's repo marker files or @<repo>.marker. Earlier run 43559324793 timed out at the default 600s, so use test_timeout=1800.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xut2hws3L7Tg4oUvpSvuYD

TrapsterDK and others added 2 commits November 14, 2026 06:29
…True repository rules (part of #11443)

The probe drops the modules the consumer overrides with local_path_override,
then asks `bazel mod show_repo --output=streamed_jsonproto` for the rule of
each remaining refetched repository and drops those whose repository_rule
declaration sets local = True. mod probes carry only common flags.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xut2hws3L7Tg4oUvpSvuYD
…h its own repository cache (part of #11443)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xut2hws3L7Tg4oUvpSvuYD
@TrapsterDK
TrapsterDK force-pushed the fix-11443-no-refetch-local branch from c279325 to 32e7862 Compare November 14, 2026 05:29
@TrapsterDK

Copy link
Copy Markdown
Collaborator Author

Run 43560435796 (combo of this PR + #12485 + #12641) failed in refetch_rules the same way: offline bazel mod show_repo evaluates rules_python's pip extension. c279325 runs refetch_rules with network on the cold output root and its own repository cache, so it neither needs pypi offline nor seeds the shared cache. Rebased on main (with #12485). Evidence run 43561755161 on combo-11443-12482 (this PR + #12641), test_timeout=3000.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xut2hws3L7Tg4oUvpSvuYD

@TrapsterDK

Copy link
Copy Markdown
Collaborator Author

Run 43561755161 (this PR + #12641, test_timeout=3000): the online refetch_rules probe no longer hits pypi, prints 48 show_repo records, but exits 2: bazel mod evaluates every extension and rules_rust crate_universe fails with crate_universe extension call crates is in a non-root module but has no lockfile ("Results may be incomplete as 1 extension failed"). Next: either accept a partial mod exit when stdout parses, or read the rule from repo marker files instead of bazel mod. Branch combo-11443-12482 kept for the rerun.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xut2hws3L7Tg4oUvpSvuYD

@TrapsterDK

Copy link
Copy Markdown
Collaborator Author

Root cause of the refetch_rules exit 2 in run 43561755161: the failing crates call is protobuf's, not rules_dx's. protobuf 36.2 (the rules_dx floor) declares a non-dev crate = use_extension("@rules_rust//crate_universe:extension.bzl", "crate") with crate.from_specs() and no lockfile. rules_dx strips it with //patches/protobuf:no_crate_universe.patch, but that single_version_override only applies when rules_dx is root, so every consumer's bazel mod evaluates the unpatched call and rules_rust 0.75.0 fails it. All rules_dx crate.from_cargo calls already ship lockfiles.

protobuf 37.0-rc3 (BCR) and protobuf main add lockfile = "//rust:Cargo.Bazel.lock" to from_specs, which fixes this upstream. No final 37.0 is on BCR yet. Blocked on the protobuf 37.0 release; then bump the protobuf floor, drop no_crate_universe.patch, and rerun //tools/testing/hermetic:cpp_rust evidence on combo-11443-12482.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xut2hws3L7Tg4oUvpSvuYD

@TrapsterDK

Copy link
Copy Markdown
Collaborator Author

The protobuf 37.0-rc3 bump is in draft #12723 (drops no_crate_universe.patch, adds a grpc upb_reflection_cc.patch). This PR rebases on it once #12723 merges. Before that, #12723 needs the paket lock bump, the nested harness re-record, and a fix for the lock-extension budget. The cpp_rust evidence rerun waits on those.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xut2hws3L7Tg4oUvpSvuYD

…ws 1024-character command line (part of #11443)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xut2hws3L7Tg4oUvpSvuYD
@TrapsterDK

Copy link
Copy Markdown
Collaborator Author

Run 43651406826 (Windows cpp_rust): every probe passed, including fetch_coverage. The run failed only at the no_refetch check: "a restarted server refetched repositories" (local_config_winsdk, host_platform, local_jdk, ...). #12703 fixes that check. I stacked #12703's 3 commits, plus a unit-test fix that writes the rules_julia patches in the_consumer_materializes_from_dx_init (that test also fails on main), onto fix-11418-julia-package-name. Then I dispatched run 43653857809. The coverage probe, which covers Kotlin, runs after no_refetch, so it is still unverified.

TrapsterDK and others added 3 commits November 15, 2026 01:10
…aluates every extension (part of #11443)

The cpp_rust refetch_rules probe runs `bazel mod show_repo`, which evaluates
every module extension. protobuf 36.2's non-root crate.from_specs() has no
lockfile, so crate_universe failed (Windows run 43653857809). The composed
consumer MODULE.bazel now carries rules_dx's protobuf single_version_override
and its patch, as it does for rules_py and rules_flutter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xut2hvuiHK9vSNJZWiqKfX
…aterialize test writes rules_julia patches (part of #11443)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xut2hvuiHK9vSNJZWiqKfX
@TrapsterDK
TrapsterDK marked this pull request as ready for review November 15, 2026 01:33
@TrapsterDK

Copy link
Copy Markdown
Collaborator Author

All changes on this branch landed on main via the squash of #12641 (merge commit a8bb68c), which carried this branch. After merging main into this branch, git diff origin/main HEAD is empty. Windows evidence: run 43660885365 (cpp_rust, windows_x86_64) green.

@TrapsterDK TrapsterDK closed this Nov 15, 2026
@TrapsterDK
TrapsterDK deleted the fix-11443-no-refetch-local branch November 15, 2026 01:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant