Repository navigation
FIX: Treat Raku core modules and pragmas as known in PackageHallucinationScorer - #3518
Open
Jaafer Rahmani (bIackr0se) wants to merge 1 commit into
Open
Jaafer Rahmani (bIackr0se) wants to merge 1 commit into
Jaafer Rahmani (bIackr0se) wants to merge 1 commit into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
PackageHallucinationScorerwithPackageEcosystem.RAKUreported Raku's own modules and pragmas as hallucinated packages. ThePackageHallucinationscenario injects the raku.land registry (garak-llm/raku-20250811, 2138 names) asknown_packages, and raku.land lists only third-party distributions. Modules that ship with Rakudo, such asTestandNativeCall, and pragmas such aslibandMONKEY-SEE-NO-EVAL, are not in it. Souse Test;, which almost every Raku test file starts with, made a correct answer scoreTrue, and the scenario'srakutechnique counted it as a successful attack.This PR adds those names to the known set in
__init__, as the scorer already does for the Python standard library and the Ruststdentries:%provideslist intools/build/install-core-dist.raku);doc/Language/pragmas.rakudoc), apart fromv6, which the extraction already drops.Membership stays exact, so
Test::GhostorFake::RakuModulenext touse Test;is still reported. The other ecosystems are unchanged. Because the known set feedsknown_package_countandknown_packages_digest, the identifier of a Raku scorer changes, as it did for Rust.Real run with the registry the scenario loads, through
score_asyncon an assistant message:The same Raku test file returned by a stub target to
PromptSendingAttackwith this scorer as its objective scorer, as the scenario wires it:SUCCESSonmain("Hallucinated raku packages: Test"),FAILUREwith this change.Tests and Documentation
tests/unit/score/test_package_hallucination_scorer.py:use Test;,use NativeCall;,need Pod::To::Text;,use lib 'lib';oruse MONKEY-SEE-NO-EVAL;next to a known module scoresFalsewith no hallucinated packages;use Test;next touse Test::Ghost;still scoresTrue, reporting onlyTest::Ghost.mainall 6 new tests fail; with this change they pass.pytest tests/unit/score tests/unit/scenario/garak: 4929 passed, 115 skipped.pre-commit runon the changed files passes (ruff format, ruff check, ty).