Raw output of bash scripts/fetch-revisions.sh && bash scripts/benchmark.sh.
| date | 2026-09-16 |
| agentbound | 0.1.8 |
| sisakulint | v0.3.7 (built from @latest; the binary reports Tool version: unknown) |
| zizmor | 1.30.1 |
Versions matter. Every number below is reproducible only against the tool versions above, and the fixtures are pinned by commit SHA so the inputs cannot drift underneath them.
These numbers were first taken at 0.1.3. Between that run and this one the
tool changed in ways that removed six false-positive classes and two false
negatives — job-scoped reachability, the permissions: block-scalar misread, the
empty opt-out, the named-account opt-out — and then added a third severity tier
for how far the agent’s mutating tools reach.
Every verdict on real/1-vulnerable.yml, real/2-fix-commit.yml and
real/3-later.yml is identical across all three versions. The published revisions
are the case the rule was generalised from, so this is the check that the
precision work did not cost recall on the thing it was written for.
The new tier is not exercised by anything in this repository, and that is worth
saying rather than leaving a reader to assume otherwise: the vulnerable fixture’s
patterns are unbounded, and the fixed one has no mutating tool at all, so neither
reaches the middle. It was exercised on the wider corpus instead —
MHSanaei/3x-ui reports medium because every mutating pattern it grants
names $, and evcc-io/evcc and
tokio-rs/toasty report high because theirs match any argument. A fixture
that cannot reach a branch is not a test of it.
The only line numbers that moved are in fixtures/fixed.yml, and only because
the fixture’s header gained the layer annotations that
docs/mitigations.md refers to. The findings are the same
two; they sit 16 lines lower.
--- fixtures/vulnerable.yml
HIGH ci-agent-write-scope-on-untrusted-trigger @30
HIGH ci-agent-untrusted-issue-content @46
--- fixtures/fixed.yml
CRITICAL ci-agent-missing-author-association @50
LOW ci-agent-write-scope-on-untrusted-trigger @57
--- real/1-vulnerable.yml
HIGH ci-agent-write-scope-on-untrusted-trigger @17
HIGH ci-agent-untrusted-issue-content @33
--- real/2-fix-commit.yml
CRITICAL ci-agent-missing-author-association @30
HIGH ci-agent-write-scope-on-untrusted-trigger @38
--- real/3-later.yml
CRITICAL ci-agent-missing-author-association @41
LOW ci-agent-write-scope-on-untrusted-trigger @49
--- fixtures/vulnerable.yml
2 [commit-sha]
1 [deprecated-node-runtime]
1 [dependabot-github-actions]
1 [artipacked]
1 [ai-action-prompt-injection]
--- fixtures/fixed.yml
4 [commit-sha]
2 [deprecated-node-runtime]
2 [ai-action-execution-order]
2 [ai-action-excessive-tools]
1 [dependabot-github-actions]
1 [bot-conditions]
--- real/1-vulnerable.yml
2 [commit-sha]
1 [deprecated-node-runtime]
1 [dependabot-github-actions]
1 [bot-conditions]
1 [artipacked]
1 [ai-action-prompt-injection]
--- real/2-fix-commit.yml
4 [commit-sha]
2 [deprecated-node-runtime]
2 [bot-conditions]
2 [artipacked]
1 [dependabot-github-actions]
--- real/3-later.yml
4 [commit-sha]
2 [deprecated-node-runtime]
2 [bot-conditions]
2 [ai-action-execution-order]
2 [ai-action-excessive-tools]
1 [dependabot-github-actions]
--- fixtures/vulnerable.yml
2 error[unpinned-uses]
1 warning[artipacked]
--- fixtures/fixed.yml
4 error[unpinned-uses]
--- real/1-vulnerable.yml
2 error[unpinned-uses]
1 warning[artipacked]
--- real/2-fix-commit.yml
4 error[unpinned-uses]
2 warning[artipacked]
--- real/3-later.yml
4 error[unpinned-uses]
No tool is wrong here; they answer different questions, and the differences are the useful part.
zizmor is a general-purpose Actions auditor. It reports the same thing on every revision — pinned-action and credential-persistence hygiene — and nothing about the injection class. That is not a defect; it is out of scope for the tool. It is included because “the popular linter was clean on the vulnerable file” is the kind of thing that gets mistaken for a clean bill of health.
sisakulint has purpose-built AI rules, and it is the only tool here that names
the CVE’s own mechanism: ai-action-prompt-injection fires on
1-vulnerable.yml, where the issue title and body are interpolated into
prompt:, and stops firing once the fix removes that interpolation. Correct, and
the strongest signal in this table. Its two findings on the later revision are
worth reading before acting on them:
ai-action-excessive-tools flags Write as dangerous. nnU-Net uses Write
on purpose — the agent writes /tmp/issue-comment.md, and a later step posts
it precisely so the agent cannot post directly. Removing Write would break
the mitigation, not improve it.ai-action-execution-order wants the agent to be the last step. That is the
opposite of what this file wants: the agent is not last because the
privilege-bearing steps come after it, outside the model’s reach.Both are honest observations about a general pattern that this particular file
has already reasoned about. They are the same shape of finding as
agentbound’s ci-agent-missing-author-association: the rule is describing a
real condition, and the file is a deliberate exception to it.
agentbound is the only tool here that distinguishes 2-fix-commit.yml from
3-later.yml, and that distinction is the reason this repository exists — see
the README. Its ci-agent-missing-author-association finding is reported at
critical on both, and is described in its own README as over-severe rather than
false, because auto-triage is meant to be public.
Every mitigation the fixture set demonstrates is tagged with the layer it
implements, and with whether it is load-bearing, in
fixtures/fixed.yml and
docs/mitigations.md. Two of the six changes in that file
are not injection bounds — persist-credentials: false is general credential
hygiene, and fetching the issue instead of interpolating it reduces the chance of
compliance without removing the capability. They are labelled that way on
purpose: a reader who counts them as mitigations would overestimate the file.