Address CodeRabbit/Codex review on AI triage v2

- A (security): authenticate the ai-plan comment selector — only accept a
  plan from a trusted author (OWNER/MEMBER/COLLABORATOR), so a reporter can't
  inject a plan that executes on approval. Fail-safe to no-plan otherwise.
- B: clear ai:approved in the no-plan branch so a later real plan can be
  re-approved (re-adding a present label fires no labeled event).
- C: claim ai:needs-info via a live re-check inside the serialized job so
  queued reporter replies can't each run a classification; restore the flag if
  no verdict was produced so the issue doesn't drop out of the retry path.
- D: gate workflow_dispatch of the tier-3 executor to github.actor == alexbelgium.
- E: exempt ai:approved from the stale bot.
- F: filter catch-up candidates server-side (search) instead of capping at the
  100 newest issues, so older untriaged issues aren't silently missed.
- G: run ai_guard_paths.sh from the trusted default-branch copy, not the
  in-tree copy a job could have modified.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
alexbelgium
2026-07-24 08:20:49 +02:00
parent ddf06a7647
commit 731c79f2b7
5 changed files with 107 additions and 21 deletions

View File

@@ -178,4 +178,15 @@ jobs:
env:
GH_TOKEN: ${{ secrets.AI_PR_TOKEN }}
REPO: ${{ github.repository }}
run: bash .github/scripts/ai_guard_paths.sh
run: |
set -euo pipefail
# Run the guard from the trusted default-branch copy, never the in-tree
# copy the model could have modified in this same job.
DEFAULT=$(gh api "repos/${REPO}" --jq '.default_branch' 2>/dev/null || echo master)
if git fetch --depth=1 origin "$DEFAULT" >/dev/null 2>&1 \
&& git cat-file -e "FETCH_HEAD:.github/scripts/ai_guard_paths.sh" 2>/dev/null; then
git show "FETCH_HEAD:.github/scripts/ai_guard_paths.sh" | bash
else
echo "::warning::trusted guard copy unavailable, using in-tree copy"
bash .github/scripts/ai_guard_paths.sh
fi

View File

@@ -32,5 +32,5 @@ jobs:
# AI triage labels are exempt: an issue waiting on @alexbelgium's
# `ai:approved`, or queued for the tier-2 sweep, must not be auto-closed
# out from under the pipeline before it is acted on.
exempt-issue-labels: 'prevent stale,ai-triage,ai:plan-pending,ai:needs-info,ai:needs-human'
exempt-issue-labels: 'prevent stale,ai-triage,ai:plan-pending,ai:approved,ai:needs-info,ai:needs-human'
remove-stale-when-updated: true

View File

@@ -62,9 +62,13 @@ jobs:
"The \`ai:approved\` label only takes effect when applied by @alexbelgium; removing it."
execute:
# workflow_dispatch is a maintainer override, so it must also be gated to
# @alexbelgium — otherwise any collaborator with run-workflow rights could
# execute a Tier 3 plan without the approval label or sender check.
if: >-
vars.AI_DISABLED != 'true' &&
( github.event_name == 'workflow_dispatch' ||
( ( github.event_name == 'workflow_dispatch' &&
github.actor == 'alexbelgium' ) ||
( github.event.label.name == 'ai:approved' &&
github.event.sender.login == 'alexbelgium' ) )
runs-on: ubuntu-latest
@@ -103,14 +107,28 @@ jobs:
mkdir -p /tmp/ai-exec
gh issue view "$ISSUE" --repo "$REPO" \
--json number,title,body,author,labels,comments > /tmp/ai-exec/issue.json
# The plan is the most recent comment carrying the ai-plan marker.
jq -r '[.comments[] | select(.body | contains("<!-- ai-plan -->"))]
# The plan is the most recent ai-plan comment FROM A TRUSTED AUTHOR.
# The marker alone is not proof of origin: any reporter can paste
# "<!-- ai-plan -->" into a comment, and picking it by marker+last
# would let them swap in a plan that then executes on approval.
# Tier 2 posts under AI_PR_TOKEN, whose identity is a repo OWNER/
# MEMBER/COLLABORATOR; a reporter is never one of those. (If you
# switch AI_PR_TOKEN to a GitHub App, add its bot login here.)
jq -r '[.comments[]
| select((.body | contains("<!-- ai-plan -->"))
and (.authorAssociation == "OWNER"
or .authorAssociation == "MEMBER"
or .authorAssociation == "COLLABORATOR"))]
| last | .body // ""' /tmp/ai-exec/issue.json > /tmp/ai-exec/plan.md
if [ ! -s /tmp/ai-exec/plan.md ]; then
echo "has_plan=false" >> "$GITHUB_OUTPUT"
echo "::warning::no <!-- ai-plan --> comment on issue #$ISSUE; nothing to execute"
echo "::warning::no trusted <!-- ai-plan --> comment on issue #$ISSUE; nothing to execute"
# Consume the approval here too, so a later real plan can be
# re-approved (re-adding ai:approved to an issue that still carries
# it would not fire a fresh labeled event).
gh issue edit "$ISSUE" --repo "$REPO" --remove-label ai:approved >/dev/null 2>&1 || true
gh issue comment "$ISSUE" --repo "$REPO" --body \
"No AI plan (\`<!-- ai-plan -->\`) was found on this issue, so \`ai:approved\` has nothing to execute. Run the tier-2 sweep on it first (\`AI fix sweep\` → issue $ISSUE)."
"No AI plan (\`<!-- ai-plan -->\`) from the triage bot was found on this issue, so \`ai:approved\` has nothing to execute (removed). Run the tier-2 sweep on it first (\`AI fix sweep\` → issue $ISSUE), then approve the plan it posts."
else
echo "has_plan=true" >> "$GITHUB_OUTPUT"
echo "plan: $(wc -c < /tmp/ai-exec/plan.md) bytes"
@@ -161,4 +179,15 @@ jobs:
env:
GH_TOKEN: ${{ secrets.AI_PR_TOKEN }}
REPO: ${{ github.repository }}
run: bash .github/scripts/ai_guard_paths.sh
run: |
set -euo pipefail
# Run the guard from the trusted default-branch copy, never the in-tree
# copy the model could have modified in this same job.
DEFAULT=$(gh api "repos/${REPO}" --jq '.default_branch' 2>/dev/null || echo master)
if git fetch --depth=1 origin "$DEFAULT" >/dev/null 2>&1 \
&& git cat-file -e "FETCH_HEAD:.github/scripts/ai_guard_paths.sh" 2>/dev/null; then
git show "FETCH_HEAD:.github/scripts/ai_guard_paths.sh" | bash
else
echo "::warning::trusted guard copy unavailable, using in-tree copy"
bash .github/scripts/ai_guard_paths.sh
fi

View File

@@ -80,16 +80,33 @@ jobs:
environment: CR_PAT
steps:
# The reply that re-triggered this run consumes the needs-info flag, so a
# later comment on the same thread cannot re-run tier 1 again. apply-verdict
# re-adds it only if the issue still lacks the info (one more round).
- name: Consume needs-info flag
# A reporter reply re-triggered this run. Multiple replies can each pass
# the job `if` before the first run clears the flag; cancel-in-progress
# is false, so without this they would each run a full classification.
# Re-check the LIVE label inside the serialized job and consume it here:
# the first queued run finds it present and proceeds (go=true); any run
# behind it finds it already gone and skips every downstream step.
# apply-verdict re-adds the flag if the issue still needs info (one more
# round), or restores it if no verdict was produced (so a later reply can
# still retry instead of the issue silently dropping out).
- name: Claim needs-info reply
id: claim
if: github.event_name == 'issue_comment'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
ISSUE: ${{ github.event.issue.number }}
REPO: ${{ github.repository }}
run: gh issue edit "$ISSUE" --repo "$REPO" --remove-label ai:needs-info || true
run: |
set -euo pipefail
HAS=$(gh issue view "$ISSUE" --repo "$REPO" --json labels \
--jq '[.labels[].name] | index("ai:needs-info") != null')
if [ "$HAS" != "true" ]; then
echo "ai:needs-info already consumed by an earlier queued run; skipping"
echo "go=false" >> "$GITHUB_OUTPUT"
exit 0
fi
gh issue edit "$ISSUE" --repo "$REPO" --remove-label ai:needs-info || true
echo "go=true" >> "$GITHUB_OUTPUT"
# on_issues_ping_submitter.yml has to land first: the classifier reads
# the existing comments and bails out if someone already owns the issue.
@@ -102,7 +119,11 @@ jobs:
if: github.event_name == 'issues'
run: sleep 60
# Skip everything below for a needs-info reply that was already consumed
# by an earlier queued run (steps.claim.go == false). Non-comment events
# (issues.opened, dispatch) never set claim, so they always proceed.
- name: Checkout tooling
if: github.event_name != 'issue_comment' || steps.claim.outputs.go == 'true'
uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5
with:
fetch-depth: 1
@@ -113,6 +134,7 @@ jobs:
sparse-checkout-cone-mode: false
- name: Build context bundle
if: github.event_name != 'issue_comment' || steps.claim.outputs.go == 'true'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
ISSUE_NUMBER: ${{ github.event.issue.number || inputs.issue }}
@@ -121,6 +143,7 @@ jobs:
- name: Classify
id: classify
if: github.event_name != 'issue_comment' || steps.claim.outputs.go == 'true'
continue-on-error: true
uses: anthropics/claude-code-action@44423bdec74b97d67543eb16c110546762c110b2 # v1
with:
@@ -142,15 +165,23 @@ jobs:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Apply verdict
if: github.event_name != 'issue_comment' || steps.claim.outputs.go == 'true'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
ISSUE: ${{ github.event.issue.number || inputs.issue }}
REPO: ${{ github.repository }}
EVENT_NAME: ${{ github.event_name }}
run: |
set -euo pipefail
F=/tmp/ai-triage/verdict.json
if [ ! -s "$F" ] || ! jq -e . "$F" >/dev/null 2>&1; then
echo "::warning::no usable verdict produced, leaving issue untouched"
# A reporter reply consumed ai:needs-info in the claim step above.
# With no verdict we would otherwise leave the issue with the flag
# gone, so the next reply could never re-trigger — restore it.
if [ "${EVENT_NAME:-}" = "issue_comment" ]; then
gh issue edit "$ISSUE" --repo "$REPO" --add-label ai:needs-info >/dev/null 2>&1 || true
fi
exit 0
fi
@@ -258,17 +289,21 @@ jobs:
run: |
set -euo pipefail
NOW=$(date -u +%s)
gh issue list --repo "$REPO" --state open --limit 100 \
--json number,createdAt,author,labels > /tmp/catchup.json
# Untriaged = no ai:* / ai-triage / no-ai label, not the maintainer's
# own issue, and older than 2h (so a just-opened issue whose tier-1
# run is still in flight is not double-dispatched). Cap 5 per day.
# Filter untriaged issues SERVER-SIDE so the total open-issue count is
# irrelevant (a plain --limit would silently drop everything past the
# cap, and gh lists newest-first). The search excludes every ai:* /
# ai-triage / no-ai label, so what comes back is already the candidate
# set; newest 50 is far more than the daily cap of 5.
gh issue list --repo "$REPO" --limit 50 \
--search 'is:open sort:created-desc -label:ai-triage -label:"ai:classified" -label:"ai:needs-info" -label:"ai:needs-human" -label:"ai:plan-pending" -label:"ai:approved" -label:"ai:fixed" -label:"ai:upstream" -label:no-ai' \
--json number,createdAt,author > /tmp/catchup.json
# Not the maintainer's own issue, and older than 2h (so a just-opened
# issue whose tier-1 run is still in flight is not double-dispatched).
# Cap 5 per day.
jq -r --argjson now "$NOW" '
.[]
| select(.author.login != "alexbelgium")
| select((.createdAt | fromdateiso8601) < ($now - 7200))
| select([.labels[].name]
| any(. == "no-ai" or . == "ai-triage" or startswith("ai:")) | not)
| .number' /tmp/catchup.json | head -n 5 > /tmp/todo.txt
COUNT=$(grep -c . /tmp/todo.txt || true)

View File

@@ -103,4 +103,15 @@ jobs:
GH_TOKEN: ${{ secrets.AI_PR_TOKEN }}
REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
run: bash .github/scripts/ai_guard_paths.sh
run: |
set -euo pipefail
# Run the guard from the trusted default-branch copy, never the in-tree
# copy on the PR branch the model just pushed to.
DEFAULT=$(gh api "repos/${REPO}" --jq '.default_branch' 2>/dev/null || echo master)
if git fetch --depth=1 origin "$DEFAULT" >/dev/null 2>&1 \
&& git cat-file -e "FETCH_HEAD:.github/scripts/ai_guard_paths.sh" 2>/dev/null; then
git show "FETCH_HEAD:.github/scripts/ai_guard_paths.sh" | bash
else
echo "::warning::trusted guard copy unavailable, using in-tree copy"
bash .github/scripts/ai_guard_paths.sh
fi