Files
hassio-addons/.github/scripts/ai_triage_context.sh
Alexandre a04d818479 fix(ci): stop tier 1 wasting its turn budget; escalate max-turns after one retry (#2951)
* fix(ci): stop tier 1 wasting its turn budget; escalate max-turns after one retry

Now that classification actually runs, the 12-turn budget got its first real
exercise — and #2949 died on it. The budget was never the problem; how it was
spent was. Turn-by-turn from that run: 3 turns retrying Bash (not in
allowedTools, and failing against the bubblewrap sandbox that
allowed_non_write_users switches on), 6 hunting .templates/ha_entrypoint.sh and
ha_automodules.sh which are not in the sparse checkout, leaving 3 for the issue.

Fixed at the cause rather than by raising the cap, which stays at 12:

* .templates is now checked out. Most add-ons are thin wrappers around those
  shared scripts, so a large share of reports can only be explained by reading
  them — this makes triage more accurate, not merely faster. 184K, 25 files.
  It has to be added in TWO places: ai_triage_context.sh calls
  `git sparse-checkout set`, which REPLACES the list, so omitting it there
  would silently undo the workflow's checkout at exactly the wrong moment.

* The prompt now states the environment up front: three tools, no Bash, and
  precisely which paths exist on disk. The model cannot discover these cheaply
  — every probe costs a turn it then does not have for the analysis.

Separately, a max-turns death is NOT a workflow fault, but GATE 1 treated every
action failure as systemic and never escalated. So #2949 failed red, stayed
unlabelled, and the catch-up re-dispatched it daily forever — taking the first
of only five slots each time, since it sorts newest-first. It is now handled
like GATE 2: one retry, then ai:needs-human. Detected from the action's
execution_file, which is written even on failure. Warning rather than error,
because a red run per day for a per-issue condition is alarm fatigue, and the
outcome is recorded durably on the issue itself.

The two escalation sites are now one shared function, so they cannot drift.

Re-tested all 15 paths: max-turns across the three events, genuine action
failure with and without an execution file, and the full existing sweep.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(ci): make the max-turns probe fail closed on an unexpected file shape

Copilot: hit_max_turns scanned with `.[]?` and no root-type check. jq's `.[]?`
iterates the VALUES of an object, so if the action ever changed the execution
file's shape, {"result":{"subtype":"error_max_turns"}} would have matched —
downgrading a genuine workflow failure from a red run to a warning. That is the
silent-failure class this workflow exists to remove, arriving through the door
I had just built.

Reproduced: with the old filter that object matched; with `(type == "array")`
prepended it does not. Anything that is not the array we expect now falls
through to the loud path.

Verified: the real array shape is still detected and still escalates on the
second look; object-root, nested-object and non-JSON execution files all exit 1
red instead of being swallowed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(ci): fail loudly when escalation doesn't land; don't escalate a manual first look

Two CodeRabbit findings, both reproduced before accepting.

1. escalate_to_human suppressed `gh issue edit` with `|| true`, so it returned
   success even when ai:needs-human never landed. Both callers then exited 0
   reporting a hand-off that had not happened — and, having no label, the issue
   went straight back into the retry rotation the escalation existed to remove.
   The edit now propagates its status and callers exit 1 with an explicit error.
   `gh label create` stays best effort; the edit fails on its own if the label
   is genuinely missing. Verified that removing a label an issue does not carry
   is a no-op, so this cannot fail spuriously.

2. EVENT_NAME was doing duty as an attempt counter, but workflow_dispatch is
   BOTH the daily catch-up retry and the maintainer's manual re-triage — so a
   hand-dispatched FIRST attempt was escalated immediately.

   Rather than the suggested explicit retry state, the two are already
   distinguishable: the catch-up dispatches with GITHUB_TOKEN and arrives as
   github-actions[bot], a manual run as the maintainer. Confirmed against run
   metadata (catch-up 2026-08-10 = github-actions[bot]; manual 2026-07-27 =
   alexbelgium). is_automated_retry() keys on both, which makes "one retry then
   a human" literally true without new persistent state: a manual attempt that
   fails leaves the issue unlabelled, so the catch-up still gets its go.

Re-tested 15 paths including a stubbed `gh` failure at the escalation site.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(ci): raise max-turns to 25; no max-turns path may end in a silent green run

Three changes, one requested and two from an independent Codex review.

* --max-turns 12 -> 25, per the maintainer's updated call. The prompt preamble
  and comments were carrying the old number and are updated with it. The
  upfront optimisation stays: the earlier waste was 3 turns retrying an
  unavailable Bash and 6 hunting files outside the sparse checkout, and a
  bigger budget should buy analysis rather than more of that.

* Codex objected that the max-turns branch reintroduced the very failure class
  this workflow exists to prevent. It was right. On the SECOND look the outcome
  is durable (ai:needs-human), but on a FIRST attempt nothing was recorded
  anywhere except an annotation, so exiting 0 was a green run over triage that
  silently did not happen. Now the only exit 0 is the one where the escalation
  label actually landed; every other max-turns path is red. My "alarm fatigue"
  argument was overstated: escalation ends the rotation, so this costs at most
  one red run per problem issue, not one per day.

* Codex also flagged inferring the retry from github.actor as brittle — a
  re-run, a PAT- or App-issued dispatch, or a different maintainer all change
  it, and the false NEGATIVE (an automated retry never recognised as one, so it
  retries forever) is the dangerous direction. Replaced with an explicit
  `source` dispatch input that only the catch-up sets. Unknown provenance is
  now safe by construction because that path ends red rather than green.

Re-tested: max-turns across first look / manual dispatch / catch-up retry /
catch-up-with-failing-label / issue_comment, plus the full existing sweep.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(ci): correct two triage comments the recent logic changes left stale

Comments only — no behaviour change, confirmed by diffing out comment lines
(nothing else moved) and re-running the behavioural suite to identical results.

* The prompt preamble still said "the turn budget is 12" and computed
  "leaving 3 for the actual issue" off it. The budget is 25 now. Reworded to
  keep the #2949 evidence, which is still true as history (3 turns retrying
  Bash, 6 hunting files outside the sparse checkout), while stating the
  current budget and why it is not licence to probe more.

* GATE 2 still said "A workflow_dispatch is the catch-up or a manual
  re-triage, i.e. the second look". That stopped being true when escalation
  moved to is_automated_retry(): only source=catchup counts as the second
  attempt, and a manual dispatch is a first look that deliberately does not
  escalate, leaving the issue unlabelled so the catch-up still gets its go.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 12:55:32 +02:00

125 lines
4.6 KiB
Bash
Executable File

#!/usr/bin/env bash
# Destination: .github/scripts/ai_triage_context.sh
#
# Builds /tmp/ai-triage/context.md so Claude does not have to explore a
# 100-addon, 34k-commit monorepo to answer one question. Everything the
# model needs is assembled here by cheap shell instead of by expensive turns.
#
# Env: GH_TOKEN, ISSUE_NUMBER, REPO
set -euo pipefail
OUT=/tmp/ai-triage
mkdir -p "$OUT"
CTX="$OUT/context.md"
: > "$CTX"
gh issue view "$ISSUE_NUMBER" --repo "$REPO" \
--json number,title,body,author,labels,createdAt,comments > "$OUT/issue.json"
TITLE=$(jq -r '.title' "$OUT/issue.json")
# ---------------------------------------------------------------- addon slug
# Titles follow "🐛 [Immich Frame] ENV_VARS arent being picked up".
RAW=$(sed -n 's/.*\[\([^]]*\)\].*/\1/p' <<<"$TITLE" | head -n1)
ADDON=""
if [ -n "$RAW" ]; then
CAND=$(tr '[:upper:] ' '[:lower:]_' <<<"$RAW")
# Directory list without checking out any of them.
git ls-tree -d --name-only HEAD > "$OUT/dirs.txt"
for guess in "$CAND" "${CAND//_/-}" "${CAND//_/.}"; do
if grep -qxF "$guess" "$OUT/dirs.txt"; then ADDON="$guess"; break; fi
done
# Separator-insensitive exact match: a title like "[Calibre-web]" (hyphen)
# against a directory named calibre_web (underscore) matches neither exact
# guess above, and would otherwise fall through to the substring fallback
# below, which picks the shorter "calibre" instead — the wrong add-on.
# Stripping -, _, . from both sides before comparing catches this case.
if [ -z "$ADDON" ]; then
CAND_STRIPPED=$(tr -d '_.-' <<<"$CAND")
while IFS= read -r dir; do
if [ "$(tr -d '_.-' <<<"$dir")" = "$CAND_STRIPPED" ]; then ADDON="$dir"; break; fi
done < "$OUT/dirs.txt"
fi
# Last resort: longest directory name contained in the candidate.
if [ -z "$ADDON" ]; then
ADDON=$(awk -v c="$CAND" 'length($0)>2 && index(c,$0){print length($0)"\t"$0}' \
"$OUT/dirs.txt" | sort -rn | head -n1 | cut -f2)
fi
fi
{
echo "# Issue #${ISSUE_NUMBER}"
echo
echo "Repo: ${REPO}"
echo "Addon resolved from title: ${ADDON:-UNRESOLVED}"
echo
echo "## Title"
echo "$TITLE"
echo
echo "## Author"
jq -r '.author.login' "$OUT/issue.json"
echo
echo "## Body"
echo '```'
jq -r '.body // "(empty)"' "$OUT/issue.json"
echo '```'
echo
echo "## Existing comments (in order)"
jq -r '.comments[]? | "### @\(.author.login)\n\(.body)\n"' "$OUT/issue.json"
echo
echo "## Existing labels"
jq -r '[.labels[]?.name] | join(", ")' "$OUT/issue.json"
} >> "$CTX"
# ------------------------------------------------------------- addon sources
if [ -n "$ADDON" ]; then
{
echo
echo "## Addon files: ${ADDON}/"
# `set` REPLACES the checkout list, so .templates has to be repeated here
# or the workflow's sparse-checkout of it is silently undone at this point
# — which is exactly the state that starved #2949 of its turn budget.
if ! git sparse-checkout set --no-cone .github/prompts .github/scripts .templates "$ADDON" 2>&1; then
# Swallowing this used to leave ADDON resolved with no files behind it,
# so the classifier could still reach high confidence off the addon
# name alone. Say so explicitly, in the same word Rule 2 already keys
# its low-confidence check on.
echo
echo "**Could not check out this add-on's source. Treat as UNRESOLVED for confidence purposes.**"
else
for f in config.yaml config.json Dockerfile CHANGELOG.md DOCS.md README.md; do
[ -f "$ADDON/$f" ] || continue
echo
echo "### ${ADDON}/${f}"
echo '```'
head -c 8000 "$ADDON/$f"
echo '```'
done
echo
echo "## Recent commits touching ${ADDON}/"
git log -n 15 --date=short --pretty='- %ad %h %s' -- "$ADDON" 2>/dev/null || true
fi
} >> "$CTX"
fi
# -------------------------------------------------------- possible duplicates
{
echo
echo "## Similar existing issues (candidate duplicates)"
KEYWORDS=$(tr -cs '[:alnum:]' ' ' <<<"$TITLE" \
| tr '[:upper:]' '[:lower:]' \
| tr ' ' '\n' | awk 'length($0)>3' | head -n6 | paste -sd' ')
# Excludes the issue being triaged: if it's already indexed by GitHub search
# by the time this runs, keyword overlap with its own title would otherwise
# list it as a "candidate duplicate" of itself.
gh search issues --repo "$REPO" --limit 15 \
--json number,title,state,url -- "$KEYWORDS" 2>/dev/null \
| jq -r --argjson self "$ISSUE_NUMBER" \
'.[] | select(.number != $self) | "- #\(.number) [\(.state)] \(.title)"' \
|| echo "(search unavailable)"
} >> "$CTX"
echo "context bundle: $(wc -c < "$CTX") bytes, addon=${ADDON:-none}"