chore: add two-tier AI issue triage (workflows, script, prompts)

Install tier-1 classifier (on_issues_ai_triage.yaml) and tier-2 fix sweep
(weekly_ai_fix.yaml) plus their helper script and prompts.

Pre-merge fixes from verification:
- Wait-for-ping sleep 150s -> 60s. on_issues_ping_submitter completes in
  6-11s of job time across the last 10 runs; 60s covers runner-queue skew
  with margin.
- Rule 0 rewritten to match the real ownership signal: ping_submitter posts
  a github-actions[bot] comment with a stable marker
  `<!-- addon-submitter-ping:<addon> -->`. Rule 0 now keys off that literal
  marker instead of fuzzy prose, and guards against @<user> == alexbelgium.
- Silence one intentional shellcheck SC2016 (literal Markdown backticks) so
  actionlint runs clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
alexbelgium
2026-07-23 09:59:54 +02:00
parent 970b73a2be
commit 954f1db7eb
5 changed files with 522 additions and 0 deletions

99
.github/prompts/issue-classify.md vendored Normal file
View File

@@ -0,0 +1,99 @@
# Issue classifier — tier 1
You are triaging a new issue on `alexbelgium/hassio-addons`, a monorepo of
100+ Home Assistant add-ons. Each add-on is a thin wrapper (Dockerfile,
`run.sh`, s6 services, nginx config, `config.yaml`) around an upstream
application that Alex does not maintain.
Your entire output is one JSON object written to `/tmp/ai-triage/verdict.json`.
You do not comment, label, or edit anything.
## Rule 0 — ownership short-circuit
Read the existing comments in the context bundle first. The
`on_issues_ping_submitter` workflow signals ownership by posting a **comment**
(authored by `github-actions[bot]`) that pings the add-on's original submitter.
Its exact, machine-stable format is:
```
<!-- addon-submitter-ping:<addon> -->
Heads up @<user>: this issue appears to mention `<addon>`.
```
Match it on the literal marker `<!-- addon-submitter-ping:` — that string is
the reliable signal; do not infer ownership from prose. If a comment carrying
that marker is present **and** the pinged `@<user>` is not `alexbelgium`, stop
immediately and emit:
```json
{"verdict": "owned", "confidence": "high"}
```
Do not spend turns on anything else. (The workflow only ever pings a mapped
submitter, so in practice `@<user>` is always someone other than `alexbelgium`;
the check is a guard, not a common case.)
## Rule 1 — pick exactly one verdict
| verdict | when |
|---|---|
| `duplicate` | An existing open or closed issue reports the same thing. Set `duplicate_of`. |
| `needs-info` | You cannot tell what is wrong without the add-on version, HA version, architecture, config, or the actual log output. |
| `question` | A usage question answerable from `DOCS.md`, the wiki, or the add-on config. Not a defect. |
| `upstream-bug` | The fault is in the upstream application or its image, not in this repo's wrapper. |
| `addon-bug` | The fault is in something this repo owns: the Dockerfile, `run.sh`, s6 service files, nginx config, `config.yaml` schema, or an option that is not being passed through. |
| `feature-request` | New capability, new add-on, new option. |
**The `upstream-bug` / `addon-bug` split is the one that matters.** Only
`addon-bug` triggers the expensive fix pass. Getting it wrong means the bot
opens a pull request against code that does not exist in this repository.
Test it explicitly: name the file in this repo you would have to change. If you
cannot name one, it is not `addon-bug`.
## Rule 2 — confidence is a real signal
Set `confidence` to `low` whenever any of these hold:
- The add-on could not be resolved from the title (`UNRESOLVED` in the bundle).
- The issue mixes several unrelated problems.
- You are choosing between `upstream-bug` and `addon-bug` and could argue both.
- The report is in a language you are not confident reading.
`low` confidence suppresses the comment entirely and flags a human instead.
Prefer that over a fluent guess. A wrong answer on a support issue costs Alex
more trust than no answer.
## Rule 3 — writing the comment
Only `duplicate`, `needs-info`, and `question` get a comment. The other verdicts
are labelled silently and handled later.
- **duplicate** — one line, link the other issue, no explanation.
- **needs-info** — ask only for what is *strictly* required to proceed, as a
short checklist. Never more than four items. Say where to find each one
(e.g. the add-on log tab, the Configuration tab). Do not ask for anything
already present in the issue body.
- **question** — answer only from files in the context bundle, and quote the
file path you took it from. If the bundle does not contain the answer, this
is `needs-info`, not `question`. Never invent option names.
Never close an issue. Never promise a timeline. Never say a fix is coming.
## Output schema
```json
{
"verdict": "owned|duplicate|needs-info|question|upstream-bug|addon-bug|feature-request",
"addon": "birdnet-go",
"confidence": "high|medium|low",
"duplicate_of": 1234,
"labels": ["bug"],
"root_cause_hint": "one sentence for the tier-2 pass, or empty",
"comment": "markdown, or empty string"
}
```
`labels` should contain at most two, from the repo's existing set. Do not
invent new label names; the workflow adds `ai-triage` and `ai:classified`
on its own.

74
.github/prompts/issue-fix.md vendored Normal file
View File

@@ -0,0 +1,74 @@
# Issue fix sweep — tier 2
You are working through a batch of confirmed add-on bugs on
`alexbelgium/hassio-addons`. Each add-on is a thin wrapper around an upstream
application. You own the wrapper. You do not own the upstream app.
Read `/tmp/ai-fix/batch.json`. Work add-on by add-on, not issue by issue —
grouping is the point of the batch.
## Hard limits
These are not guidelines. A workflow step enforces them after you finish, and
anything that violates them gets blocked and flagged.
1. **Never modify `.github/` or `.templates/`.** Those are inherited by every
add-on in the repo. A change there is a 100-add-on incident, not a fix.
2. **Never touch the `version` or `upstream` fields in `config.yaml`.** The
`addons_updater` job owns those. Editing them causes merge conflicts you
will not be around to resolve.
3. **One add-on per branch, one branch per pull request.** Branch name
`ai-fix/<addon>-<issue-number>`.
4. **Draft pull requests only.** Never merge, never mark ready for review,
never close an issue.
5. If the fix requires changing more than roughly 60 lines, or touching more
than three files, stop. Post the analysis, open no pull request, and say
plainly that the change is too large for an unattended fix.
## Per add-on, do this in order
**1. Read before you write.** The add-on's `CLAUDE.md` if it has one, then
`DOCS.md`, `config.yaml`, `Dockerfile`, and everything under `rootfs/`. Read
`CHANGELOG.md` and `git log` for the last few weeks — a bug that appeared
suddenly usually has a commit behind it, and finding that commit is worth more
than reading the whole tree.
**2. Establish the root cause, and be honest about confidence.** Name the exact
file and line. If you cannot, you have a hypothesis, not a root cause, and you
must label it as such in the comment. Do not dress a guess up as a diagnosis.
Alex has to trust these comments without re-deriving them.
**3. Re-check the upstream/wrapper split.** Tier 1 already made this call, but
it made it cheaply and without reading the source. If the real fault is
upstream, say so, do not open a pull request, and suggest what to file with the
upstream project instead. Reversing tier 1's classification is a correct and
valuable outcome, not a failure.
**4. Fix it.** Match the surrounding style — this repo is bash and Dockerfiles,
and the conventions vary between add-ons. Run `shellcheck` on any shell you
change. Add a `CHANGELOG.md` entry in the add-on's existing format.
**5. Open the draft pull request.** Body must contain: the root cause with file
and line, what the change does, how you verified it (or an explicit statement
that you could not verify it), and `Closes #<n>`.
**6. Comment on the issue.** Root cause, the fix in one or two sentences, and
the pull request link. Plain language — the reader is a Home Assistant user,
not a Go developer. If you found no fix, say what you ruled out and what you
would need to go further. Close with a note that this is automated analysis
pending Alex's review.
## Meta-findings
This is the part a per-issue run cannot do, so do not skip it.
After the batch, look across everything you read. If several issues share a
cause — one base image bump, one s6 change, one upstream release, one bad
option default replicated across add-ons — open a single issue titled
`[meta] <pattern>` describing it, linking the affected issues, and proposing
the systemic fix rather than the individual patches.
Report honestly if the batch produced nothing. A sweep that fixes zero issues
and says so clearly is more useful than one that manufactures three plausible
patches. You will be judged on whether Alex can trust the output without
checking it, not on how many pull requests you opened.

99
.github/scripts/ai_triage_context.sh vendored Executable file
View File

@@ -0,0 +1,99 @@
#!/usr/bin/env bash
# Destination: .github/scripts/ai_triage_context.sh
#
# Builds /tmp/ai-triage/context.md so Claude does not have to explore a
# 100-addon, 34k-commit monorepo to answer one question. Everything the
# model needs is assembled here by cheap shell instead of by expensive turns.
#
# Env: GH_TOKEN, ISSUE_NUMBER, REPO
set -euo pipefail
OUT=/tmp/ai-triage
mkdir -p "$OUT"
CTX="$OUT/context.md"
: > "$CTX"
gh issue view "$ISSUE_NUMBER" --repo "$REPO" \
--json number,title,body,author,labels,createdAt,comments > "$OUT/issue.json"
TITLE=$(jq -r '.title' "$OUT/issue.json")
# ---------------------------------------------------------------- addon slug
# Titles follow "🐛 [Immich Frame] ENV_VARS arent being picked up".
RAW=$(sed -n 's/.*\[\([^]]*\)\].*/\1/p' <<<"$TITLE" | head -n1)
ADDON=""
if [ -n "$RAW" ]; then
CAND=$(tr '[:upper:] ' '[:lower:]_' <<<"$RAW")
# Directory list without checking out any of them.
git ls-tree -d --name-only HEAD > "$OUT/dirs.txt"
for guess in "$CAND" "${CAND//_/-}" "${CAND//_/.}"; do
if grep -qxF "$guess" "$OUT/dirs.txt"; then ADDON="$guess"; break; fi
done
# Last resort: longest directory name contained in the candidate.
if [ -z "$ADDON" ]; then
ADDON=$(awk -v c="$CAND" 'length($0)>2 && index(c,$0){print length($0)"\t"$0}' \
"$OUT/dirs.txt" | sort -rn | head -n1 | cut -f2)
fi
fi
{
echo "# Issue #${ISSUE_NUMBER}"
echo
echo "Repo: ${REPO}"
echo "Addon resolved from title: ${ADDON:-UNRESOLVED}"
echo
echo "## Title"
echo "$TITLE"
echo
echo "## Author"
jq -r '.author.login' "$OUT/issue.json"
echo
echo "## Body"
echo '```'
jq -r '.body // "(empty)"' "$OUT/issue.json"
echo '```'
echo
echo "## Existing comments (in order)"
jq -r '.comments[]? | "### @\(.author.login)\n\(.body)\n"' "$OUT/issue.json"
echo
echo "## Existing labels"
jq -r '[.labels[]?.name] | join(", ")' "$OUT/issue.json"
} >> "$CTX"
# ------------------------------------------------------------- addon sources
if [ -n "$ADDON" ]; then
git sparse-checkout set --no-cone .github/prompts .github/scripts "$ADDON" || true
{
echo
echo "## Addon files: ${ADDON}/"
for f in config.yaml config.json Dockerfile CHANGELOG.md DOCS.md README.md; do
[ -f "$ADDON/$f" ] || continue
echo
echo "### ${ADDON}/${f}"
echo '```'
head -c 8000 "$ADDON/$f"
echo '```'
done
echo
echo "## Recent commits touching ${ADDON}/"
git log -n 15 --date=short --pretty='- %ad %h %s' -- "$ADDON" 2>/dev/null || true
} >> "$CTX"
fi
# -------------------------------------------------------- possible duplicates
{
echo
echo "## Similar existing issues (candidate duplicates)"
KEYWORDS=$(tr -cs '[:alnum:]' ' ' <<<"$TITLE" \
| tr '[:upper:]' '[:lower:]' \
| tr ' ' '\n' | awk 'length($0)>3' | head -n6 | paste -sd' ')
gh search issues --repo "$REPO" --limit 15 \
--json number,title,state,url -- "$KEYWORDS" 2>/dev/null \
| jq -r '.[] | "- #\(.number) [\(.state)] \(.title)"' \
|| echo "(search unavailable)"
} >> "$CTX"
echo "context bundle: $(wc -c < "$CTX") bytes, addon=${ADDON:-none}"

View File

@@ -0,0 +1,136 @@
---
# Destination: .github/workflows/on_issues_ai_triage.yaml
#
# Tier 1. Fires on every new issue, costs cents, finishes in ~2 minutes.
# Classifies, de-duplicates, asks for missing info, answers simple questions,
# and applies the `ai-triage` label that tier 2 (weekly_ai_fix.yaml) picks up.
#
# Required secret: ANTHROPIC_API_KEY (or swap to claude_code_oauth_token)
name: AI issue triage
on:
issues:
types: [opened]
permissions:
contents: read
issues: write
concurrency:
group: ai-triage-${{ github.event.issue.number }}
cancel-in-progress: false
env:
MAINTAINER: alexbelgium
# Leave "true" for the first couple of weeks. The verdict is printed in the
# job log and nothing is written to the issue. Flip when it looks right.
DRY_RUN: "true"
jobs:
classify:
if: >-
github.event.issue.user.login != 'alexbelgium' &&
!contains(github.event.issue.labels.*.name, 'no-ai')
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
# on_issues_ping_submitter.yml has to land first: the classifier reads
# the existing comments and bails out if someone already owns the issue.
# Both workflows fire on the same issues.opened event and race. The
# submitter ping completes in 6-11s of job time across recent runs; 60s
# leaves a generous margin for runner-queue skew between the two jobs.
- name: Wait for ping_submitter
run: sleep 60
- name: Checkout tooling
uses: actions/checkout@v5
with:
fetch-depth: 1
sparse-checkout: |
.github/prompts
.github/scripts
sparse-checkout-cone-mode: false
- name: Build context bundle
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
ISSUE_NUMBER: ${{ github.event.issue.number }}
REPO: ${{ github.repository }}
run: bash .github/scripts/ai_triage_context.sh
- name: Classify
uses: anthropics/claude-code-action@v1
with:
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
prompt: |
Read /tmp/ai-triage/context.md, then follow the instructions in
.github/prompts/issue-classify.md exactly.
Write your verdict as a single JSON object to
/tmp/ai-triage/verdict.json and write nothing else anywhere.
Do NOT comment on or label the issue yourself.
claude_args: |
--model claude-sonnet-5
--effort low
--max-turns 12
--allowedTools "Read,Write,Glob,Grep,Bash(gh issue list:*),Bash(gh search issues:*)"
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Apply verdict
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
ISSUE: ${{ github.event.issue.number }}
REPO: ${{ github.repository }}
run: |
set -euo pipefail
F=/tmp/ai-triage/verdict.json
if [ ! -s "$F" ] || ! jq -e . "$F" >/dev/null 2>&1; then
echo "::warning::no usable verdict produced, leaving issue untouched"
exit 0
fi
echo "--- verdict ---"; jq . "$F"; echo "---------------"
VERDICT=$(jq -r '.verdict // "unknown"' "$F")
CONF=$(jq -r '.confidence // "low"' "$F")
COMMENT=$(jq -r '.comment // ""' "$F")
mapfile -t LABELS < <(jq -r '.labels[]? // empty' "$F")
# Someone already owns this one: ping_submitter did its job.
if [ "$VERDICT" = "owned" ]; then
echo "issue already has an owner, nothing to do"; exit 0
fi
# Low confidence never speaks. It just flags for a human.
if [ "$CONF" = "low" ]; then
LABELS=("ai:needs-human"); COMMENT=""
fi
[ "$VERDICT" = "addon-bug" ] && LABELS+=("ai-triage")
LABELS+=("ai:classified")
if [ "${DRY_RUN:-true}" = "true" ]; then
echo "DRY_RUN: would apply labels: ${LABELS[*]}"
echo "DRY_RUN: would post comment:"; printf '%s\n' "$COMMENT"
exit 0
fi
for l in "${LABELS[@]}"; do
gh label create "$l" --repo "$REPO" --color ededed --force >/dev/null 2>&1 || true
done
gh issue edit "$ISSUE" --repo "$REPO" \
"${LABELS[@]/#/--add-label=}"
if [ -n "$COMMENT" ]; then
{
printf '%s\n\n' "$COMMENT"
printf -- '---\n'
printf '<sub>Automated triage. Not verified by a human yet '
# shellcheck disable=SC2016 # backticks are literal Markdown, not a subshell
printf -- '— @%s will confirm. Add the `no-ai` label to opt out.</sub>\n' "$MAINTAINER"
} > /tmp/ai-triage/comment.md
gh issue comment "$ISSUE" --repo "$REPO" --body-file /tmp/ai-triage/comment.md
fi

114
.github/workflows/weekly_ai_fix.yaml vendored Normal file
View File

@@ -0,0 +1,114 @@
---
# Destination: .github/workflows/weekly_ai_fix.yaml
#
# Tier 2. One Opus 4.8 run at xhigh over the whole batch of `ai-triage` issues,
# grouped by add-on, so it can spot the cross-issue patterns a per-issue run
# never sees ("these four reports are all the same base image bump").
#
# Required secrets:
# ANTHROPIC_API_KEY
# AI_PR_TOKEN — GitHub App token or PAT (repo scope). NOT GITHUB_TOKEN:
# pull requests created with GITHUB_TOKEN do not trigger
# other workflows, so your PR Check Build would never run.
name: AI fix sweep
on:
schedule:
- cron: "0 3 * * SUN"
workflow_dispatch:
inputs:
issue:
description: "Single issue number (blank = all ai-triage issues)"
required: false
limit:
description: "Max issues in the batch"
required: false
default: "8"
permissions:
contents: write
issues: write
pull-requests: write
concurrency:
group: ai-fix-sweep
cancel-in-progress: false
jobs:
sweep:
runs-on: ubuntu-latest
timeout-minutes: 180
steps:
- uses: actions/checkout@v5
with:
fetch-depth: 0
token: ${{ secrets.AI_PR_TOKEN }}
- name: Collect batch
id: batch
env:
GH_TOKEN: ${{ secrets.AI_PR_TOKEN }}
REPO: ${{ github.repository }}
run: |
set -euo pipefail
mkdir -p /tmp/ai-fix
if [ -n "${{ inputs.issue }}" ]; then
gh issue view "${{ inputs.issue }}" --repo "$REPO" \
--json number,title,body,labels,comments \
| jq '[.]' > /tmp/ai-fix/batch.json
else
gh issue list --repo "$REPO" --state open \
--label ai-triage --limit "${{ inputs.limit || '8' }}" \
--json number,title,body,labels,comments > /tmp/ai-fix/batch.json
fi
N=$(jq 'length' /tmp/ai-fix/batch.json)
echo "count=$N" >> "$GITHUB_OUTPUT"
echo "batch size: $N"
- name: Configure git
if: steps.batch.outputs.count != '0'
run: |
git config user.name "claude-ai-fix[bot]"
git config user.email "claude-ai-fix[bot]@users.noreply.github.com"
- name: Analyse and fix
if: steps.batch.outputs.count != '0'
uses: anthropics/claude-code-action@v1
with:
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
prompt: |
The batch of issues to work through is /tmp/ai-fix/batch.json.
Follow .github/prompts/issue-fix.md exactly. Do not deviate from
the path restrictions in that file under any circumstances.
claude_args: |
--model claude-opus-4-8
--effort xhigh
--max-turns 300
--allowedTools "Read,Write,Edit,Glob,Grep,Bash(git:*),Bash(gh:*),Bash(shellcheck:*),Bash(yamllint:*),Bash(docker build:*)"
env:
GH_TOKEN: ${{ secrets.AI_PR_TOKEN }}
# Belt and braces. The prompt forbids these paths; this enforces it.
- name: Guard forbidden paths
if: always() && steps.batch.outputs.count != '0'
env:
GH_TOKEN: ${{ secrets.AI_PR_TOKEN }}
REPO: ${{ github.repository }}
run: |
set -euo pipefail
PRS=$(gh pr list --repo "$REPO" --state open --limit 50 \
--json number,headRefName \
--jq '.[] | select(.headRefName|startswith("ai-fix/")) | .number')
for pr in $PRS; do
BAD=$(gh pr diff "$pr" --repo "$REPO" --name-only \
| grep -E '^(\.github/|\.templates/)' || true)
if [ -n "$BAD" ]; then
echo "::error::PR #$pr touches protected paths:"; echo "$BAD"
gh pr ready "$pr" --repo "$REPO" --undo || true
gh pr edit "$pr" --repo "$REPO" --add-label "ai:blocked"
gh pr comment "$pr" --repo "$REPO" --body \
"Blocked automatically: this PR modifies shared infrastructure (\`.github/\` or \`.templates/\`), which is inherited by every add-on in the repo. Needs manual review before it goes anywhere."
fi
done