fix(birdnet-pi): stop the restart loop and ffmpeg spin when the audio source is unreachable (#3152) (#3153)

* fix(birdnet-pi): stop the restart loop and ffmpeg spin when the audio source is unreachable (#3152)

Two independent loops react badly to an unreachable RTSP source or a
missing soundcard.

rootfs/custom-services.d/30-monitoring.sh: check_analyzing_now() treats an
unchanged analyzing_now.txt as a hung analyzer and runs restart_services.sh.
Upstream only rewrites that file in birdnet_analysis.py's process_file(), i.e.
once per recording picked up, so with no audio source it is static by design.
With the monitoring loop at sleep 61 and SAME_FILE_THRESHOLD=2 (the counter is
reset after firing) that is a full service restart every ~122 s forever, and
every restart leaves the previous daily_plot.py and recording loop alive next
to the new one - which is where the RAM goes. Skip the hang verdict when
nothing is queued in the ingest dir (idle, not stuck - restarting cannot bring
the source back) and alert instead, and never restart more than once per
30 minutes on the remaining path.

rootfs/etc/cont-init.d/81-modifications.sh: upstream loop_ffmpeg() retries a
failed ffmpeg after a fixed "sleep 1", so an unreachable host means roughly one
ffmpeg start per second at 25-37 % CPU each. Patch the retry delay to grow
1 s -> 64 s, resetting once ffmpeg has run for more than 2 minutes so a healthy
stream that hiccups is still picked up straight away. The patch is applied in
the add-on's existing upstream-patch layer, guarded by a grep and validated
with bash -n before it replaces the script.

version in config.yaml is deliberately left at 2026.08.15 - see the PR body.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: address CodeRabbit review

Measure ffmpeg runtime excluding the retry sleep, and clarify the
changelog wording for the restart-rate-limit path.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(birdnet-pi): clear leftover processes when restarting the services (#3152)

The systemctl replacement has no cgroups: stop kills the main PID plus the
descendants it can still trace by PPid, and returns early when the main PID is
already dead, so orphaned loop_ffmpeg shells, ffmpeg/arecord recorders or a
stray daily_plot.py survive and each restart adds another copy next to them.
ExecStartPre cannot be used for the cleanup: systemctl3 runs it before its
"already running" check, so a start on a running unit would kill it and then
refuse to start.

Inject one "sudo pkill -f" into restart_services.sh right after it stops the
recorder and before its restart loop, matching only daily_plot.py --daemon,
/usr/local/bin/birdnet_recording.sh and ffmpeg/arecord writing to StreamData.
Bump version so Supervisor offers the rebuild.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(birdnet-pi): keep the silent-source condition flagged until recording resumes

Resetting same_file_counter in the new no-recordings and rate-limit branches
made the next monitoring pass report success while analyzing_now.txt was still
unchanged, so the main loop sent a false "back to normal" notification after
every (30-min throttled) alert. Leave the counter alone there; it resets as soon
as the file changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

birdnet-pi/CHANGELOG.md

@@ -1,3 +1,7 @@
+## 2026.10.26 (26-10-2026)
+- Fix: restarting the services no longer leaves a previous `daily_plot.py` or recording loop running next to the new one. The `systemctl` replacement has no cgroups and skips the children of a main process that already died, so `restart_services.sh` now clears leftover `daily_plot.py`, `birdnet_recording.sh`, and `ffmpeg`/`arecord` recorders writing to `StreamData` once the recorder is stopped
+- Fix: an unreachable audio source (RTSP host switched off, missing soundcard) no longer causes a restart loop. The monitoring service treated a static `analyzing_now.txt` as a hung analyzer and ran `restart_services.sh` every two minutes, which stacked up a second `daily_plot.py` and a second recording loop each time, so RAM kept growing. It now reports the silent source instead of restarting when there is nothing queued to analyze, and limits that static `analyzing_now.txt` restart path to once every 30 minutes
+- Fix: the recording retry loop no longer restarts `ffmpeg` once per second while the audio source is unreachable (100%+ CPU). The retry delay now grows from 1s to 64s, and resets as soon as `ffmpeg` has run for more than two minutes, so a healthy stream that hiccups is still picked up straight away
 - Local disks: `localdisks` can now mount a single folder of a disk with `disk/folder` (e.g. `sda1/public` or `NAS/public`, mounted at `/mnt/<disk>/<folder>`). Existing values are unchanged.
 - Migrate legacy add-on configuration map names to current app configuration terminology.
 ## 2026.08.15 (15-08-2026)

birdnet-pi/config.yaml

@@ -116,5 +116,5 @@ tmpfs: true
 udev: true
 url: https://github.com/alexbelgium/hassio-addons/tree/master/birdnet-pi
 usb: true
-version: 2026.08.15
+version: 2026.10.26
 video: true

birdnet-pi/rootfs/custom-services.d/30-monitoring.sh

@@ -60,6 +60,11 @@ DISK_USAGE_THRESHOLD=95
 # "Analyzing" file check variables
 same_file_counter=0
 SAME_FILE_THRESHOLD=2
+# Minimum delay between two automatic service restarts (seconds). Restarting
+# every couple of minutes has never recovered a stuck analyzer, and each restart
+# leaves processes of the previous run behind.
+RESTART_MIN_INTERVAL=1800
+last_restart_time=0
 if [[ -f "$ANALYZING_NOW_FILE" ]]; then
     analyzing_now=$(< "$ANALYZING_NOW_FILE")
 else
@@ -157,9 +162,32 @@ check_analyzing_now() {
     fi
 
     if ((same_file_counter >= SAME_FILE_THRESHOLD)); then
+        # birdnet_analysis only rewrites 'analyzing_now' when it picks up a new
+        # recording, so the file also stops changing when there is nothing to
+        # analyze at all - an unreachable RTSP source, a silent or missing
+        # soundcard. With an empty queue the analyzer is idle, not stuck, and
+        # restarting the services cannot bring the audio source back: it only
+        # leaves a second daily_plot.py and a second recording loop behind every
+        # couple of minutes. Report it, but do not restart.
+        local wav_waiting
+        wav_waiting=$(find -L "$INGEST_DIR" -maxdepth 1 -name '*.wav' | wc -l)
+        if ((wav_waiting == 0)); then
+            log_red "$(date) INFO: no recordings to analyze - the audio source looks silent or unreachable. Not restarting."
+            apprisealert "No new recordings: the audio source looks silent or unreachable"
+            return 1
+        fi
+
+        local now
+        now=$(date +%s)
+        if ((now - last_restart_time < RESTART_MIN_INTERVAL)); then
+            log_yellow "$(date) INFO: 'analyzing_now' still unchanged, but services were already restarted $(((now - last_restart_time) / 60)) minutes ago. Waiting."
+            return 1
+        fi
+
         log_red "$(date) INFO: 'analyzing_now' file unchanged for $SAME_FILE_THRESHOLD iterations."
         apprisealert "No change in analyzing_now for ${SAME_FILE_THRESHOLD} iterations"
         "$HOME/BirdNET-Pi/scripts/restart_services.sh"
+        last_restart_time=$now
         same_file_counter=0
         return 1
     else

birdnet-pi/rootfs/etc/cont-init.d/81-modifications.sh

@@ -89,6 +89,43 @@ fi
 echo "... allow pulseaudio as root as backup"
 sed -i 's#pulseaudio --start#pulseaudio --start || pulseaudio --system#g' "$HOME"/BirdNET-Pi/scripts/birdnet_recording.sh
 
+# Back off the recording retry loop
+# loop_ffmpeg() retries a failed ffmpeg after a fixed "sleep 1". When the audio
+# source is unreachable ffmpeg exits immediately ("No route to host"), so that is
+# one ffmpeg start per second for as long as the source is down - enough to keep
+# a 2-core host at 100% CPU. Grow the delay 1s -> 64s instead, and reset it once
+# ffmpeg has managed to run for more than 2 minutes, so a healthy stream that
+# hiccups is still picked up straight away. The sleep only runs when ffmpeg
+# fails, so a working source is unaffected.
+RECORDING_SCRIPT="$HOME/BirdNET-Pi/scripts/birdnet_recording.sh"
+if grep -q '^[[:space:]]*sleep 1$' "$RECORDING_SCRIPT"; then
+    echo "... backing off the recording retry loop"
+    sed 's|^\([[:space:]]*\)sleep 1$|\1RETRY_DELAY=$(( SECONDS - ${RETRY_LAST:-0} > 120 ? 1 : ${RETRY_DELAY:-1} ))\n\1sleep "$RETRY_DELAY"\n\1RETRY_LAST=$SECONDS\n\1RETRY_DELAY=$(( RETRY_DELAY < 64 ? RETRY_DELAY * 2 : 64 ))|' \
+        "$RECORDING_SCRIPT" > "$RECORDING_SCRIPT.backoff"
+    # Only keep the patched script if it still parses, a broken recording script
+    # would take down recording altogether
+    if bash -n "$RECORDING_SCRIPT.backoff"; then
+        chown --reference="$RECORDING_SCRIPT" "$RECORDING_SCRIPT.backoff"
+        chmod --reference="$RECORDING_SCRIPT" "$RECORDING_SCRIPT.backoff"
+        mv "$RECORDING_SCRIPT.backoff" "$RECORDING_SCRIPT"
+    else
+        rm -f "$RECORDING_SCRIPT.backoff"
+        bashio::log.warning "Could not add a backoff to the recording retry loop, leaving it unchanged"
+    fi
+fi
+
+# Clear leftovers of the previous run when restarting the services
+# The systemctl replacement has no cgroups: it only stops the processes it can
+# still trace back to a living main process. If that one died first, its
+# children (loop_ffmpeg shells, ffmpeg, arecord) survive, and every restart adds
+# a second daily_plot.py or recording loop next to them. The [x] brackets keep
+# the pattern from matching the sudo/pkill command line itself.
+RESTART_SCRIPT="$HOME/BirdNET-Pi/scripts/restart_services.sh"
+if ! grep -qF '[d]aily_plot' "$RESTART_SCRIPT"; then
+    echo "... clearing leftover processes on service restarts"
+    sed -i '/systemctl stop birdnet_recording.service/a sudo pkill -f "[d]aily_plot.py --daemon|/usr/local/bin/[b]irdnet_recording.sh|[f]fmpeg .*StreamData/.*wav|[a]record .*StreamData/.*wav" || true' "$RESTART_SCRIPT"
+fi
+
 # Send services log to container logs
 echo "... redirecting services logs to container logs"
 while IFS= read -r file; do