marek-miklewicz/alt-mysql
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix worker.sh lock permanently stalling after a crash

Browse Source 
The mkdir-based worker lock had no staleness detection at all, unlike
BackupQueueService::withQueueLock()'s 120s override for its own queue
lock. If the root cron-driven worker was ever killed ungracefully (OOM,
kill -9, reboot mid-job), the lock directory was left behind forever -
every subsequent cron tick would silently exit without processing any
job, so backups/restores would queue forever with no visible error.

Track the owning PID inside the lock directory and reclaim the lock
when that PID is no longer running, or when the lock has no PID file
and is older than WORKER_LOCK_STALE_SECONDS (default 300s). Log to
stderr on both the "held by a live worker" and "reclaiming stale lock"
paths instead of exiting silently either way. cleanup() now uses rm -rf
since the lock directory holds a pid file, not just an empty marker.
This commit is contained in:
Marek Miklewicz
2026-07-04 21:56:52 +02:00
parent 60ae1e1697
commit bbc1afb5e1
2 changed files with 113 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/worker.sh
+42 -3
View File
@@ -1,7 +1,7 @@
#!/bin/bash
set -euo pipefail
PLUGIN_DIR="$(cd "$(dirname "$0")/.." && pwd)"
PLUGIN_DIR="${PLUGIN_DIR:-$(cd "$(dirname "$0")/.." && pwd)}"
DATA_DIR="$PLUGIN_DIR/data"
JOB_DIR="$DATA_DIR/jobs/pending"
PROCESSING_DIR="$DATA_DIR/jobs/processing"
@@ -10,15 +10,54 @@ WORKER_LOCK_DIR="$DATA_DIR/worker.lock"
DB_LOCK_DIR="$DATA_DIR/lock"
PID_DIR="$DATA_DIR/pids"
CANCEL_DIR="$DATA_DIR/cancel"
WORKER_LOCK_STALE_SECONDS="${WORKER_LOCK_STALE_SECONDS:-300}"
mkdir -p "$JOB_DIR" "$PROCESSING_DIR" "$DONE_DIR" "$PID_DIR" "$CANCEL_DIR" "$DB_LOCK_DIR"
if ! mkdir "$WORKER_LOCK_DIR" 2>/dev/null; then
acquire_worker_lock() {
if mkdir "$WORKER_LOCK_DIR" 2>/dev/null; then
printf '%s' "$$" > "$WORKER_LOCK_DIR/pid" 2>/dev/null || true
return 0
fi
local existing_pid="" lock_mtime=0 now lock_age=0
if [ -r "$WORKER_LOCK_DIR/pid" ]; then
existing_pid="$(cat "$WORKER_LOCK_DIR/pid" 2>/dev/null || true)"
fi
if [ -n "$existing_pid" ]; then
if kill -0 "$existing_pid" 2>/dev/null; then
echo "alt-mysql worker: lock held by running pid $existing_pid, exiting" >&2
return 1
fi
else
lock_mtime="$(stat -c '%Y' "$WORKER_LOCK_DIR" 2>/dev/null || stat -f '%m' "$WORKER_LOCK_DIR" 2>/dev/null || echo 0)"
now="$(date +%s)"
if [ "$lock_mtime" -gt 0 ]; then
lock_age=$((now - lock_mtime))
fi
if [ "$lock_age" -lt "$WORKER_LOCK_STALE_SECONDS" ]; then
echo "alt-mysql worker: lock present without a live pid but younger than ${WORKER_LOCK_STALE_SECONDS}s (age ${lock_age}s), exiting" >&2
return 1
fi
fi
echo "alt-mysql worker: reclaiming stale lock (pid=${existing_pid:-none}, age=${lock_age}s)" >&2
rm -rf "$WORKER_LOCK_DIR"
if mkdir "$WORKER_LOCK_DIR" 2>/dev/null; then
printf '%s' "$$" > "$WORKER_LOCK_DIR/pid" 2>/dev/null || true
return 0
fi
return 1
}
if ! acquire_worker_lock; then
exit 0
fi
cleanup() {
rmdir "$WORKER_LOCK_DIR" 2>/dev/null || true
rm -rf "$WORKER_LOCK_DIR" 2>/dev/null || true
}
trap cleanup EXIT