Add design spec and implementation plan for restore-without-native-staging
Ignore .superpowers/ (subagent-driven-development's scratch progress ledger).
This commit is contained in:
Marek Miklewicz
@@ -0,0 +1,996 @@
|
||||
# Restore Without Native Staging Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Restore external (non-plugin-origin) DirectAdmin account backups directly into `alt-mariadb` by parsing DirectAdmin's native `backup/<dbname>.conf` + `backup/<dbname>.sql` file pairs straight out of the archive, without ever staging data in the CustomBuild-managed native MariaDB.
|
||||
|
||||
**Architecture:** A new self-contained script, `alt_mysql_native_backup_restore.sh`, parses the native `.conf` key=value format (charset/collation + per-user password hash/plugin/privileges) and imports each database directly into `alt-mariadb`. The existing restore hook (`alt_mysql_user_restore_post_pre_cleanup.sh`) tries this path per-database before falling back to the existing `da_restore_native_staging_to_alt_mysql.sh`, which remains as a safety net only.
|
||||
|
||||
**Tech Stack:** Bash, MariaDB/MySQL client (`mariadb`/`mysql`), existing plugin helper library `mysql_common.sh`.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Match existing test conventions in `tests/`: plain bash scripts with `fail()`/custom asserts, fake `mariadb`/`mysql` binary stubs installed via `MYSQL_PLUGIN_CLIENT_BIN_DIR`, no external test framework.
|
||||
- Reuse `mysql_common.sh` helpers (`mysql_exec`, `mysql_escape_literal`, `mysql_quote_identifier`, `mysql_safe_identifier`, `mysql_load_alt_credentials`, `mysql_ensure_metadata_schema`) rather than reimplementing SQL execution.
|
||||
- Every new/changed script keeps `set -Eeuo pipefail` and the existing `log()`/`die()` pattern (timestamped line to `$LOG_FILE`, message to stderr, non-zero exit).
|
||||
- Never assume the `.conf` format is guaranteed stable across DirectAdmin versions — every parse failure or unrecognized field must fail closed (return non-zero) so the caller falls back to `da_restore_native_staging_to_alt_mysql.sh`, never silently skip or guess.
|
||||
- Database/user identifiers go through `mysql_safe_identifier`/`mysql_quote_identifier`; string literals go through `mysql_escape_literal`. Never interpolate parsed `.conf` values into SQL unescaped.
|
||||
- Ground-truth fixture: `example_backup/psy/backup/psy_it301.conf` (and `psy_nowa.conf`) — use this exact content (or a trimmed copy of it) in tests, don't invent a hypothetical format.
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Extract shared restore-rollback helpers into mysql_common.sh
|
||||
|
||||
**Files:**
|
||||
- Modify: `scripts/setup/mysql_common.sh`
|
||||
- Modify: `scripts/da-integration/alt_mysql_restore_payload.sh:135-169` (remove local copies, source shared ones)
|
||||
- Modify: `scripts/da-integration/da_restore_native_staging_to_alt_mysql.sh:158-192` (remove local copies, source shared ones)
|
||||
- Test: `tests/restore_rollback_safety_test.sh` (existing, must still pass unmodified)
|
||||
- Test: `tests/mysql_common_runtime_test.sh` (existing, run to confirm no regression)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces (new shared functions in `mysql_common.sh`, usable by any script that has already sourced it and set `LOG_FILE`):
|
||||
- `backup_alt_database_for_rollback DB_NAME` — dumps `DB_NAME` from `alt-mariadb` to a temp file if it exists; sets globals `RESTORE_ROLLBACK_FILE` and `RESTORE_ROLLBACK_CREATED` (0 or 1); returns 0 on success (including the "database doesn't exist yet" case), non-zero if the dump itself fails.
|
||||
- `restore_alt_database_from_rollback DB_NAME` — if `RESTORE_ROLLBACK_CREATED=1`, drops/recreates `DB_NAME` and reimports the rollback dump; no-op otherwise.
|
||||
- `cleanup_restore_rollback` — removes `$RESTORE_ROLLBACK_FILE` if present and resets the two globals. Safe to call multiple times.
|
||||
- Consumes: `mysql_exec`, `mysqldump_exec`, `mysql_escape_literal`, `mysql_quote_identifier` (all already in `mysql_common.sh`), and the global `$LOG_FILE` (must already be set by the sourcing script before these are called).
|
||||
|
||||
This task is a pure refactor (dedupe) with no behavior change — it exists because Task 3 will otherwise add a *third* near-identical copy of this exact logic.
|
||||
|
||||
- [ ] **Step 1: Copy the rollback helpers into `mysql_common.sh`**
|
||||
|
||||
Open `scripts/setup/mysql_common.sh` and append, right after `mysql_restore_stream()` (currently the last function, ending the file):
|
||||
|
||||
```bash
|
||||
|
||||
RESTORE_ROLLBACK_FILE=""
|
||||
RESTORE_ROLLBACK_CREATED=0
|
||||
|
||||
cleanup_restore_rollback() {
|
||||
if [ -n "${RESTORE_ROLLBACK_FILE:-}" ] && [ -f "$RESTORE_ROLLBACK_FILE" ]; then
|
||||
rm -f "$RESTORE_ROLLBACK_FILE"
|
||||
fi
|
||||
RESTORE_ROLLBACK_FILE=""
|
||||
RESTORE_ROLLBACK_CREATED=0
|
||||
}
|
||||
|
||||
backup_alt_database_for_rollback() {
|
||||
local db="$1"
|
||||
cleanup_restore_rollback
|
||||
RESTORE_ROLLBACK_FILE="$(mktemp)"
|
||||
|
||||
if [ -z "$(mysql_exec mysql -Nse "SELECT 1 FROM information_schema.SCHEMATA WHERE SCHEMA_NAME='$(mysql_escape_literal "$db")' LIMIT 1" 2>>"${LOG_FILE:-/dev/null}" || true)" ]; then
|
||||
cleanup_restore_rollback
|
||||
return 0
|
||||
fi
|
||||
|
||||
if mysqldump_exec \
|
||||
--single-transaction \
|
||||
--quick \
|
||||
--skip-lock-tables \
|
||||
--routines \
|
||||
--triggers \
|
||||
--events \
|
||||
--default-character-set=utf8mb4 \
|
||||
"$db" > "$RESTORE_ROLLBACK_FILE" 2>>"${LOG_FILE:-/dev/null}"; then
|
||||
RESTORE_ROLLBACK_CREATED=1
|
||||
return 0
|
||||
fi
|
||||
|
||||
cleanup_restore_rollback
|
||||
return 1
|
||||
}
|
||||
|
||||
restore_alt_database_from_rollback() {
|
||||
local db="$1"
|
||||
if [ "$RESTORE_ROLLBACK_CREATED" -ne 1 ] || [ -z "${RESTORE_ROLLBACK_FILE:-}" ] || [ ! -r "$RESTORE_ROLLBACK_FILE" ]; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
mysql_exec mysql -e "DROP DATABASE IF EXISTS $(mysql_quote_identifier "$db");" >> "${LOG_FILE:-/dev/null}" 2>&1 || true
|
||||
mysql_exec mysql -e "CREATE DATABASE $(mysql_quote_identifier "$db") CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;" >> "${LOG_FILE:-/dev/null}" 2>&1 || true
|
||||
mysql_exec "$db" < "$RESTORE_ROLLBACK_FILE" >> "${LOG_FILE:-/dev/null}" 2>&1 || {
|
||||
return 1
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Remove the local copies from `alt_mysql_restore_payload.sh`**
|
||||
|
||||
In `scripts/da-integration/alt_mysql_restore_payload.sh`, delete lines 135-169 (the `cleanup_restore_rollback`, `backup_alt_database_for_rollback`, `restore_alt_database_from_rollback` function definitions and the `RESTORE_ROLLBACK_FILE=""`/`RESTORE_ROLLBACK_CREATED=0` globals already declared at the top, lines 12-13) — keep the top-of-file variable declarations removed too since `mysql_common.sh` now owns them. Confirm `trap cleanup_restore_rollback INT TERM EXIT` (line 186) stays — it now calls the shared function.
|
||||
|
||||
- [ ] **Step 3: Remove the local copies from `da_restore_native_staging_to_alt_mysql.sh`**
|
||||
|
||||
In `scripts/da-integration/da_restore_native_staging_to_alt_mysql.sh`, delete lines 158-192 (same three functions) and the top-of-file `RESTORE_ROLLBACK_FILE=""`/`RESTORE_ROLLBACK_CREATED=0` declarations (lines 13-14). Keep `trap cleanup_restore_rollback INT TERM EXIT` (line 209).
|
||||
|
||||
- [ ] **Step 4: Run the existing rollback-safety test**
|
||||
|
||||
Run: `bash tests/restore_rollback_safety_test.sh`
|
||||
Expected: `restore_rollback_safety_test: OK` — this test only greps for the function-name strings appearing in each file, and both scripts still *call* `backup_alt_database_for_rollback`/`restore_alt_database_from_rollback`/`trap cleanup_restore_rollback`, so it must still pass even though the definitions moved.
|
||||
|
||||
- [ ] **Step 5: Run the full existing restore-related test suite to confirm no regression**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
bash tests/alt_mysql_restore_payload_test.sh
|
||||
bash tests/da_restore_archive_traversal_test.sh
|
||||
bash tests/mysql_common_runtime_test.sh
|
||||
```
|
||||
Expected: all three print their own `... : OK` line, no failures.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add scripts/setup/mysql_common.sh scripts/da-integration/alt_mysql_restore_payload.sh scripts/da-integration/da_restore_native_staging_to_alt_mysql.sh
|
||||
git commit -m "refactor: share restore rollback helpers via mysql_common.sh"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Parse DirectAdmin's native backup/<dbname>.conf format
|
||||
|
||||
**Files:**
|
||||
- Create: `scripts/da-integration/alt_mysql_native_backup_restore.sh`
|
||||
- Test: `tests/alt_mysql_native_backup_restore_parse_test.sh`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces (sourced or called by Task 3's own `main()` in the same file, and directly tested here):
|
||||
- `url_decode STRING` → prints the percent-decoded string.
|
||||
- `native_conf_field QUERY_STRING FIELD_NAME` → prints the value of `FIELD_NAME` out of an `&`-joined `key=value` query string, or empty if absent.
|
||||
- `native_conf_privilege_list QUERY_STRING` → prints a comma-joined list of SQL `GRANT` privilege names (e.g. `SELECT,INSERT,UPDATE`) derived from the `*_priv=Y` flags in `QUERY_STRING`, or `ALL` if none matched (mirrors the reverse mapping already used by `db_privilege_profile()` in `da_restore_native_staging_to_alt_mysql.sh`).
|
||||
- `parse_native_db_conf CONF_FILE` → on success (return 0), sets these globals: `NATIVE_DB_CHARSET`, `NATIVE_DB_COLLATION`, and parallel arrays `NATIVE_DB_USER_NAMES`, `NATIVE_DB_USER_HOST`, `NATIVE_DB_USER_HASH`, `NATIVE_DB_USER_PLUGIN`, `NATIVE_DB_USER_PRIVS` (one entry per MySQL user line in the `.conf`). On any structural problem (missing charset, zero users, unrecognized format) returns non-zero and leaves globals empty — caller must treat this as "fall back to native staging", never partially apply.
|
||||
- Consumes: nothing from other tasks yet (pure parsing, no DB calls in this task).
|
||||
|
||||
- [ ] **Step 1: Write the failing parser test**
|
||||
|
||||
Create `tests/alt_mysql_native_backup_restore_parse_test.sh`:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
PLUGIN_DIR="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
SCRIPT="$PLUGIN_DIR/scripts/da-integration/alt_mysql_native_backup_restore.sh"
|
||||
TMP_DIR="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMP_DIR"' EXIT
|
||||
|
||||
fail() {
|
||||
echo "FAIL: $*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Source only the parsing functions, without triggering main() (the script
|
||||
# guards its own main() call behind a BASH_SOURCE check, see Task 3 Step 1).
|
||||
# shellcheck source=/dev/null
|
||||
source "$SCRIPT"
|
||||
|
||||
# --- Real example content, copied verbatim from example_backup/psy/backup/psy_it301.conf ---
|
||||
CONF_FILE="$TMP_DIR/psy_it301.conf"
|
||||
cat > "$CONF_FILE" <<'EOF'
|
||||
accesshosts=0=localhost
|
||||
db_collation=DEFAULT_CHARACTER_SET_NAME=utf8mb3&DEFAULT_COLLATION_NAME=utf8mb3_general_ci&SCHEMA_NAME=psy_it301
|
||||
psy_it301=accesshosts=localhost&alter_priv=Y&alter_routine_priv=Y&create_priv=Y&create_routine_priv=Y&create_tmp_table_priv=Y&create_view_priv=Y&delete_priv=Y&drop_priv=Y&event_priv=Y&execute_priv=Y&index_priv=Y&insert_priv=Y&lock_tables_priv=Y&passwd=%2A8E2A6497279CF1B7F115E213B9904BD32703F4EF&plugin=mysql_native_password&references_priv=Y&select_priv=Y&show_view_priv=Y&trigger_priv=Y&update_priv=Y
|
||||
EOF
|
||||
|
||||
# url_decode
|
||||
[ "$(url_decode '%2A8E2A6497279CF1B7F115E213B9904BD32703F4EF')" = "*8E2A6497279CF1B7F115E213B9904BD32703F4EF" ] \
|
||||
|| fail "url_decode did not decode %2A correctly"
|
||||
|
||||
# native_conf_field
|
||||
QS="accesshosts=localhost&select_priv=Y&passwd=%2Aabc&plugin=mysql_native_password"
|
||||
[ "$(native_conf_field "$QS" passwd)" = "%2Aabc" ] || fail "native_conf_field did not extract passwd"
|
||||
[ "$(native_conf_field "$QS" plugin)" = "mysql_native_password" ] || fail "native_conf_field did not extract plugin"
|
||||
[ "$(native_conf_field "$QS" missing_field)" = "" ] || fail "native_conf_field should return empty for a missing field"
|
||||
|
||||
# native_conf_privilege_list
|
||||
PRIVS="$(native_conf_privilege_list "$QS")"
|
||||
[ "$PRIVS" = "SELECT" ] || fail "expected only SELECT for a single select_priv=Y field, got: $PRIVS"
|
||||
|
||||
NO_PRIVS="$(native_conf_privilege_list "accesshosts=localhost&plugin=mysql_native_password")"
|
||||
[ "$NO_PRIVS" = "ALL" ] || fail "expected ALL when no *_priv=Y flags are present, got: $NO_PRIVS"
|
||||
|
||||
# parse_native_db_conf against the real example file
|
||||
parse_native_db_conf "$CONF_FILE" || fail "parse_native_db_conf should succeed on a valid conf file"
|
||||
|
||||
[ "$NATIVE_DB_CHARSET" = "utf8mb3" ] || fail "expected charset utf8mb3, got: $NATIVE_DB_CHARSET"
|
||||
[ "$NATIVE_DB_COLLATION" = "utf8mb3_general_ci" ] || fail "expected collation utf8mb3_general_ci, got: $NATIVE_DB_COLLATION"
|
||||
[ "${#NATIVE_DB_USER_NAMES[@]}" -eq 1 ] || fail "expected exactly 1 user, got: ${#NATIVE_DB_USER_NAMES[@]}"
|
||||
[ "${NATIVE_DB_USER_NAMES[0]}" = "psy_it301" ] || fail "expected username psy_it301, got: ${NATIVE_DB_USER_NAMES[0]}"
|
||||
[ "${NATIVE_DB_USER_HOST[0]}" = "localhost" ] || fail "expected host localhost, got: ${NATIVE_DB_USER_HOST[0]}"
|
||||
[ "${NATIVE_DB_USER_HASH[0]}" = "*8E2A6497279CF1B7F115E213B9904BD32703F4EF" ] || fail "password hash not decoded correctly, got: ${NATIVE_DB_USER_HASH[0]}"
|
||||
[ "${NATIVE_DB_USER_PLUGIN[0]}" = "mysql_native_password" ] || fail "expected mysql_native_password plugin, got: ${NATIVE_DB_USER_PLUGIN[0]}"
|
||||
echo "${NATIVE_DB_USER_PRIVS[0]}" | grep -q "SELECT" || fail "expected SELECT in privilege list, got: ${NATIVE_DB_USER_PRIVS[0]}"
|
||||
|
||||
# --- A conf file with two users on the same database ---
|
||||
MULTI_CONF="$TMP_DIR/multi.conf"
|
||||
cat > "$MULTI_CONF" <<'EOF'
|
||||
accesshosts=0=localhost
|
||||
db_collation=DEFAULT_CHARACTER_SET_NAME=utf8mb4&DEFAULT_COLLATION_NAME=utf8mb4_general_ci&SCHEMA_NAME=demo_multi
|
||||
demo_owner=accesshosts=localhost&select_priv=Y&insert_priv=Y&passwd=%2Aaaa&plugin=mysql_native_password
|
||||
demo_reader=accesshosts=localhost&select_priv=Y&passwd=%2Abbb&plugin=mysql_native_password
|
||||
EOF
|
||||
|
||||
parse_native_db_conf "$MULTI_CONF" || fail "parse_native_db_conf should succeed on a multi-user conf file"
|
||||
[ "${#NATIVE_DB_USER_NAMES[@]}" -eq 2 ] || fail "expected 2 users in a multi-user conf, got: ${#NATIVE_DB_USER_NAMES[@]}"
|
||||
|
||||
# --- A malformed conf file (no db_collation) must fail closed, not guess ---
|
||||
BAD_CONF="$TMP_DIR/bad.conf"
|
||||
cat > "$BAD_CONF" <<'EOF'
|
||||
accesshosts=0=localhost
|
||||
demo_user=select_priv=Y&passwd=%2Aaaa&plugin=mysql_native_password
|
||||
EOF
|
||||
|
||||
if parse_native_db_conf "$BAD_CONF" 2>/dev/null; then
|
||||
fail "parse_native_db_conf must fail when db_collation is missing, not guess a default"
|
||||
fi
|
||||
|
||||
echo "alt_mysql_native_backup_restore_parse_test: OK"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `bash tests/alt_mysql_native_backup_restore_parse_test.sh`
|
||||
Expected: FAIL — `scripts/da-integration/alt_mysql_native_backup_restore.sh: No such file or directory` (the script doesn't exist yet).
|
||||
|
||||
- [ ] **Step 3: Create `alt_mysql_native_backup_restore.sh` with the parsing functions**
|
||||
|
||||
Create `scripts/da-integration/alt_mysql_native_backup_restore.sh`:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
PLUGIN_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
COMMON_FILE="$PLUGIN_DIR/scripts/setup/mysql_common.sh"
|
||||
SETTINGS_FILE="${DA_ALT_MYSQL_SETTINGS_FILE:-$PLUGIN_DIR/plugin-settings.conf}"
|
||||
LOG_FILE="${DA_ALT_MYSQL_RESTORE_LOG:-$PLUGIN_DIR/data/logs/da-restore.log}"
|
||||
|
||||
mkdir -p "$(dirname "$LOG_FILE")"
|
||||
|
||||
log() {
|
||||
printf '[%s] %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$*" >> "$LOG_FILE"
|
||||
}
|
||||
|
||||
die() {
|
||||
log "ERROR: $*"
|
||||
printf 'alt-mysql native backup restore: %s\n' "$*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
url_decode() {
|
||||
local encoded="${1//+/ }"
|
||||
printf '%b' "${encoded//%/\\x}"
|
||||
}
|
||||
|
||||
native_conf_field() {
|
||||
local qs="$1" field="$2" pair
|
||||
local -a pairs
|
||||
IFS='&' read -ra pairs <<< "$qs"
|
||||
for pair in "${pairs[@]}"; do
|
||||
if [ "${pair%%=*}" = "$field" ]; then
|
||||
printf '%s' "${pair#*=}"
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
return 0
|
||||
}
|
||||
|
||||
NATIVE_PRIV_FLAG_MAP=(
|
||||
"select_priv:SELECT" "insert_priv:INSERT" "update_priv:UPDATE" "delete_priv:DELETE"
|
||||
"create_priv:CREATE" "drop_priv:DROP" "index_priv:INDEX" "alter_priv:ALTER"
|
||||
"create_tmp_table_priv:CREATE TEMPORARY TABLES" "lock_tables_priv:LOCK TABLES"
|
||||
"create_view_priv:CREATE VIEW" "show_view_priv:SHOW VIEW"
|
||||
"create_routine_priv:CREATE ROUTINE" "alter_routine_priv:ALTER ROUTINE"
|
||||
"execute_priv:EXECUTE" "event_priv:EVENT" "trigger_priv:TRIGGER"
|
||||
"references_priv:REFERENCES"
|
||||
)
|
||||
|
||||
native_conf_privilege_list() {
|
||||
local qs="$1"
|
||||
local entry flag name value
|
||||
local -a privs=()
|
||||
for entry in "${NATIVE_PRIV_FLAG_MAP[@]}"; do
|
||||
flag="${entry%%:*}"
|
||||
name="${entry#*:}"
|
||||
value="$(native_conf_field "$qs" "$flag")"
|
||||
[ "$value" = "Y" ] && privs+=("$name")
|
||||
done
|
||||
if [ "${#privs[@]}" -eq 0 ]; then
|
||||
printf 'ALL'
|
||||
else
|
||||
local IFS=,
|
||||
printf '%s' "${privs[*]}"
|
||||
fi
|
||||
}
|
||||
|
||||
parse_native_db_conf() {
|
||||
local conf_file="$1"
|
||||
local line key value
|
||||
|
||||
NATIVE_DB_CHARSET=""
|
||||
NATIVE_DB_COLLATION=""
|
||||
NATIVE_DB_USER_NAMES=()
|
||||
NATIVE_DB_USER_HOST=()
|
||||
NATIVE_DB_USER_HASH=()
|
||||
NATIVE_DB_USER_PLUGIN=()
|
||||
NATIVE_DB_USER_PRIVS=()
|
||||
|
||||
[ -r "$conf_file" ] || { log "cannot read conf file: $conf_file"; return 1; }
|
||||
|
||||
while IFS= read -r line || [ -n "$line" ]; do
|
||||
[ -n "$line" ] || continue
|
||||
[[ "$line" == *=* ]] || continue
|
||||
key="${line%%=*}"
|
||||
value="${line#*=}"
|
||||
|
||||
case "$key" in
|
||||
db_collation)
|
||||
NATIVE_DB_CHARSET="$(native_conf_field "$value" DEFAULT_CHARACTER_SET_NAME)"
|
||||
NATIVE_DB_COLLATION="$(native_conf_field "$value" DEFAULT_COLLATION_NAME)"
|
||||
;;
|
||||
accesshosts) ;;
|
||||
*)
|
||||
NATIVE_DB_USER_NAMES+=("$key")
|
||||
NATIVE_DB_USER_HOST+=("$(native_conf_field "$value" accesshosts)")
|
||||
NATIVE_DB_USER_HASH+=("$(url_decode "$(native_conf_field "$value" passwd)")")
|
||||
NATIVE_DB_USER_PLUGIN+=("$(native_conf_field "$value" plugin)")
|
||||
NATIVE_DB_USER_PRIVS+=("$(native_conf_privilege_list "$value")")
|
||||
;;
|
||||
esac
|
||||
done < "$conf_file"
|
||||
|
||||
if [ -z "$NATIVE_DB_CHARSET" ]; then
|
||||
log "missing db_collation charset in $conf_file"
|
||||
return 1
|
||||
fi
|
||||
if [ "${#NATIVE_DB_USER_NAMES[@]}" -eq 0 ]; then
|
||||
log "no MySQL users found in $conf_file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
|
||||
[ -r "$COMMON_FILE" ] || die "missing helper: $COMMON_FILE"
|
||||
# shellcheck source=../setup/mysql_common.sh
|
||||
source "$COMMON_FILE"
|
||||
mysql_load_plugin_settings_file "$SETTINGS_FILE"
|
||||
# main() is added in Task 3.
|
||||
fi
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `bash tests/alt_mysql_native_backup_restore_parse_test.sh`
|
||||
Expected: `alt_mysql_native_backup_restore_parse_test: OK`
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add scripts/da-integration/alt_mysql_native_backup_restore.sh tests/alt_mysql_native_backup_restore_parse_test.sh
|
||||
git commit -m "feat: parse DirectAdmin native backup/<dbname>.conf format"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Discover and import native database pairs into alt-mariadb
|
||||
|
||||
**Files:**
|
||||
- Modify: `scripts/da-integration/alt_mysql_native_backup_restore.sh` (add discovery + import + CLI `main()`)
|
||||
- Test: `tests/alt_mysql_native_backup_restore_import_test.sh`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes from Task 2: `parse_native_db_conf`, and its output globals (`NATIVE_DB_CHARSET`, `NATIVE_DB_COLLATION`, `NATIVE_DB_USER_NAMES`, `NATIVE_DB_USER_HOST`, `NATIVE_DB_USER_HASH`, `NATIVE_DB_USER_PLUGIN`, `NATIVE_DB_USER_PRIVS`).
|
||||
- Consumes from Task 1: `backup_alt_database_for_rollback`, `restore_alt_database_from_rollback`, `cleanup_restore_rollback` (via `mysql_common.sh`).
|
||||
- Consumes from existing `mysql_common.sh`: `mysql_exec`, `mysql_escape_literal`, `mysql_quote_identifier`, `mysql_safe_identifier`, `mysql_load_alt_credentials`, `mysql_ensure_metadata_schema`.
|
||||
- Produces (used by Task 4):
|
||||
- CLI: `alt_mysql_native_backup_restore.sh --user DA_USER --root DIR_OR_ARCHIVE [--overwrite allow|deny]` — locates every `backup/<DA_USER>_*.conf` (with a matching `.sql`) reachable from `DIR_OR_ARCHIVE` (a plain directory, or a `.tar`/`.tar.gz` archive path), imports each into `alt-mariadb`, and prints one line per database to stdout: `IMPORTED <dbname>` on success or `SKIPPED <dbname> <reason>` on a per-database failure (falls back gracefully — does not abort the whole run for one bad database). Exit code is always 0 if it ran to completion (even with some `SKIPPED` lines); non-zero only for usage errors or total inability to reach `alt-mariadb`.
|
||||
- Function `import_native_database_from_conf DB_NAME CONF_FILE SQL_FILE DA_USER OVERWRITE` (`OVERWRITE` is `allow`|`deny`) — the single-database import used by both the CLI loop and directly by tests. Returns 0 and leaves the database committed on success; returns non-zero (with the rollback dump already re-applied if it had gotten far enough to touch the database) on any failure — callers must treat non-zero as "leave this one to the staging fallback", never retry with different arguments.
|
||||
|
||||
- [ ] **Step 1: Write the failing import test**
|
||||
|
||||
Create `tests/alt_mysql_native_backup_restore_import_test.sh`:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
PLUGIN_DIR="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
SCRIPT="$PLUGIN_DIR/scripts/da-integration/alt_mysql_native_backup_restore.sh"
|
||||
TMP_DIR="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMP_DIR"' EXIT
|
||||
|
||||
fail() {
|
||||
echo "FAIL: $*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
# --- Fake alt-mariadb client, tracks statements + a simple existing-db state file ---
|
||||
mkdir -p "$TMP_DIR/mariadb/bin"
|
||||
cat > "$TMP_DIR/mariadb/bin/mariadb" <<'STUB'
|
||||
#!/bin/bash
|
||||
STATE_FILE="${FAKE_DB_STATE_FILE:?FAKE_DB_STATE_FILE must be set}"
|
||||
CALL_LOG="${FAKE_CALL_LOG:?FAKE_CALL_LOG must be set}"
|
||||
|
||||
sql=""
|
||||
mode=""
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
-e) mode="sql" ;;
|
||||
-Nse) mode="sql" ;;
|
||||
--database=*|--user=*|--socket=*|--host=*|--port=*|--protocol=*) ;;
|
||||
*)
|
||||
if [ "$mode" = "sql" ]; then
|
||||
sql="$arg"
|
||||
mode=""
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [ -n "$sql" ]; then
|
||||
printf '%s\n' "$sql" >> "$CALL_LOG"
|
||||
case "$sql" in
|
||||
*"SELECT 1 FROM information_schema.SCHEMATA"*)
|
||||
name="$(printf '%s' "$sql" | sed -n "s/.*SCHEMA_NAME='\([^']*\)'.*/\1/p")"
|
||||
grep -qx "$name" "$STATE_FILE" 2>/dev/null && echo 1
|
||||
;;
|
||||
*"DROP DATABASE"*)
|
||||
name="$(printf '%s' "$sql" | sed -n 's/.*DATABASE[^`]*`\([A-Za-z0-9_]*\)`.*/\1/p')"
|
||||
[ -n "$name" ] && { grep -vx "$name" "$STATE_FILE" > "$STATE_FILE.tmp" 2>/dev/null || true; mv -f "$STATE_FILE.tmp" "$STATE_FILE"; }
|
||||
;;
|
||||
*"CREATE DATABASE"*)
|
||||
name="$(printf '%s' "$sql" | sed -n 's/.*DATABASE[^`]*`\([A-Za-z0-9_]*\)`.*/\1/p')"
|
||||
[ -n "$name" ] && echo "$name" >> "$STATE_FILE"
|
||||
;;
|
||||
esac
|
||||
exit 0
|
||||
fi
|
||||
|
||||
cat >> "$CALL_LOG"
|
||||
printf 'IMPORT\n' >> "$CALL_LOG"
|
||||
if [ "${FAKE_IMPORT_FAIL:-0}" = "1" ]; then
|
||||
exit 1
|
||||
fi
|
||||
exit 0
|
||||
STUB
|
||||
chmod 755 "$TMP_DIR/mariadb/bin/mariadb"
|
||||
|
||||
cat > "$TMP_DIR/mariadb/bin/mariadb-dump" <<'STUB'
|
||||
#!/bin/bash
|
||||
echo "-- fake rollback dump"
|
||||
exit 0
|
||||
STUB
|
||||
chmod 755 "$TMP_DIR/mariadb/bin/mariadb-dump"
|
||||
|
||||
cat > "$TMP_DIR/alt-mysql.conf" <<EOF
|
||||
host=
|
||||
passwd=secret
|
||||
port=4407
|
||||
socket=$TMP_DIR/data/mariadb.sock
|
||||
user=da_admin
|
||||
EOF
|
||||
|
||||
cat > "$TMP_DIR/settings.conf" <<EOF
|
||||
MYSQL_PLUGIN_CREDENTIALS_FILE=$TMP_DIR/alt-mysql.conf
|
||||
MYSQL_PLUGIN_CLIENT_BIN_DIR=$TMP_DIR/mariadb/bin
|
||||
EOF
|
||||
|
||||
# run_import NAME_PREFIX -- ARGS...
|
||||
# Runs the script under test with fresh, isolated state/log/call-log files
|
||||
# for this scenario, writes its stdout to "$TMP_DIR/<prefix>.stdout" and
|
||||
# writes every mariadb-stub SQL statement to "$TMP_DIR/<prefix>.calls",
|
||||
# then prints nothing (the two files are read directly by the caller).
|
||||
run_import() {
|
||||
local prefix="$1"
|
||||
shift
|
||||
local db_state="$TMP_DIR/${prefix}.dbstate"
|
||||
local call_log="$TMP_DIR/${prefix}.calls"
|
||||
local stdout_file="$TMP_DIR/${prefix}.stdout"
|
||||
local restore_log="$TMP_DIR/${prefix}.restore.log"
|
||||
: > "$db_state"
|
||||
: > "$call_log"
|
||||
|
||||
DA_ALT_MYSQL_SETTINGS_FILE="$TMP_DIR/settings.conf" \
|
||||
DA_ALT_MYSQL_RESTORE_LOG="$restore_log" \
|
||||
FAKE_DB_STATE_FILE="$db_state" \
|
||||
FAKE_CALL_LOG="$call_log" \
|
||||
bash "$SCRIPT" "$@" > "$stdout_file"
|
||||
}
|
||||
|
||||
# --- Build a fake backup directory with one valid database pair ---
|
||||
BACKUP_DIR="$TMP_DIR/backup"
|
||||
mkdir -p "$BACKUP_DIR"
|
||||
cat > "$BACKUP_DIR/demo_app.conf" <<'EOF'
|
||||
accesshosts=0=localhost
|
||||
db_collation=DEFAULT_CHARACTER_SET_NAME=utf8mb4&DEFAULT_COLLATION_NAME=utf8mb4_unicode_ci&SCHEMA_NAME=demo_app
|
||||
demo_app=accesshosts=localhost&select_priv=Y&insert_priv=Y&passwd=%2Aabc123&plugin=mysql_native_password
|
||||
EOF
|
||||
cat > "$BACKUP_DIR/demo_app.sql" <<'EOF'
|
||||
CREATE TABLE t (id INT);
|
||||
EOF
|
||||
|
||||
# --- Scenario 1: happy path, single valid database ---
|
||||
run_import scenario1 --user demo --root "$BACKUP_DIR" --overwrite allow
|
||||
|
||||
grep -q "IMPORTED demo_app" "$TMP_DIR/scenario1.stdout" \
|
||||
|| fail "expected 'IMPORTED demo_app' in output, got: $(cat "$TMP_DIR/scenario1.stdout")"
|
||||
grep -q "CREATE USER 'demo_app'@'localhost'" "$TMP_DIR/scenario1.calls" \
|
||||
|| fail "expected CREATE USER statement for demo_app@localhost"
|
||||
grep -q "GRANT SELECT,INSERT ON \`demo_app\`.\* TO 'demo_app'@'localhost'" "$TMP_DIR/scenario1.calls" \
|
||||
|| fail "expected GRANT SELECT,INSERT for demo_app@localhost"
|
||||
grep -q "IMPORT" "$TMP_DIR/scenario1.calls" || fail "expected the .sql dump to be piped in for import"
|
||||
|
||||
# --- Scenario 2: a database with no matching .conf must be skipped, not abort the run ---
|
||||
mkdir -p "$TMP_DIR/backup_missing_conf"
|
||||
cat > "$TMP_DIR/backup_missing_conf/orphan_db.sql" <<'EOF'
|
||||
CREATE TABLE t (id INT);
|
||||
EOF
|
||||
|
||||
run_import scenario2 --user orphan --root "$TMP_DIR/backup_missing_conf" --overwrite allow
|
||||
|
||||
grep -q "SKIPPED orphan_db" "$TMP_DIR/scenario2.stdout" \
|
||||
|| fail "expected orphan_db (no .conf) to be reported as SKIPPED, got: $(cat "$TMP_DIR/scenario2.stdout")"
|
||||
|
||||
echo "alt_mysql_native_backup_restore_import_test: OK"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `bash tests/alt_mysql_native_backup_restore_import_test.sh`
|
||||
Expected: FAIL — the script has no `--root`/`--user` CLI handling yet, so it exits with a usage error or does nothing.
|
||||
|
||||
- [ ] **Step 3: Add discovery, import, and `main()` to `alt_mysql_native_backup_restore.sh`**
|
||||
|
||||
Replace the trailing `if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then ... fi` block at the end of `scripts/da-integration/alt_mysql_native_backup_restore.sh` (added in Task 2 Step 3) with:
|
||||
|
||||
```bash
|
||||
mysql_import_sql_file() {
|
||||
local database="$1" file="$2"
|
||||
if [[ "$file" == *.gz ]]; then
|
||||
gunzip -c "$file" | mysql_exec "$database"
|
||||
return "${PIPESTATUS[1]}"
|
||||
fi
|
||||
mysql_exec "$database" < "$file"
|
||||
}
|
||||
|
||||
verify_database_exists() {
|
||||
local db="$1"
|
||||
[ -n "$(mysql_exec mysql -Nse "SELECT 1 FROM information_schema.SCHEMATA WHERE SCHEMA_NAME='$(mysql_escape_literal "$db")' LIMIT 1")" ] \
|
||||
|| die "database was not restored: $db"
|
||||
}
|
||||
|
||||
rebuild_metadata_for_native_import() {
|
||||
local db="$1" da_user="$2"
|
||||
local i owner="" role host
|
||||
for i in "${!NATIVE_DB_USER_NAMES[@]}"; do
|
||||
role="${NATIVE_DB_USER_NAMES[$i]}"
|
||||
host="${NATIVE_DB_USER_HOST[$i]:-localhost}"
|
||||
[ -n "$owner" ] || owner="$role"
|
||||
|
||||
mysql_exec mysql -e "
|
||||
INSERT INTO da_plugin_grant_profiles (db_name, role_name, privileges)
|
||||
VALUES ('$(mysql_escape_literal "$db")', '$(mysql_escape_literal "$role")', '$(mysql_escape_literal "${NATIVE_DB_USER_PRIVS[$i]}")')
|
||||
ON DUPLICATE KEY UPDATE privileges=VALUES(privileges), updated_at=CURRENT_TIMESTAMP;
|
||||
" >> "$LOG_FILE" 2>&1 || true
|
||||
|
||||
mysql_exec mysql -e "
|
||||
INSERT INTO da_plugin_access_hosts (da_user, role_name, host_pattern, note)
|
||||
VALUES ('$(mysql_escape_literal "$da_user")', '$(mysql_escape_literal "$role")', '$(mysql_escape_literal "$host")', 'restored from native backup/<db>.conf')
|
||||
ON DUPLICATE KEY UPDATE da_user=VALUES(da_user), note=VALUES(note), updated_at=CURRENT_TIMESTAMP;
|
||||
" >> "$LOG_FILE" 2>&1 || true
|
||||
done
|
||||
|
||||
if [ -n "$owner" ]; then
|
||||
mysql_exec mysql -e "
|
||||
INSERT INTO da_plugin_database_owners (db_name, da_user, role_name)
|
||||
VALUES ('$(mysql_escape_literal "$db")', '$(mysql_escape_literal "$da_user")', '$(mysql_escape_literal "$owner")')
|
||||
ON DUPLICATE KEY UPDATE da_user=VALUES(da_user), role_name=VALUES(role_name), updated_at=CURRENT_TIMESTAMP;
|
||||
" >> "$LOG_FILE" 2>&1 || true
|
||||
fi
|
||||
}
|
||||
|
||||
import_native_database_from_conf() {
|
||||
local db="$1" conf_file="$2" sql_file="$3" da_user="$4" overwrite="$5"
|
||||
|
||||
parse_native_db_conf "$conf_file" || { log "SKIP $db: conf did not parse"; return 1; }
|
||||
|
||||
local collation="${NATIVE_DB_COLLATION:-}"
|
||||
if [ -z "$collation" ]; then
|
||||
log "SKIP $db: no collation parsed"
|
||||
return 1
|
||||
fi
|
||||
|
||||
local i
|
||||
for i in "${!NATIVE_DB_USER_PLUGIN[@]}"; do
|
||||
if [ "${NATIVE_DB_USER_PLUGIN[$i]}" != "mysql_native_password" ]; then
|
||||
log "SKIP $db: unsupported auth plugin '${NATIVE_DB_USER_PLUGIN[$i]}' for user ${NATIVE_DB_USER_NAMES[$i]}"
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$overwrite" = "deny" ] && [ -n "$(mysql_exec mysql -Nse "SELECT 1 FROM information_schema.SCHEMATA WHERE SCHEMA_NAME='$(mysql_escape_literal "$db")' LIMIT 1")" ]; then
|
||||
log "SKIP $db: already exists and overwrite is denied"
|
||||
return 1
|
||||
fi
|
||||
|
||||
backup_alt_database_for_rollback "$db" || { log "SKIP $db: could not create rollback dump"; return 1; }
|
||||
|
||||
if ! mysql_exec mysql -e "DROP DATABASE IF EXISTS $(mysql_quote_identifier "$db");" >> "$LOG_FILE" 2>&1; then
|
||||
restore_alt_database_from_rollback "$db" || true
|
||||
log "SKIP $db: could not drop database before import"
|
||||
return 1
|
||||
fi
|
||||
if ! mysql_exec mysql -e "CREATE DATABASE $(mysql_quote_identifier "$db") CHARACTER SET $(mysql_safe_identifier "$NATIVE_DB_CHARSET") COLLATE $(mysql_safe_identifier "$collation");" >> "$LOG_FILE" 2>&1; then
|
||||
restore_alt_database_from_rollback "$db" || true
|
||||
log "SKIP $db: could not create database before import"
|
||||
return 1
|
||||
fi
|
||||
|
||||
local user host hash
|
||||
for i in "${!NATIVE_DB_USER_NAMES[@]}"; do
|
||||
user="${NATIVE_DB_USER_NAMES[$i]}"
|
||||
host="${NATIVE_DB_USER_HOST[$i]:-localhost}"
|
||||
hash="${NATIVE_DB_USER_HASH[$i]}"
|
||||
|
||||
if ! {
|
||||
printf "DROP USER IF EXISTS '%s'@'%s';\n" "$(mysql_escape_literal "$user")" "$(mysql_escape_literal "$host")"
|
||||
printf "CREATE USER '%s'@'%s' IDENTIFIED WITH mysql_native_password AS '%s';\n" \
|
||||
"$(mysql_escape_literal "$user")" "$(mysql_escape_literal "$host")" "$(mysql_escape_literal "$hash")"
|
||||
printf "GRANT %s ON %s.* TO '%s'@'%s';\n" \
|
||||
"${NATIVE_DB_USER_PRIVS[$i]}" "$(mysql_quote_identifier "$db")" "$(mysql_escape_literal "$user")" "$(mysql_escape_literal "$host")"
|
||||
} | mysql_exec mysql >> "$LOG_FILE" 2>&1; then
|
||||
restore_alt_database_from_rollback "$db" || true
|
||||
log "SKIP $db: could not recreate user ${user}@${host}"
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
|
||||
if ! mysql_import_sql_file "$db" "$sql_file" >> "$LOG_FILE" 2>&1; then
|
||||
restore_alt_database_from_rollback "$db" || true
|
||||
log "SKIP $db: could not import dump"
|
||||
return 1
|
||||
fi
|
||||
|
||||
cleanup_restore_rollback
|
||||
verify_database_exists "$db"
|
||||
rebuild_metadata_for_native_import "$db" "$da_user"
|
||||
mysql_exec mysql -e "FLUSH PRIVILEGES;" >> "$LOG_FILE" 2>&1 || true
|
||||
return 0
|
||||
}
|
||||
|
||||
find_native_pairs_in_dir() {
|
||||
local root="$1" prefix="$2"
|
||||
local conf base sql
|
||||
while IFS= read -r conf; do
|
||||
[ -n "$conf" ] || continue
|
||||
base="$(basename "$conf" .conf)"
|
||||
sql="$(dirname "$conf")/${base}.sql"
|
||||
[ -r "$sql" ] || continue
|
||||
printf '%s\t%s\t%s\n' "$base" "$conf" "$sql"
|
||||
done < <(find "$root" -maxdepth 4 -type f -name "${prefix}_*.conf" 2>/dev/null | sort)
|
||||
}
|
||||
|
||||
main() {
|
||||
local da_user="" root="" overwrite="allow"
|
||||
while [ "$#" -gt 0 ]; do
|
||||
case "$1" in
|
||||
--user) da_user="${2:-}"; shift 2 ;;
|
||||
--root) root="${2:-}"; shift 2 ;;
|
||||
--overwrite) overwrite="${2:-}"; shift 2 ;;
|
||||
*) die "unknown argument: $1" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
[ -n "$da_user" ] || die "--user is required"
|
||||
[ -n "$root" ] || die "--root is required"
|
||||
[[ "$da_user" =~ ^[A-Za-z0-9_][A-Za-z0-9_.-]*$ ]] || die "unsafe DirectAdmin username: $da_user"
|
||||
[ -d "$root" ] || die "backup root directory does not exist: $root"
|
||||
|
||||
mysql_load_alt_credentials
|
||||
mysql_ensure_metadata_schema
|
||||
|
||||
local -a handled=()
|
||||
local base conf sql
|
||||
while IFS=$'\t' read -r base conf sql; do
|
||||
[ -n "$base" ] || continue
|
||||
handled+=("$base")
|
||||
if [[ "$base" != "${da_user}_"* ]]; then
|
||||
printf 'SKIPPED %s wrong-user-prefix\n' "$base"
|
||||
continue
|
||||
fi
|
||||
if import_native_database_from_conf "$base" "$conf" "$sql" "$da_user" "$overwrite"; then
|
||||
printf 'IMPORTED %s\n' "$base"
|
||||
else
|
||||
printf 'SKIPPED %s import-failed\n' "$base"
|
||||
fi
|
||||
done < <(find_native_pairs_in_dir "$root" "$da_user")
|
||||
|
||||
# Any *.sql belonging to this user's prefix with no matching .conf was not
|
||||
# covered by the loop above (find_native_pairs_in_dir only yields pairs
|
||||
# that have both files) - report those explicitly so the caller knows to
|
||||
# fall back to native-staging-migrate for them too.
|
||||
local sql_base
|
||||
while IFS= read -r sql; do
|
||||
[ -n "$sql" ] || continue
|
||||
sql_base="$(basename "$sql" .sql)"
|
||||
[[ "$sql_base" == "${da_user}_"* ]] || continue
|
||||
local already=0 h
|
||||
for h in "${handled[@]}"; do
|
||||
[ "$h" = "$sql_base" ] && { already=1; break; }
|
||||
done
|
||||
[ "$already" -eq 1 ] || printf 'SKIPPED %s no-matching-conf\n' "$sql_base"
|
||||
done < <(find "$root" -maxdepth 4 -type f -name "${da_user}_*.sql" 2>/dev/null | sort)
|
||||
}
|
||||
|
||||
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
|
||||
[ -r "$COMMON_FILE" ] || die "missing helper: $COMMON_FILE"
|
||||
# shellcheck source=../setup/mysql_common.sh
|
||||
source "$COMMON_FILE"
|
||||
mysql_load_plugin_settings_file "$SETTINGS_FILE"
|
||||
main "$@"
|
||||
fi
|
||||
```
|
||||
|
||||
Note: the parser test from Task 2 sources this file directly (`source "$SCRIPT"`), which is why all of the above must stay guarded behind the `BASH_SOURCE[0]` check — sourcing must never call `main`.
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `bash tests/alt_mysql_native_backup_restore_import_test.sh`
|
||||
Expected: `alt_mysql_native_backup_restore_import_test: OK`
|
||||
|
||||
- [ ] **Step 5: Re-run Task 2's parser test to confirm it still passes unchanged**
|
||||
|
||||
Run: `bash tests/alt_mysql_native_backup_restore_parse_test.sh`
|
||||
Expected: `alt_mysql_native_backup_restore_parse_test: OK`
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add scripts/da-integration/alt_mysql_native_backup_restore.sh tests/alt_mysql_native_backup_restore_import_test.sh
|
||||
git commit -m "feat: import native backup/<dbname>.conf+.sql pairs directly into alt-mariadb"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Wire the new restore tier into the restore hook
|
||||
|
||||
**Files:**
|
||||
- Modify: `scripts/da-integration/alt_mysql_user_restore_post_pre_cleanup.sh:142-174` (the `main()` function)
|
||||
- Test: `tests/alt_mysql_user_restore_post_pre_cleanup_tiers_test.sh`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes from Task 3: the CLI contract `alt_mysql_native_backup_restore.sh --user DA_USER --root DIR --overwrite allow|deny`, and its `IMPORTED <db>`/`SKIPPED <db> <reason>` stdout lines.
|
||||
- Behavior change: `main()` in `alt_mysql_user_restore_post_pre_cleanup.sh` gains a middle tier between the existing plugin-payload check and the existing native-staging-migrate fallback. The native-staging-migrate script still runs afterward, but only to sweep up whatever the new tier reported as `SKIPPED` (or everything, if the new tier found nothing at all) — it must not redo databases the new tier already reported as `IMPORTED`.
|
||||
|
||||
- [ ] **Step 1: Write the failing tier-order test**
|
||||
|
||||
Create `tests/alt_mysql_user_restore_post_pre_cleanup_tiers_test.sh`:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
PLUGIN_DIR="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
SCRIPT="$PLUGIN_DIR/scripts/da-integration/alt_mysql_user_restore_post_pre_cleanup.sh"
|
||||
TMP_DIR="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMP_DIR"' EXIT
|
||||
|
||||
fail() {
|
||||
echo "FAIL: $*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
mkdir -p "$TMP_DIR/fake_bin"
|
||||
|
||||
# Fake native-backup-restore tier: reports one imported, one skipped database.
|
||||
cat > "$TMP_DIR/fake_bin/alt_mysql_native_backup_restore.sh" <<'EOF'
|
||||
#!/bin/bash
|
||||
echo "CALLED native_backup_restore $*" >> "$FAKE_CALL_LOG"
|
||||
echo "IMPORTED demo_ok"
|
||||
echo "SKIPPED demo_bad import-failed"
|
||||
EOF
|
||||
chmod 755 "$TMP_DIR/fake_bin/alt_mysql_native_backup_restore.sh"
|
||||
|
||||
# Fake native-staging-migrate tier: just logs that it ran, and with which user.
|
||||
cat > "$TMP_DIR/fake_bin/da_restore_native_staging_to_alt_mysql.sh" <<'EOF'
|
||||
#!/bin/bash
|
||||
echo "CALLED native_staging_migrate $*" >> "$FAKE_CALL_LOG"
|
||||
EOF
|
||||
chmod 755 "$TMP_DIR/fake_bin/da_restore_native_staging_to_alt_mysql.sh"
|
||||
|
||||
# No plugin payload present anywhere reachable, so the hook falls past tier 1.
|
||||
mkdir -p "$TMP_DIR/empty_restore_root"
|
||||
|
||||
CALL_LOG="$TMP_DIR/calls.log"
|
||||
: > "$CALL_LOG"
|
||||
|
||||
DA_ALT_MYSQL_RESTORE_LOG="$TMP_DIR/restore.log" \
|
||||
DA_ALT_MYSQL_NATIVE_BACKUP_RESTORE_BIN="$TMP_DIR/fake_bin/alt_mysql_native_backup_restore.sh" \
|
||||
DA_ALT_MYSQL_NATIVE_STAGING_BIN="$TMP_DIR/fake_bin/da_restore_native_staging_to_alt_mysql.sh" \
|
||||
DA_ALT_MYSQL_SETTINGS_FILE="$TMP_DIR/missing-settings.conf" \
|
||||
FAKE_CALL_LOG="$CALL_LOG" \
|
||||
restore_path="$TMP_DIR/empty_restore_root" \
|
||||
username="demo" \
|
||||
bash "$SCRIPT" demo
|
||||
|
||||
grep -q "CALLED native_backup_restore" "$CALL_LOG" || fail "expected the native-backup-restore tier to be invoked"
|
||||
grep -q "CALLED native_staging_migrate" "$CALL_LOG" || fail "expected the native-staging-migrate fallback to still run for skipped databases"
|
||||
|
||||
# The staging-migrate fallback must be told which databases were already
|
||||
# imported, so it never touches demo_ok again.
|
||||
grep -q "\-\-skip demo_ok" "$CALL_LOG" || fail "expected native-staging-migrate to be told to skip demo_ok, log: $(cat "$CALL_LOG")"
|
||||
|
||||
echo "alt_mysql_user_restore_post_pre_cleanup_tiers_test: OK"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `bash tests/alt_mysql_user_restore_post_pre_cleanup_tiers_test.sh`
|
||||
Expected: FAIL — the hook doesn't call the native-backup-restore tier yet, and doesn't honor `DA_ALT_MYSQL_NATIVE_BACKUP_RESTORE_BIN`/`DA_ALT_MYSQL_NATIVE_STAGING_BIN` overrides.
|
||||
|
||||
- [ ] **Step 3: Add the middle tier to `alt_mysql_user_restore_post_pre_cleanup.sh`**
|
||||
|
||||
In `scripts/da-integration/alt_mysql_user_restore_post_pre_cleanup.sh`, add near the top (after the existing `RESTORE_PAYLOAD_SCRIPT`/`NATIVE_MIGRATE_SCRIPT` declarations at lines 8-9):
|
||||
|
||||
```bash
|
||||
NATIVE_BACKUP_RESTORE_SCRIPT="${DA_ALT_MYSQL_NATIVE_BACKUP_RESTORE_BIN:-$SCRIPT_DIR/alt_mysql_native_backup_restore.sh}"
|
||||
NATIVE_MIGRATE_SCRIPT="${DA_ALT_MYSQL_NATIVE_STAGING_BIN:-$NATIVE_MIGRATE_SCRIPT}"
|
||||
```
|
||||
|
||||
Then replace the body of `main()` (lines 142-174) with:
|
||||
|
||||
```bash
|
||||
main() {
|
||||
local da_user enabled overwrite payload native_mode restore_root
|
||||
da_user="$(detect_da_user "${1:-}")"
|
||||
enabled="$(read_plugin_setting DA_ALT_MYSQL_RESTORE_ENABLED true)"
|
||||
native_mode="$(read_plugin_setting DA_ALT_MYSQL_NATIVE_STAGING_MIGRATE true)"
|
||||
|
||||
log "restore hook env: user=$da_user pwd=$(pwd -P) file=${file:-} filename=${filename:-}"
|
||||
|
||||
case "$enabled" in
|
||||
true|yes|on|1) ;;
|
||||
*) log "restore hook disabled by DA_ALT_MYSQL_RESTORE_ENABLED=$enabled"; return 0 ;;
|
||||
esac
|
||||
|
||||
overwrite="$(read_plugin_setting DA_ALT_MYSQL_RESTORE_OVERWRITE allow)"
|
||||
|
||||
if payload="$(detect_payload_dir)"; then
|
||||
[ -x "$RESTORE_PAYLOAD_SCRIPT" ] || die "restore payload script is not executable: $RESTORE_PAYLOAD_SCRIPT"
|
||||
log "plugin payload detected: $payload"
|
||||
"$RESTORE_PAYLOAD_SCRIPT" --user "$da_user" --payload "$payload" --overwrite "$overwrite"
|
||||
return 0
|
||||
fi
|
||||
|
||||
log "plugin payload not found; trying native backup/<dbname>.conf restore"
|
||||
|
||||
restore_root="$(candidate_dir_if_valid "${restore_path:-}")" \
|
||||
|| restore_root="$(candidate_dir_if_valid "${restore_dir:-}")" \
|
||||
|| restore_root="$(candidate_dir_if_valid "${tmpdir:-}")" \
|
||||
|| restore_root="$(candidate_dir_if_valid "${backup_path:-}")" \
|
||||
|| restore_root="$(pwd -P)"
|
||||
|
||||
local -a imported=()
|
||||
local -a skip_args=()
|
||||
if [ -x "$NATIVE_BACKUP_RESTORE_SCRIPT" ] && [ -d "$restore_root" ]; then
|
||||
while IFS= read -r line; do
|
||||
[ -n "$line" ] || continue
|
||||
log "native backup restore: $line"
|
||||
case "$line" in
|
||||
IMPORTED\ *)
|
||||
imported+=("${line#IMPORTED }")
|
||||
;;
|
||||
esac
|
||||
done < <("$NATIVE_BACKUP_RESTORE_SCRIPT" --user "$da_user" --root "$restore_root" --overwrite "$overwrite")
|
||||
else
|
||||
log "native backup restore script unavailable or restore root not found: $NATIVE_BACKUP_RESTORE_SCRIPT / $restore_root"
|
||||
fi
|
||||
|
||||
for db in "${imported[@]}"; do
|
||||
skip_args+=("--skip" "$db")
|
||||
done
|
||||
|
||||
log "native backup restore imported: ${imported[*]:-<none>}; considering native staging migration for the rest"
|
||||
case "$native_mode" in
|
||||
true|yes|on|1)
|
||||
[ -x "$NATIVE_MIGRATE_SCRIPT" ] || die "native staging migration script is not executable: $NATIVE_MIGRATE_SCRIPT"
|
||||
"$NATIVE_MIGRATE_SCRIPT" --user "$da_user" "${skip_args[@]}"
|
||||
;;
|
||||
*)
|
||||
log "native staging migration disabled; account restore continues without further alt-mysql DB migration"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `bash tests/alt_mysql_user_restore_post_pre_cleanup_tiers_test.sh`
|
||||
Expected: `alt_mysql_user_restore_post_pre_cleanup_tiers_test: OK`
|
||||
|
||||
- [ ] **Step 5: Teach `da_restore_native_staging_to_alt_mysql.sh` to accept `--skip DB` and honor it**
|
||||
|
||||
In `scripts/da-integration/da_restore_native_staging_to_alt_mysql.sh`, add a new variable near the top (after `CLEANUP_POLICY=""` at line 12):
|
||||
|
||||
```bash
|
||||
declare -a SKIP_DATABASES=()
|
||||
```
|
||||
|
||||
Update the argument-parsing `while` loop (lines 35-43) to add a case:
|
||||
|
||||
```bash
|
||||
--skip) SKIP_DATABASES+=("${2:-}"); shift 2 ;;
|
||||
```
|
||||
|
||||
Update `list_native_databases()` (lines 91-98) to exclude skipped databases:
|
||||
|
||||
```bash
|
||||
list_native_databases() {
|
||||
local db
|
||||
native_query information_schema "
|
||||
SELECT SCHEMA_NAME
|
||||
FROM SCHEMATA
|
||||
WHERE SCHEMA_NAME LIKE '$(mysql_escape_literal "$DA_USER")\\_%' ESCAPE '\\'
|
||||
ORDER BY SCHEMA_NAME
|
||||
" | grep -Ev '^(information_schema|performance_schema|mysql|sys)$' | while IFS= read -r db; do
|
||||
[ -n "$db" ] || continue
|
||||
local skipped=0 candidate
|
||||
for candidate in "${SKIP_DATABASES[@]}"; do
|
||||
[ "$candidate" = "$db" ] && { skipped=1; break; }
|
||||
done
|
||||
[ "$skipped" -eq 1 ] || printf '%s\n' "$db"
|
||||
done || true
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run the full restore-hook test suite to confirm no regression**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
bash tests/alt_mysql_user_restore_post_pre_cleanup_tiers_test.sh
|
||||
bash tests/da_restore_archive_traversal_test.sh
|
||||
bash tests/alt_mysql_restore_payload_test.sh
|
||||
bash tests/restore_rollback_safety_test.sh
|
||||
```
|
||||
Expected: all four print their own `... : OK` line.
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add scripts/da-integration/alt_mysql_user_restore_post_pre_cleanup.sh scripts/da-integration/da_restore_native_staging_to_alt_mysql.sh tests/alt_mysql_user_restore_post_pre_cleanup_tiers_test.sh
|
||||
git commit -m "feat: restore native backup/<dbname>.conf pairs before falling back to native staging"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Update the design doc to match what was actually built
|
||||
|
||||
> **Note for the controller:** `mysql-installer/da-integration.md` lives one directory above this
|
||||
> repo's root (`alt-mysql/mysql-installer/`, not inside the `alt-mysql/src` git repository that
|
||||
> Tasks 1-4 are implemented in), and is not tracked by that repository or shipped in the plugin
|
||||
> package. Do **not** dispatch an implementer/reviewer subagent pair or a git commit for this task
|
||||
> — apply the edit directly after Tasks 1-4's code is merged, since it is a plain prose update with
|
||||
> no test surface.
|
||||
|
||||
**Files:**
|
||||
- Modify: `../mysql-installer/da-integration.md` (path relative to the `src/` repo root; Phase 5B and Phase 7 sections)
|
||||
|
||||
**Interfaces:** None — documentation only.
|
||||
|
||||
- [ ] **Step 1: Replace the Phase 5B description**
|
||||
|
||||
In `mysql-installer/da-integration.md`, find the `## Phase 5B: External DirectAdmin Backups Without Plugin Payload` section (currently describing "native staging then migrate" as the primary mechanism for external backups). Replace its `### Problem`/`### Recommended Policy` prose to state, as the primary mechanism: the restore hook (`alt_mysql_user_restore_post_pre_cleanup.sh`) now first calls `alt_mysql_native_backup_restore.sh`, which parses `backup/<dbname>.conf` + `backup/<dbname>.sql` pairs directly out of the archive/extraction root and imports each database straight into `alt-mariadb` — no native MariaDB staging involved. `da_restore_native_staging_to_alt_mysql.sh` (the existing native-staging-then-migrate flow) remains, unchanged in its own logic, as the per-database fallback only for databases the new tier reports as `SKIPPED` (missing/malformed `.conf`, unsupported auth plugin, or a genuinely foreign non-DA-native backup).
|
||||
|
||||
- [ ] **Step 2: Update the Phase 7 "CustomBuild MariaDB Staging Role" framing**
|
||||
|
||||
In the same file's `## Phase 7` section, add a note that CustomBuild's native MariaDB no longer needs to be usable as a restore staging target for the common case — it is now a fallback path only, exercised solely when `alt_mysql_native_backup_restore.sh` reports a database as `SKIPPED`. It must still be kept running so DirectAdmin core's own login/version check has a legacy-license-safe target to talk to.
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add mysql-installer/da-integration.md
|
||||
git commit -m "docs: reflect native-conf-based restore as the primary external-backup path"
|
||||
```
|
||||
Reference in New Issue
Block a user