From 74ac47f75bcdca97d033dea3d6e2420237a62b11 Mon Sep 17 00:00:00 2001 From: Marek Miklewicz Date: Sun, 5 Jul 2026 15:01:52 +0200 Subject: [PATCH] Add design spec and implementation plan for restore-without-native-staging Ignore .superpowers/ (subagent-driven-development's scratch progress ledger). --- .gitignore | 1 + ...26-07-05-restore-without-native-staging.md | 996 ++++++++++++++++++ ...5-restore-without-native-staging-design.md | 139 +++ 3 files changed, 1136 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-05-restore-without-native-staging.md create mode 100644 docs/superpowers/specs/2026-07-05-restore-without-native-staging-design.md diff --git a/.gitignore b/.gitignore index a40718e..10c570e 100644 --- a/.gitignore +++ b/.gitignore @@ -4,3 +4,4 @@ __MACOSX/ error.log data/* assets/adminer/* +.superpowers/ diff --git a/docs/superpowers/plans/2026-07-05-restore-without-native-staging.md b/docs/superpowers/plans/2026-07-05-restore-without-native-staging.md new file mode 100644 index 0000000..45340bf --- /dev/null +++ b/docs/superpowers/plans/2026-07-05-restore-without-native-staging.md @@ -0,0 +1,996 @@ +# Restore Without Native Staging Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Restore external (non-plugin-origin) DirectAdmin account backups directly into `alt-mariadb` by parsing DirectAdmin's native `backup/.conf` + `backup/.sql` file pairs straight out of the archive, without ever staging data in the CustomBuild-managed native MariaDB. + +**Architecture:** A new self-contained script, `alt_mysql_native_backup_restore.sh`, parses the native `.conf` key=value format (charset/collation + per-user password hash/plugin/privileges) and imports each database directly into `alt-mariadb`. The existing restore hook (`alt_mysql_user_restore_post_pre_cleanup.sh`) tries this path per-database before falling back to the existing `da_restore_native_staging_to_alt_mysql.sh`, which remains as a safety net only. + +**Tech Stack:** Bash, MariaDB/MySQL client (`mariadb`/`mysql`), existing plugin helper library `mysql_common.sh`. + +## Global Constraints + +- Match existing test conventions in `tests/`: plain bash scripts with `fail()`/custom asserts, fake `mariadb`/`mysql` binary stubs installed via `MYSQL_PLUGIN_CLIENT_BIN_DIR`, no external test framework. +- Reuse `mysql_common.sh` helpers (`mysql_exec`, `mysql_escape_literal`, `mysql_quote_identifier`, `mysql_safe_identifier`, `mysql_load_alt_credentials`, `mysql_ensure_metadata_schema`) rather than reimplementing SQL execution. +- Every new/changed script keeps `set -Eeuo pipefail` and the existing `log()`/`die()` pattern (timestamped line to `$LOG_FILE`, message to stderr, non-zero exit). +- Never assume the `.conf` format is guaranteed stable across DirectAdmin versions — every parse failure or unrecognized field must fail closed (return non-zero) so the caller falls back to `da_restore_native_staging_to_alt_mysql.sh`, never silently skip or guess. +- Database/user identifiers go through `mysql_safe_identifier`/`mysql_quote_identifier`; string literals go through `mysql_escape_literal`. Never interpolate parsed `.conf` values into SQL unescaped. +- Ground-truth fixture: `example_backup/psy/backup/psy_it301.conf` (and `psy_nowa.conf`) — use this exact content (or a trimmed copy of it) in tests, don't invent a hypothetical format. + +--- + +## Task 1: Extract shared restore-rollback helpers into mysql_common.sh + +**Files:** +- Modify: `scripts/setup/mysql_common.sh` +- Modify: `scripts/da-integration/alt_mysql_restore_payload.sh:135-169` (remove local copies, source shared ones) +- Modify: `scripts/da-integration/da_restore_native_staging_to_alt_mysql.sh:158-192` (remove local copies, source shared ones) +- Test: `tests/restore_rollback_safety_test.sh` (existing, must still pass unmodified) +- Test: `tests/mysql_common_runtime_test.sh` (existing, run to confirm no regression) + +**Interfaces:** +- Produces (new shared functions in `mysql_common.sh`, usable by any script that has already sourced it and set `LOG_FILE`): + - `backup_alt_database_for_rollback DB_NAME` — dumps `DB_NAME` from `alt-mariadb` to a temp file if it exists; sets globals `RESTORE_ROLLBACK_FILE` and `RESTORE_ROLLBACK_CREATED` (0 or 1); returns 0 on success (including the "database doesn't exist yet" case), non-zero if the dump itself fails. + - `restore_alt_database_from_rollback DB_NAME` — if `RESTORE_ROLLBACK_CREATED=1`, drops/recreates `DB_NAME` and reimports the rollback dump; no-op otherwise. + - `cleanup_restore_rollback` — removes `$RESTORE_ROLLBACK_FILE` if present and resets the two globals. Safe to call multiple times. +- Consumes: `mysql_exec`, `mysqldump_exec`, `mysql_escape_literal`, `mysql_quote_identifier` (all already in `mysql_common.sh`), and the global `$LOG_FILE` (must already be set by the sourcing script before these are called). + +This task is a pure refactor (dedupe) with no behavior change — it exists because Task 3 will otherwise add a *third* near-identical copy of this exact logic. + +- [ ] **Step 1: Copy the rollback helpers into `mysql_common.sh`** + +Open `scripts/setup/mysql_common.sh` and append, right after `mysql_restore_stream()` (currently the last function, ending the file): + +```bash + +RESTORE_ROLLBACK_FILE="" +RESTORE_ROLLBACK_CREATED=0 + +cleanup_restore_rollback() { + if [ -n "${RESTORE_ROLLBACK_FILE:-}" ] && [ -f "$RESTORE_ROLLBACK_FILE" ]; then + rm -f "$RESTORE_ROLLBACK_FILE" + fi + RESTORE_ROLLBACK_FILE="" + RESTORE_ROLLBACK_CREATED=0 +} + +backup_alt_database_for_rollback() { + local db="$1" + cleanup_restore_rollback + RESTORE_ROLLBACK_FILE="$(mktemp)" + + if [ -z "$(mysql_exec mysql -Nse "SELECT 1 FROM information_schema.SCHEMATA WHERE SCHEMA_NAME='$(mysql_escape_literal "$db")' LIMIT 1" 2>>"${LOG_FILE:-/dev/null}" || true)" ]; then + cleanup_restore_rollback + return 0 + fi + + if mysqldump_exec \ + --single-transaction \ + --quick \ + --skip-lock-tables \ + --routines \ + --triggers \ + --events \ + --default-character-set=utf8mb4 \ + "$db" > "$RESTORE_ROLLBACK_FILE" 2>>"${LOG_FILE:-/dev/null}"; then + RESTORE_ROLLBACK_CREATED=1 + return 0 + fi + + cleanup_restore_rollback + return 1 +} + +restore_alt_database_from_rollback() { + local db="$1" + if [ "$RESTORE_ROLLBACK_CREATED" -ne 1 ] || [ -z "${RESTORE_ROLLBACK_FILE:-}" ] || [ ! -r "$RESTORE_ROLLBACK_FILE" ]; then + return 0 + fi + + mysql_exec mysql -e "DROP DATABASE IF EXISTS $(mysql_quote_identifier "$db");" >> "${LOG_FILE:-/dev/null}" 2>&1 || true + mysql_exec mysql -e "CREATE DATABASE $(mysql_quote_identifier "$db") CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;" >> "${LOG_FILE:-/dev/null}" 2>&1 || true + mysql_exec "$db" < "$RESTORE_ROLLBACK_FILE" >> "${LOG_FILE:-/dev/null}" 2>&1 || { + return 1 + } +} +``` + +- [ ] **Step 2: Remove the local copies from `alt_mysql_restore_payload.sh`** + +In `scripts/da-integration/alt_mysql_restore_payload.sh`, delete lines 135-169 (the `cleanup_restore_rollback`, `backup_alt_database_for_rollback`, `restore_alt_database_from_rollback` function definitions and the `RESTORE_ROLLBACK_FILE=""`/`RESTORE_ROLLBACK_CREATED=0` globals already declared at the top, lines 12-13) — keep the top-of-file variable declarations removed too since `mysql_common.sh` now owns them. Confirm `trap cleanup_restore_rollback INT TERM EXIT` (line 186) stays — it now calls the shared function. + +- [ ] **Step 3: Remove the local copies from `da_restore_native_staging_to_alt_mysql.sh`** + +In `scripts/da-integration/da_restore_native_staging_to_alt_mysql.sh`, delete lines 158-192 (same three functions) and the top-of-file `RESTORE_ROLLBACK_FILE=""`/`RESTORE_ROLLBACK_CREATED=0` declarations (lines 13-14). Keep `trap cleanup_restore_rollback INT TERM EXIT` (line 209). + +- [ ] **Step 4: Run the existing rollback-safety test** + +Run: `bash tests/restore_rollback_safety_test.sh` +Expected: `restore_rollback_safety_test: OK` — this test only greps for the function-name strings appearing in each file, and both scripts still *call* `backup_alt_database_for_rollback`/`restore_alt_database_from_rollback`/`trap cleanup_restore_rollback`, so it must still pass even though the definitions moved. + +- [ ] **Step 5: Run the full existing restore-related test suite to confirm no regression** + +Run: +```bash +bash tests/alt_mysql_restore_payload_test.sh +bash tests/da_restore_archive_traversal_test.sh +bash tests/mysql_common_runtime_test.sh +``` +Expected: all three print their own `... : OK` line, no failures. + +- [ ] **Step 6: Commit** + +```bash +git add scripts/setup/mysql_common.sh scripts/da-integration/alt_mysql_restore_payload.sh scripts/da-integration/da_restore_native_staging_to_alt_mysql.sh +git commit -m "refactor: share restore rollback helpers via mysql_common.sh" +``` + +--- + +## Task 2: Parse DirectAdmin's native backup/.conf format + +**Files:** +- Create: `scripts/da-integration/alt_mysql_native_backup_restore.sh` +- Test: `tests/alt_mysql_native_backup_restore_parse_test.sh` + +**Interfaces:** +- Produces (sourced or called by Task 3's own `main()` in the same file, and directly tested here): + - `url_decode STRING` → prints the percent-decoded string. + - `native_conf_field QUERY_STRING FIELD_NAME` → prints the value of `FIELD_NAME` out of an `&`-joined `key=value` query string, or empty if absent. + - `native_conf_privilege_list QUERY_STRING` → prints a comma-joined list of SQL `GRANT` privilege names (e.g. `SELECT,INSERT,UPDATE`) derived from the `*_priv=Y` flags in `QUERY_STRING`, or `ALL` if none matched (mirrors the reverse mapping already used by `db_privilege_profile()` in `da_restore_native_staging_to_alt_mysql.sh`). + - `parse_native_db_conf CONF_FILE` → on success (return 0), sets these globals: `NATIVE_DB_CHARSET`, `NATIVE_DB_COLLATION`, and parallel arrays `NATIVE_DB_USER_NAMES`, `NATIVE_DB_USER_HOST`, `NATIVE_DB_USER_HASH`, `NATIVE_DB_USER_PLUGIN`, `NATIVE_DB_USER_PRIVS` (one entry per MySQL user line in the `.conf`). On any structural problem (missing charset, zero users, unrecognized format) returns non-zero and leaves globals empty — caller must treat this as "fall back to native staging", never partially apply. +- Consumes: nothing from other tasks yet (pure parsing, no DB calls in this task). + +- [ ] **Step 1: Write the failing parser test** + +Create `tests/alt_mysql_native_backup_restore_parse_test.sh`: + +```bash +#!/bin/bash +set -euo pipefail + +PLUGIN_DIR="$(cd "$(dirname "$0")/.." && pwd)" +SCRIPT="$PLUGIN_DIR/scripts/da-integration/alt_mysql_native_backup_restore.sh" +TMP_DIR="$(mktemp -d)" +trap 'rm -rf "$TMP_DIR"' EXIT + +fail() { + echo "FAIL: $*" >&2 + exit 1 +} + +# Source only the parsing functions, without triggering main() (the script +# guards its own main() call behind a BASH_SOURCE check, see Task 3 Step 1). +# shellcheck source=/dev/null +source "$SCRIPT" + +# --- Real example content, copied verbatim from example_backup/psy/backup/psy_it301.conf --- +CONF_FILE="$TMP_DIR/psy_it301.conf" +cat > "$CONF_FILE" <<'EOF' +accesshosts=0=localhost +db_collation=DEFAULT_CHARACTER_SET_NAME=utf8mb3&DEFAULT_COLLATION_NAME=utf8mb3_general_ci&SCHEMA_NAME=psy_it301 +psy_it301=accesshosts=localhost&alter_priv=Y&alter_routine_priv=Y&create_priv=Y&create_routine_priv=Y&create_tmp_table_priv=Y&create_view_priv=Y&delete_priv=Y&drop_priv=Y&event_priv=Y&execute_priv=Y&index_priv=Y&insert_priv=Y&lock_tables_priv=Y&passwd=%2A8E2A6497279CF1B7F115E213B9904BD32703F4EF&plugin=mysql_native_password&references_priv=Y&select_priv=Y&show_view_priv=Y&trigger_priv=Y&update_priv=Y +EOF + +# url_decode +[ "$(url_decode '%2A8E2A6497279CF1B7F115E213B9904BD32703F4EF')" = "*8E2A6497279CF1B7F115E213B9904BD32703F4EF" ] \ + || fail "url_decode did not decode %2A correctly" + +# native_conf_field +QS="accesshosts=localhost&select_priv=Y&passwd=%2Aabc&plugin=mysql_native_password" +[ "$(native_conf_field "$QS" passwd)" = "%2Aabc" ] || fail "native_conf_field did not extract passwd" +[ "$(native_conf_field "$QS" plugin)" = "mysql_native_password" ] || fail "native_conf_field did not extract plugin" +[ "$(native_conf_field "$QS" missing_field)" = "" ] || fail "native_conf_field should return empty for a missing field" + +# native_conf_privilege_list +PRIVS="$(native_conf_privilege_list "$QS")" +[ "$PRIVS" = "SELECT" ] || fail "expected only SELECT for a single select_priv=Y field, got: $PRIVS" + +NO_PRIVS="$(native_conf_privilege_list "accesshosts=localhost&plugin=mysql_native_password")" +[ "$NO_PRIVS" = "ALL" ] || fail "expected ALL when no *_priv=Y flags are present, got: $NO_PRIVS" + +# parse_native_db_conf against the real example file +parse_native_db_conf "$CONF_FILE" || fail "parse_native_db_conf should succeed on a valid conf file" + +[ "$NATIVE_DB_CHARSET" = "utf8mb3" ] || fail "expected charset utf8mb3, got: $NATIVE_DB_CHARSET" +[ "$NATIVE_DB_COLLATION" = "utf8mb3_general_ci" ] || fail "expected collation utf8mb3_general_ci, got: $NATIVE_DB_COLLATION" +[ "${#NATIVE_DB_USER_NAMES[@]}" -eq 1 ] || fail "expected exactly 1 user, got: ${#NATIVE_DB_USER_NAMES[@]}" +[ "${NATIVE_DB_USER_NAMES[0]}" = "psy_it301" ] || fail "expected username psy_it301, got: ${NATIVE_DB_USER_NAMES[0]}" +[ "${NATIVE_DB_USER_HOST[0]}" = "localhost" ] || fail "expected host localhost, got: ${NATIVE_DB_USER_HOST[0]}" +[ "${NATIVE_DB_USER_HASH[0]}" = "*8E2A6497279CF1B7F115E213B9904BD32703F4EF" ] || fail "password hash not decoded correctly, got: ${NATIVE_DB_USER_HASH[0]}" +[ "${NATIVE_DB_USER_PLUGIN[0]}" = "mysql_native_password" ] || fail "expected mysql_native_password plugin, got: ${NATIVE_DB_USER_PLUGIN[0]}" +echo "${NATIVE_DB_USER_PRIVS[0]}" | grep -q "SELECT" || fail "expected SELECT in privilege list, got: ${NATIVE_DB_USER_PRIVS[0]}" + +# --- A conf file with two users on the same database --- +MULTI_CONF="$TMP_DIR/multi.conf" +cat > "$MULTI_CONF" <<'EOF' +accesshosts=0=localhost +db_collation=DEFAULT_CHARACTER_SET_NAME=utf8mb4&DEFAULT_COLLATION_NAME=utf8mb4_general_ci&SCHEMA_NAME=demo_multi +demo_owner=accesshosts=localhost&select_priv=Y&insert_priv=Y&passwd=%2Aaaa&plugin=mysql_native_password +demo_reader=accesshosts=localhost&select_priv=Y&passwd=%2Abbb&plugin=mysql_native_password +EOF + +parse_native_db_conf "$MULTI_CONF" || fail "parse_native_db_conf should succeed on a multi-user conf file" +[ "${#NATIVE_DB_USER_NAMES[@]}" -eq 2 ] || fail "expected 2 users in a multi-user conf, got: ${#NATIVE_DB_USER_NAMES[@]}" + +# --- A malformed conf file (no db_collation) must fail closed, not guess --- +BAD_CONF="$TMP_DIR/bad.conf" +cat > "$BAD_CONF" <<'EOF' +accesshosts=0=localhost +demo_user=select_priv=Y&passwd=%2Aaaa&plugin=mysql_native_password +EOF + +if parse_native_db_conf "$BAD_CONF" 2>/dev/null; then + fail "parse_native_db_conf must fail when db_collation is missing, not guess a default" +fi + +echo "alt_mysql_native_backup_restore_parse_test: OK" +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `bash tests/alt_mysql_native_backup_restore_parse_test.sh` +Expected: FAIL — `scripts/da-integration/alt_mysql_native_backup_restore.sh: No such file or directory` (the script doesn't exist yet). + +- [ ] **Step 3: Create `alt_mysql_native_backup_restore.sh` with the parsing functions** + +Create `scripts/da-integration/alt_mysql_native_backup_restore.sh`: + +```bash +#!/bin/bash +set -Eeuo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +PLUGIN_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)" +COMMON_FILE="$PLUGIN_DIR/scripts/setup/mysql_common.sh" +SETTINGS_FILE="${DA_ALT_MYSQL_SETTINGS_FILE:-$PLUGIN_DIR/plugin-settings.conf}" +LOG_FILE="${DA_ALT_MYSQL_RESTORE_LOG:-$PLUGIN_DIR/data/logs/da-restore.log}" + +mkdir -p "$(dirname "$LOG_FILE")" + +log() { + printf '[%s] %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$*" >> "$LOG_FILE" +} + +die() { + log "ERROR: $*" + printf 'alt-mysql native backup restore: %s\n' "$*" >&2 + exit 1 +} + +url_decode() { + local encoded="${1//+/ }" + printf '%b' "${encoded//%/\\x}" +} + +native_conf_field() { + local qs="$1" field="$2" pair + local -a pairs + IFS='&' read -ra pairs <<< "$qs" + for pair in "${pairs[@]}"; do + if [ "${pair%%=*}" = "$field" ]; then + printf '%s' "${pair#*=}" + return 0 + fi + done + return 0 +} + +NATIVE_PRIV_FLAG_MAP=( + "select_priv:SELECT" "insert_priv:INSERT" "update_priv:UPDATE" "delete_priv:DELETE" + "create_priv:CREATE" "drop_priv:DROP" "index_priv:INDEX" "alter_priv:ALTER" + "create_tmp_table_priv:CREATE TEMPORARY TABLES" "lock_tables_priv:LOCK TABLES" + "create_view_priv:CREATE VIEW" "show_view_priv:SHOW VIEW" + "create_routine_priv:CREATE ROUTINE" "alter_routine_priv:ALTER ROUTINE" + "execute_priv:EXECUTE" "event_priv:EVENT" "trigger_priv:TRIGGER" + "references_priv:REFERENCES" +) + +native_conf_privilege_list() { + local qs="$1" + local entry flag name value + local -a privs=() + for entry in "${NATIVE_PRIV_FLAG_MAP[@]}"; do + flag="${entry%%:*}" + name="${entry#*:}" + value="$(native_conf_field "$qs" "$flag")" + [ "$value" = "Y" ] && privs+=("$name") + done + if [ "${#privs[@]}" -eq 0 ]; then + printf 'ALL' + else + local IFS=, + printf '%s' "${privs[*]}" + fi +} + +parse_native_db_conf() { + local conf_file="$1" + local line key value + + NATIVE_DB_CHARSET="" + NATIVE_DB_COLLATION="" + NATIVE_DB_USER_NAMES=() + NATIVE_DB_USER_HOST=() + NATIVE_DB_USER_HASH=() + NATIVE_DB_USER_PLUGIN=() + NATIVE_DB_USER_PRIVS=() + + [ -r "$conf_file" ] || { log "cannot read conf file: $conf_file"; return 1; } + + while IFS= read -r line || [ -n "$line" ]; do + [ -n "$line" ] || continue + [[ "$line" == *=* ]] || continue + key="${line%%=*}" + value="${line#*=}" + + case "$key" in + db_collation) + NATIVE_DB_CHARSET="$(native_conf_field "$value" DEFAULT_CHARACTER_SET_NAME)" + NATIVE_DB_COLLATION="$(native_conf_field "$value" DEFAULT_COLLATION_NAME)" + ;; + accesshosts) ;; + *) + NATIVE_DB_USER_NAMES+=("$key") + NATIVE_DB_USER_HOST+=("$(native_conf_field "$value" accesshosts)") + NATIVE_DB_USER_HASH+=("$(url_decode "$(native_conf_field "$value" passwd)")") + NATIVE_DB_USER_PLUGIN+=("$(native_conf_field "$value" plugin)") + NATIVE_DB_USER_PRIVS+=("$(native_conf_privilege_list "$value")") + ;; + esac + done < "$conf_file" + + if [ -z "$NATIVE_DB_CHARSET" ]; then + log "missing db_collation charset in $conf_file" + return 1 + fi + if [ "${#NATIVE_DB_USER_NAMES[@]}" -eq 0 ]; then + log "no MySQL users found in $conf_file" + return 1 + fi + + return 0 +} + +if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then + [ -r "$COMMON_FILE" ] || die "missing helper: $COMMON_FILE" + # shellcheck source=../setup/mysql_common.sh + source "$COMMON_FILE" + mysql_load_plugin_settings_file "$SETTINGS_FILE" + # main() is added in Task 3. +fi +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `bash tests/alt_mysql_native_backup_restore_parse_test.sh` +Expected: `alt_mysql_native_backup_restore_parse_test: OK` + +- [ ] **Step 5: Commit** + +```bash +git add scripts/da-integration/alt_mysql_native_backup_restore.sh tests/alt_mysql_native_backup_restore_parse_test.sh +git commit -m "feat: parse DirectAdmin native backup/.conf format" +``` + +--- + +## Task 3: Discover and import native database pairs into alt-mariadb + +**Files:** +- Modify: `scripts/da-integration/alt_mysql_native_backup_restore.sh` (add discovery + import + CLI `main()`) +- Test: `tests/alt_mysql_native_backup_restore_import_test.sh` + +**Interfaces:** +- Consumes from Task 2: `parse_native_db_conf`, and its output globals (`NATIVE_DB_CHARSET`, `NATIVE_DB_COLLATION`, `NATIVE_DB_USER_NAMES`, `NATIVE_DB_USER_HOST`, `NATIVE_DB_USER_HASH`, `NATIVE_DB_USER_PLUGIN`, `NATIVE_DB_USER_PRIVS`). +- Consumes from Task 1: `backup_alt_database_for_rollback`, `restore_alt_database_from_rollback`, `cleanup_restore_rollback` (via `mysql_common.sh`). +- Consumes from existing `mysql_common.sh`: `mysql_exec`, `mysql_escape_literal`, `mysql_quote_identifier`, `mysql_safe_identifier`, `mysql_load_alt_credentials`, `mysql_ensure_metadata_schema`. +- Produces (used by Task 4): + - CLI: `alt_mysql_native_backup_restore.sh --user DA_USER --root DIR_OR_ARCHIVE [--overwrite allow|deny]` — locates every `backup/_*.conf` (with a matching `.sql`) reachable from `DIR_OR_ARCHIVE` (a plain directory, or a `.tar`/`.tar.gz` archive path), imports each into `alt-mariadb`, and prints one line per database to stdout: `IMPORTED ` on success or `SKIPPED ` on a per-database failure (falls back gracefully — does not abort the whole run for one bad database). Exit code is always 0 if it ran to completion (even with some `SKIPPED` lines); non-zero only for usage errors or total inability to reach `alt-mariadb`. + - Function `import_native_database_from_conf DB_NAME CONF_FILE SQL_FILE DA_USER OVERWRITE` (`OVERWRITE` is `allow`|`deny`) — the single-database import used by both the CLI loop and directly by tests. Returns 0 and leaves the database committed on success; returns non-zero (with the rollback dump already re-applied if it had gotten far enough to touch the database) on any failure — callers must treat non-zero as "leave this one to the staging fallback", never retry with different arguments. + +- [ ] **Step 1: Write the failing import test** + +Create `tests/alt_mysql_native_backup_restore_import_test.sh`: + +```bash +#!/bin/bash +set -euo pipefail + +PLUGIN_DIR="$(cd "$(dirname "$0")/.." && pwd)" +SCRIPT="$PLUGIN_DIR/scripts/da-integration/alt_mysql_native_backup_restore.sh" +TMP_DIR="$(mktemp -d)" +trap 'rm -rf "$TMP_DIR"' EXIT + +fail() { + echo "FAIL: $*" >&2 + exit 1 +} + +# --- Fake alt-mariadb client, tracks statements + a simple existing-db state file --- +mkdir -p "$TMP_DIR/mariadb/bin" +cat > "$TMP_DIR/mariadb/bin/mariadb" <<'STUB' +#!/bin/bash +STATE_FILE="${FAKE_DB_STATE_FILE:?FAKE_DB_STATE_FILE must be set}" +CALL_LOG="${FAKE_CALL_LOG:?FAKE_CALL_LOG must be set}" + +sql="" +mode="" +for arg in "$@"; do + case "$arg" in + -e) mode="sql" ;; + -Nse) mode="sql" ;; + --database=*|--user=*|--socket=*|--host=*|--port=*|--protocol=*) ;; + *) + if [ "$mode" = "sql" ]; then + sql="$arg" + mode="" + fi + ;; + esac +done + +if [ -n "$sql" ]; then + printf '%s\n' "$sql" >> "$CALL_LOG" + case "$sql" in + *"SELECT 1 FROM information_schema.SCHEMATA"*) + name="$(printf '%s' "$sql" | sed -n "s/.*SCHEMA_NAME='\([^']*\)'.*/\1/p")" + grep -qx "$name" "$STATE_FILE" 2>/dev/null && echo 1 + ;; + *"DROP DATABASE"*) + name="$(printf '%s' "$sql" | sed -n 's/.*DATABASE[^`]*`\([A-Za-z0-9_]*\)`.*/\1/p')" + [ -n "$name" ] && { grep -vx "$name" "$STATE_FILE" > "$STATE_FILE.tmp" 2>/dev/null || true; mv -f "$STATE_FILE.tmp" "$STATE_FILE"; } + ;; + *"CREATE DATABASE"*) + name="$(printf '%s' "$sql" | sed -n 's/.*DATABASE[^`]*`\([A-Za-z0-9_]*\)`.*/\1/p')" + [ -n "$name" ] && echo "$name" >> "$STATE_FILE" + ;; + esac + exit 0 +fi + +cat >> "$CALL_LOG" +printf 'IMPORT\n' >> "$CALL_LOG" +if [ "${FAKE_IMPORT_FAIL:-0}" = "1" ]; then + exit 1 +fi +exit 0 +STUB +chmod 755 "$TMP_DIR/mariadb/bin/mariadb" + +cat > "$TMP_DIR/mariadb/bin/mariadb-dump" <<'STUB' +#!/bin/bash +echo "-- fake rollback dump" +exit 0 +STUB +chmod 755 "$TMP_DIR/mariadb/bin/mariadb-dump" + +cat > "$TMP_DIR/alt-mysql.conf" < "$TMP_DIR/settings.conf" <.stdout" and +# writes every mariadb-stub SQL statement to "$TMP_DIR/.calls", +# then prints nothing (the two files are read directly by the caller). +run_import() { + local prefix="$1" + shift + local db_state="$TMP_DIR/${prefix}.dbstate" + local call_log="$TMP_DIR/${prefix}.calls" + local stdout_file="$TMP_DIR/${prefix}.stdout" + local restore_log="$TMP_DIR/${prefix}.restore.log" + : > "$db_state" + : > "$call_log" + + DA_ALT_MYSQL_SETTINGS_FILE="$TMP_DIR/settings.conf" \ + DA_ALT_MYSQL_RESTORE_LOG="$restore_log" \ + FAKE_DB_STATE_FILE="$db_state" \ + FAKE_CALL_LOG="$call_log" \ + bash "$SCRIPT" "$@" > "$stdout_file" +} + +# --- Build a fake backup directory with one valid database pair --- +BACKUP_DIR="$TMP_DIR/backup" +mkdir -p "$BACKUP_DIR" +cat > "$BACKUP_DIR/demo_app.conf" <<'EOF' +accesshosts=0=localhost +db_collation=DEFAULT_CHARACTER_SET_NAME=utf8mb4&DEFAULT_COLLATION_NAME=utf8mb4_unicode_ci&SCHEMA_NAME=demo_app +demo_app=accesshosts=localhost&select_priv=Y&insert_priv=Y&passwd=%2Aabc123&plugin=mysql_native_password +EOF +cat > "$BACKUP_DIR/demo_app.sql" <<'EOF' +CREATE TABLE t (id INT); +EOF + +# --- Scenario 1: happy path, single valid database --- +run_import scenario1 --user demo --root "$BACKUP_DIR" --overwrite allow + +grep -q "IMPORTED demo_app" "$TMP_DIR/scenario1.stdout" \ + || fail "expected 'IMPORTED demo_app' in output, got: $(cat "$TMP_DIR/scenario1.stdout")" +grep -q "CREATE USER 'demo_app'@'localhost'" "$TMP_DIR/scenario1.calls" \ + || fail "expected CREATE USER statement for demo_app@localhost" +grep -q "GRANT SELECT,INSERT ON \`demo_app\`.\* TO 'demo_app'@'localhost'" "$TMP_DIR/scenario1.calls" \ + || fail "expected GRANT SELECT,INSERT for demo_app@localhost" +grep -q "IMPORT" "$TMP_DIR/scenario1.calls" || fail "expected the .sql dump to be piped in for import" + +# --- Scenario 2: a database with no matching .conf must be skipped, not abort the run --- +mkdir -p "$TMP_DIR/backup_missing_conf" +cat > "$TMP_DIR/backup_missing_conf/orphan_db.sql" <<'EOF' +CREATE TABLE t (id INT); +EOF + +run_import scenario2 --user orphan --root "$TMP_DIR/backup_missing_conf" --overwrite allow + +grep -q "SKIPPED orphan_db" "$TMP_DIR/scenario2.stdout" \ + || fail "expected orphan_db (no .conf) to be reported as SKIPPED, got: $(cat "$TMP_DIR/scenario2.stdout")" + +echo "alt_mysql_native_backup_restore_import_test: OK" +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `bash tests/alt_mysql_native_backup_restore_import_test.sh` +Expected: FAIL — the script has no `--root`/`--user` CLI handling yet, so it exits with a usage error or does nothing. + +- [ ] **Step 3: Add discovery, import, and `main()` to `alt_mysql_native_backup_restore.sh`** + +Replace the trailing `if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then ... fi` block at the end of `scripts/da-integration/alt_mysql_native_backup_restore.sh` (added in Task 2 Step 3) with: + +```bash +mysql_import_sql_file() { + local database="$1" file="$2" + if [[ "$file" == *.gz ]]; then + gunzip -c "$file" | mysql_exec "$database" + return "${PIPESTATUS[1]}" + fi + mysql_exec "$database" < "$file" +} + +verify_database_exists() { + local db="$1" + [ -n "$(mysql_exec mysql -Nse "SELECT 1 FROM information_schema.SCHEMATA WHERE SCHEMA_NAME='$(mysql_escape_literal "$db")' LIMIT 1")" ] \ + || die "database was not restored: $db" +} + +rebuild_metadata_for_native_import() { + local db="$1" da_user="$2" + local i owner="" role host + for i in "${!NATIVE_DB_USER_NAMES[@]}"; do + role="${NATIVE_DB_USER_NAMES[$i]}" + host="${NATIVE_DB_USER_HOST[$i]:-localhost}" + [ -n "$owner" ] || owner="$role" + + mysql_exec mysql -e " + INSERT INTO da_plugin_grant_profiles (db_name, role_name, privileges) + VALUES ('$(mysql_escape_literal "$db")', '$(mysql_escape_literal "$role")', '$(mysql_escape_literal "${NATIVE_DB_USER_PRIVS[$i]}")') + ON DUPLICATE KEY UPDATE privileges=VALUES(privileges), updated_at=CURRENT_TIMESTAMP; + " >> "$LOG_FILE" 2>&1 || true + + mysql_exec mysql -e " + INSERT INTO da_plugin_access_hosts (da_user, role_name, host_pattern, note) + VALUES ('$(mysql_escape_literal "$da_user")', '$(mysql_escape_literal "$role")', '$(mysql_escape_literal "$host")', 'restored from native backup/.conf') + ON DUPLICATE KEY UPDATE da_user=VALUES(da_user), note=VALUES(note), updated_at=CURRENT_TIMESTAMP; + " >> "$LOG_FILE" 2>&1 || true + done + + if [ -n "$owner" ]; then + mysql_exec mysql -e " + INSERT INTO da_plugin_database_owners (db_name, da_user, role_name) + VALUES ('$(mysql_escape_literal "$db")', '$(mysql_escape_literal "$da_user")', '$(mysql_escape_literal "$owner")') + ON DUPLICATE KEY UPDATE da_user=VALUES(da_user), role_name=VALUES(role_name), updated_at=CURRENT_TIMESTAMP; + " >> "$LOG_FILE" 2>&1 || true + fi +} + +import_native_database_from_conf() { + local db="$1" conf_file="$2" sql_file="$3" da_user="$4" overwrite="$5" + + parse_native_db_conf "$conf_file" || { log "SKIP $db: conf did not parse"; return 1; } + + local collation="${NATIVE_DB_COLLATION:-}" + if [ -z "$collation" ]; then + log "SKIP $db: no collation parsed" + return 1 + fi + + local i + for i in "${!NATIVE_DB_USER_PLUGIN[@]}"; do + if [ "${NATIVE_DB_USER_PLUGIN[$i]}" != "mysql_native_password" ]; then + log "SKIP $db: unsupported auth plugin '${NATIVE_DB_USER_PLUGIN[$i]}' for user ${NATIVE_DB_USER_NAMES[$i]}" + return 1 + fi + done + + if [ "$overwrite" = "deny" ] && [ -n "$(mysql_exec mysql -Nse "SELECT 1 FROM information_schema.SCHEMATA WHERE SCHEMA_NAME='$(mysql_escape_literal "$db")' LIMIT 1")" ]; then + log "SKIP $db: already exists and overwrite is denied" + return 1 + fi + + backup_alt_database_for_rollback "$db" || { log "SKIP $db: could not create rollback dump"; return 1; } + + if ! mysql_exec mysql -e "DROP DATABASE IF EXISTS $(mysql_quote_identifier "$db");" >> "$LOG_FILE" 2>&1; then + restore_alt_database_from_rollback "$db" || true + log "SKIP $db: could not drop database before import" + return 1 + fi + if ! mysql_exec mysql -e "CREATE DATABASE $(mysql_quote_identifier "$db") CHARACTER SET $(mysql_safe_identifier "$NATIVE_DB_CHARSET") COLLATE $(mysql_safe_identifier "$collation");" >> "$LOG_FILE" 2>&1; then + restore_alt_database_from_rollback "$db" || true + log "SKIP $db: could not create database before import" + return 1 + fi + + local user host hash + for i in "${!NATIVE_DB_USER_NAMES[@]}"; do + user="${NATIVE_DB_USER_NAMES[$i]}" + host="${NATIVE_DB_USER_HOST[$i]:-localhost}" + hash="${NATIVE_DB_USER_HASH[$i]}" + + if ! { + printf "DROP USER IF EXISTS '%s'@'%s';\n" "$(mysql_escape_literal "$user")" "$(mysql_escape_literal "$host")" + printf "CREATE USER '%s'@'%s' IDENTIFIED WITH mysql_native_password AS '%s';\n" \ + "$(mysql_escape_literal "$user")" "$(mysql_escape_literal "$host")" "$(mysql_escape_literal "$hash")" + printf "GRANT %s ON %s.* TO '%s'@'%s';\n" \ + "${NATIVE_DB_USER_PRIVS[$i]}" "$(mysql_quote_identifier "$db")" "$(mysql_escape_literal "$user")" "$(mysql_escape_literal "$host")" + } | mysql_exec mysql >> "$LOG_FILE" 2>&1; then + restore_alt_database_from_rollback "$db" || true + log "SKIP $db: could not recreate user ${user}@${host}" + return 1 + fi + done + + if ! mysql_import_sql_file "$db" "$sql_file" >> "$LOG_FILE" 2>&1; then + restore_alt_database_from_rollback "$db" || true + log "SKIP $db: could not import dump" + return 1 + fi + + cleanup_restore_rollback + verify_database_exists "$db" + rebuild_metadata_for_native_import "$db" "$da_user" + mysql_exec mysql -e "FLUSH PRIVILEGES;" >> "$LOG_FILE" 2>&1 || true + return 0 +} + +find_native_pairs_in_dir() { + local root="$1" prefix="$2" + local conf base sql + while IFS= read -r conf; do + [ -n "$conf" ] || continue + base="$(basename "$conf" .conf)" + sql="$(dirname "$conf")/${base}.sql" + [ -r "$sql" ] || continue + printf '%s\t%s\t%s\n' "$base" "$conf" "$sql" + done < <(find "$root" -maxdepth 4 -type f -name "${prefix}_*.conf" 2>/dev/null | sort) +} + +main() { + local da_user="" root="" overwrite="allow" + while [ "$#" -gt 0 ]; do + case "$1" in + --user) da_user="${2:-}"; shift 2 ;; + --root) root="${2:-}"; shift 2 ;; + --overwrite) overwrite="${2:-}"; shift 2 ;; + *) die "unknown argument: $1" ;; + esac + done + + [ -n "$da_user" ] || die "--user is required" + [ -n "$root" ] || die "--root is required" + [[ "$da_user" =~ ^[A-Za-z0-9_][A-Za-z0-9_.-]*$ ]] || die "unsafe DirectAdmin username: $da_user" + [ -d "$root" ] || die "backup root directory does not exist: $root" + + mysql_load_alt_credentials + mysql_ensure_metadata_schema + + local -a handled=() + local base conf sql + while IFS=$'\t' read -r base conf sql; do + [ -n "$base" ] || continue + handled+=("$base") + if [[ "$base" != "${da_user}_"* ]]; then + printf 'SKIPPED %s wrong-user-prefix\n' "$base" + continue + fi + if import_native_database_from_conf "$base" "$conf" "$sql" "$da_user" "$overwrite"; then + printf 'IMPORTED %s\n' "$base" + else + printf 'SKIPPED %s import-failed\n' "$base" + fi + done < <(find_native_pairs_in_dir "$root" "$da_user") + + # Any *.sql belonging to this user's prefix with no matching .conf was not + # covered by the loop above (find_native_pairs_in_dir only yields pairs + # that have both files) - report those explicitly so the caller knows to + # fall back to native-staging-migrate for them too. + local sql_base + while IFS= read -r sql; do + [ -n "$sql" ] || continue + sql_base="$(basename "$sql" .sql)" + [[ "$sql_base" == "${da_user}_"* ]] || continue + local already=0 h + for h in "${handled[@]}"; do + [ "$h" = "$sql_base" ] && { already=1; break; } + done + [ "$already" -eq 1 ] || printf 'SKIPPED %s no-matching-conf\n' "$sql_base" + done < <(find "$root" -maxdepth 4 -type f -name "${da_user}_*.sql" 2>/dev/null | sort) +} + +if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then + [ -r "$COMMON_FILE" ] || die "missing helper: $COMMON_FILE" + # shellcheck source=../setup/mysql_common.sh + source "$COMMON_FILE" + mysql_load_plugin_settings_file "$SETTINGS_FILE" + main "$@" +fi +``` + +Note: the parser test from Task 2 sources this file directly (`source "$SCRIPT"`), which is why all of the above must stay guarded behind the `BASH_SOURCE[0]` check — sourcing must never call `main`. + +- [ ] **Step 4: Run test to verify it passes** + +Run: `bash tests/alt_mysql_native_backup_restore_import_test.sh` +Expected: `alt_mysql_native_backup_restore_import_test: OK` + +- [ ] **Step 5: Re-run Task 2's parser test to confirm it still passes unchanged** + +Run: `bash tests/alt_mysql_native_backup_restore_parse_test.sh` +Expected: `alt_mysql_native_backup_restore_parse_test: OK` + +- [ ] **Step 6: Commit** + +```bash +git add scripts/da-integration/alt_mysql_native_backup_restore.sh tests/alt_mysql_native_backup_restore_import_test.sh +git commit -m "feat: import native backup/.conf+.sql pairs directly into alt-mariadb" +``` + +--- + +## Task 4: Wire the new restore tier into the restore hook + +**Files:** +- Modify: `scripts/da-integration/alt_mysql_user_restore_post_pre_cleanup.sh:142-174` (the `main()` function) +- Test: `tests/alt_mysql_user_restore_post_pre_cleanup_tiers_test.sh` + +**Interfaces:** +- Consumes from Task 3: the CLI contract `alt_mysql_native_backup_restore.sh --user DA_USER --root DIR --overwrite allow|deny`, and its `IMPORTED `/`SKIPPED ` stdout lines. +- Behavior change: `main()` in `alt_mysql_user_restore_post_pre_cleanup.sh` gains a middle tier between the existing plugin-payload check and the existing native-staging-migrate fallback. The native-staging-migrate script still runs afterward, but only to sweep up whatever the new tier reported as `SKIPPED` (or everything, if the new tier found nothing at all) — it must not redo databases the new tier already reported as `IMPORTED`. + +- [ ] **Step 1: Write the failing tier-order test** + +Create `tests/alt_mysql_user_restore_post_pre_cleanup_tiers_test.sh`: + +```bash +#!/bin/bash +set -euo pipefail + +PLUGIN_DIR="$(cd "$(dirname "$0")/.." && pwd)" +SCRIPT="$PLUGIN_DIR/scripts/da-integration/alt_mysql_user_restore_post_pre_cleanup.sh" +TMP_DIR="$(mktemp -d)" +trap 'rm -rf "$TMP_DIR"' EXIT + +fail() { + echo "FAIL: $*" >&2 + exit 1 +} + +mkdir -p "$TMP_DIR/fake_bin" + +# Fake native-backup-restore tier: reports one imported, one skipped database. +cat > "$TMP_DIR/fake_bin/alt_mysql_native_backup_restore.sh" <<'EOF' +#!/bin/bash +echo "CALLED native_backup_restore $*" >> "$FAKE_CALL_LOG" +echo "IMPORTED demo_ok" +echo "SKIPPED demo_bad import-failed" +EOF +chmod 755 "$TMP_DIR/fake_bin/alt_mysql_native_backup_restore.sh" + +# Fake native-staging-migrate tier: just logs that it ran, and with which user. +cat > "$TMP_DIR/fake_bin/da_restore_native_staging_to_alt_mysql.sh" <<'EOF' +#!/bin/bash +echo "CALLED native_staging_migrate $*" >> "$FAKE_CALL_LOG" +EOF +chmod 755 "$TMP_DIR/fake_bin/da_restore_native_staging_to_alt_mysql.sh" + +# No plugin payload present anywhere reachable, so the hook falls past tier 1. +mkdir -p "$TMP_DIR/empty_restore_root" + +CALL_LOG="$TMP_DIR/calls.log" +: > "$CALL_LOG" + +DA_ALT_MYSQL_RESTORE_LOG="$TMP_DIR/restore.log" \ +DA_ALT_MYSQL_NATIVE_BACKUP_RESTORE_BIN="$TMP_DIR/fake_bin/alt_mysql_native_backup_restore.sh" \ +DA_ALT_MYSQL_NATIVE_STAGING_BIN="$TMP_DIR/fake_bin/da_restore_native_staging_to_alt_mysql.sh" \ +DA_ALT_MYSQL_SETTINGS_FILE="$TMP_DIR/missing-settings.conf" \ +FAKE_CALL_LOG="$CALL_LOG" \ +restore_path="$TMP_DIR/empty_restore_root" \ +username="demo" \ +bash "$SCRIPT" demo + +grep -q "CALLED native_backup_restore" "$CALL_LOG" || fail "expected the native-backup-restore tier to be invoked" +grep -q "CALLED native_staging_migrate" "$CALL_LOG" || fail "expected the native-staging-migrate fallback to still run for skipped databases" + +# The staging-migrate fallback must be told which databases were already +# imported, so it never touches demo_ok again. +grep -q "\-\-skip demo_ok" "$CALL_LOG" || fail "expected native-staging-migrate to be told to skip demo_ok, log: $(cat "$CALL_LOG")" + +echo "alt_mysql_user_restore_post_pre_cleanup_tiers_test: OK" +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `bash tests/alt_mysql_user_restore_post_pre_cleanup_tiers_test.sh` +Expected: FAIL — the hook doesn't call the native-backup-restore tier yet, and doesn't honor `DA_ALT_MYSQL_NATIVE_BACKUP_RESTORE_BIN`/`DA_ALT_MYSQL_NATIVE_STAGING_BIN` overrides. + +- [ ] **Step 3: Add the middle tier to `alt_mysql_user_restore_post_pre_cleanup.sh`** + +In `scripts/da-integration/alt_mysql_user_restore_post_pre_cleanup.sh`, add near the top (after the existing `RESTORE_PAYLOAD_SCRIPT`/`NATIVE_MIGRATE_SCRIPT` declarations at lines 8-9): + +```bash +NATIVE_BACKUP_RESTORE_SCRIPT="${DA_ALT_MYSQL_NATIVE_BACKUP_RESTORE_BIN:-$SCRIPT_DIR/alt_mysql_native_backup_restore.sh}" +NATIVE_MIGRATE_SCRIPT="${DA_ALT_MYSQL_NATIVE_STAGING_BIN:-$NATIVE_MIGRATE_SCRIPT}" +``` + +Then replace the body of `main()` (lines 142-174) with: + +```bash +main() { + local da_user enabled overwrite payload native_mode restore_root + da_user="$(detect_da_user "${1:-}")" + enabled="$(read_plugin_setting DA_ALT_MYSQL_RESTORE_ENABLED true)" + native_mode="$(read_plugin_setting DA_ALT_MYSQL_NATIVE_STAGING_MIGRATE true)" + + log "restore hook env: user=$da_user pwd=$(pwd -P) file=${file:-} filename=${filename:-}" + + case "$enabled" in + true|yes|on|1) ;; + *) log "restore hook disabled by DA_ALT_MYSQL_RESTORE_ENABLED=$enabled"; return 0 ;; + esac + + overwrite="$(read_plugin_setting DA_ALT_MYSQL_RESTORE_OVERWRITE allow)" + + if payload="$(detect_payload_dir)"; then + [ -x "$RESTORE_PAYLOAD_SCRIPT" ] || die "restore payload script is not executable: $RESTORE_PAYLOAD_SCRIPT" + log "plugin payload detected: $payload" + "$RESTORE_PAYLOAD_SCRIPT" --user "$da_user" --payload "$payload" --overwrite "$overwrite" + return 0 + fi + + log "plugin payload not found; trying native backup/.conf restore" + + restore_root="$(candidate_dir_if_valid "${restore_path:-}")" \ + || restore_root="$(candidate_dir_if_valid "${restore_dir:-}")" \ + || restore_root="$(candidate_dir_if_valid "${tmpdir:-}")" \ + || restore_root="$(candidate_dir_if_valid "${backup_path:-}")" \ + || restore_root="$(pwd -P)" + + local -a imported=() + local -a skip_args=() + if [ -x "$NATIVE_BACKUP_RESTORE_SCRIPT" ] && [ -d "$restore_root" ]; then + while IFS= read -r line; do + [ -n "$line" ] || continue + log "native backup restore: $line" + case "$line" in + IMPORTED\ *) + imported+=("${line#IMPORTED }") + ;; + esac + done < <("$NATIVE_BACKUP_RESTORE_SCRIPT" --user "$da_user" --root "$restore_root" --overwrite "$overwrite") + else + log "native backup restore script unavailable or restore root not found: $NATIVE_BACKUP_RESTORE_SCRIPT / $restore_root" + fi + + for db in "${imported[@]}"; do + skip_args+=("--skip" "$db") + done + + log "native backup restore imported: ${imported[*]:-}; considering native staging migration for the rest" + case "$native_mode" in + true|yes|on|1) + [ -x "$NATIVE_MIGRATE_SCRIPT" ] || die "native staging migration script is not executable: $NATIVE_MIGRATE_SCRIPT" + "$NATIVE_MIGRATE_SCRIPT" --user "$da_user" "${skip_args[@]}" + ;; + *) + log "native staging migration disabled; account restore continues without further alt-mysql DB migration" + ;; + esac +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `bash tests/alt_mysql_user_restore_post_pre_cleanup_tiers_test.sh` +Expected: `alt_mysql_user_restore_post_pre_cleanup_tiers_test: OK` + +- [ ] **Step 5: Teach `da_restore_native_staging_to_alt_mysql.sh` to accept `--skip DB` and honor it** + +In `scripts/da-integration/da_restore_native_staging_to_alt_mysql.sh`, add a new variable near the top (after `CLEANUP_POLICY=""` at line 12): + +```bash +declare -a SKIP_DATABASES=() +``` + +Update the argument-parsing `while` loop (lines 35-43) to add a case: + +```bash + --skip) SKIP_DATABASES+=("${2:-}"); shift 2 ;; +``` + +Update `list_native_databases()` (lines 91-98) to exclude skipped databases: + +```bash +list_native_databases() { + local db + native_query information_schema " + SELECT SCHEMA_NAME + FROM SCHEMATA + WHERE SCHEMA_NAME LIKE '$(mysql_escape_literal "$DA_USER")\\_%' ESCAPE '\\' + ORDER BY SCHEMA_NAME + " | grep -Ev '^(information_schema|performance_schema|mysql|sys)$' | while IFS= read -r db; do + [ -n "$db" ] || continue + local skipped=0 candidate + for candidate in "${SKIP_DATABASES[@]}"; do + [ "$candidate" = "$db" ] && { skipped=1; break; } + done + [ "$skipped" -eq 1 ] || printf '%s\n' "$db" + done || true +} +``` + +- [ ] **Step 6: Run the full restore-hook test suite to confirm no regression** + +Run: +```bash +bash tests/alt_mysql_user_restore_post_pre_cleanup_tiers_test.sh +bash tests/da_restore_archive_traversal_test.sh +bash tests/alt_mysql_restore_payload_test.sh +bash tests/restore_rollback_safety_test.sh +``` +Expected: all four print their own `... : OK` line. + +- [ ] **Step 7: Commit** + +```bash +git add scripts/da-integration/alt_mysql_user_restore_post_pre_cleanup.sh scripts/da-integration/da_restore_native_staging_to_alt_mysql.sh tests/alt_mysql_user_restore_post_pre_cleanup_tiers_test.sh +git commit -m "feat: restore native backup/.conf pairs before falling back to native staging" +``` + +--- + +## Task 5: Update the design doc to match what was actually built + +> **Note for the controller:** `mysql-installer/da-integration.md` lives one directory above this +> repo's root (`alt-mysql/mysql-installer/`, not inside the `alt-mysql/src` git repository that +> Tasks 1-4 are implemented in), and is not tracked by that repository or shipped in the plugin +> package. Do **not** dispatch an implementer/reviewer subagent pair or a git commit for this task +> — apply the edit directly after Tasks 1-4's code is merged, since it is a plain prose update with +> no test surface. + +**Files:** +- Modify: `../mysql-installer/da-integration.md` (path relative to the `src/` repo root; Phase 5B and Phase 7 sections) + +**Interfaces:** None — documentation only. + +- [ ] **Step 1: Replace the Phase 5B description** + +In `mysql-installer/da-integration.md`, find the `## Phase 5B: External DirectAdmin Backups Without Plugin Payload` section (currently describing "native staging then migrate" as the primary mechanism for external backups). Replace its `### Problem`/`### Recommended Policy` prose to state, as the primary mechanism: the restore hook (`alt_mysql_user_restore_post_pre_cleanup.sh`) now first calls `alt_mysql_native_backup_restore.sh`, which parses `backup/.conf` + `backup/.sql` pairs directly out of the archive/extraction root and imports each database straight into `alt-mariadb` — no native MariaDB staging involved. `da_restore_native_staging_to_alt_mysql.sh` (the existing native-staging-then-migrate flow) remains, unchanged in its own logic, as the per-database fallback only for databases the new tier reports as `SKIPPED` (missing/malformed `.conf`, unsupported auth plugin, or a genuinely foreign non-DA-native backup). + +- [ ] **Step 2: Update the Phase 7 "CustomBuild MariaDB Staging Role" framing** + +In the same file's `## Phase 7` section, add a note that CustomBuild's native MariaDB no longer needs to be usable as a restore staging target for the common case — it is now a fallback path only, exercised solely when `alt_mysql_native_backup_restore.sh` reports a database as `SKIPPED`. It must still be kept running so DirectAdmin core's own login/version check has a legacy-license-safe target to talk to. + +- [ ] **Step 3: Commit** + +```bash +git add mysql-installer/da-integration.md +git commit -m "docs: reflect native-conf-based restore as the primary external-backup path" +``` diff --git a/docs/superpowers/specs/2026-07-05-restore-without-native-staging-design.md b/docs/superpowers/specs/2026-07-05-restore-without-native-staging-design.md new file mode 100644 index 0000000..500e338 --- /dev/null +++ b/docs/superpowers/specs/2026-07-05-restore-without-native-staging-design.md @@ -0,0 +1,139 @@ +# Design: Restore Directly Into alt-mariadb Without Native Staging + +Date: 2026-07-05 +Status: Approved by user, ready for implementation planning + +## Goal + +Make DirectAdmin account restores (both plugin-origin and external/foreign backups) land +databases, database users, password hashes, and grants directly in `alt-mariadb`, without +ever writing customer data into the CustomBuild-managed native MariaDB instance as an +intermediate step. + +## Background + +The plugin already implements a "separated" architecture (see +`mysql-installer/da-integration.md`): `alt-mariadb` is the real database service, CustomBuild's +native MariaDB is kept installed only for DirectAdmin-native compatibility. Backup creation +(`alt_mysql_user_backup_compress_pre.sh`) already augments every native DA backup with a +`backup/alt_mysql/` payload containing the real `alt-mariadb` data, and restoring that payload +(`alt_mysql_restore_payload.sh`) is already a direct, single-hop restore into `alt-mariadb`. + +The remaining gap was restoring **external backups** — archives made on a server without this +plugin, which therefore lack `backup/alt_mysql/manifest.json`. The existing fallback, +`da_restore_native_staging_to_alt_mysql.sh`, assumes DirectAdmin has already restored the +account's databases into the native MariaDB instance, then dumps and migrates them into +`alt-mariadb`, then cleans up the native staged copies. + +## Rejected alternative: point DirectAdmin at alt-mariadb directly + +Early in this design process, a simpler-looking alternative was considered: install +`alt-mariadb` on port 3306, rewrite DirectAdmin's own `/usr/local/directadmin/conf/mysql.conf` +and `my.cnf` to point at it, and set `mysql_inst=no` in CustomBuild's `options.conf` so +CustomBuild stops managing/version-gating the database. This was rejected: + +- DirectAdmin's **legacy code-base license** enforces a hard version ceiling (MariaDB 10.6 / + MySQL 8.0) at DA core login time, against whatever `mysql.conf` points to — independent of + `mysql_inst`. This was confirmed by the user's own observed error: *"legacy code-base does + not support this MySQL version, downgrade MySQL or upgrade DirectAdmin license."* + `mysql_inst=no` only silences CustomBuild's own build-time/EOL-notification checks; it does + not bypass this license gate. +- `alt-mariadb` is installed specifically to run versions newer than what CustomBuild/legacy DA + supports (10.11, 11.4, 11.8 — see `install_db.sh`). Pointing DA at it would either force + capping `alt-mariadb` at 10.6 (defeating the point) or break DA login outright. +- It provides no benefit the design below doesn't already achieve without that risk. + +**Decision: no `INSTALL_MODE` toggle is added to `install_db.sh`. `alt-mariadb` is never made +reachable through DirectAdmin's own `mysql.conf`/`my.cnf`, at any version.** + +## Core design: self-service restore from the archive's native database files + +Ground truth for DirectAdmin's native backup format was obtained by inspecting a real account +backup (`example_backup/psy/backup/`), not from public docs (this format is not documented by +DirectAdmin). For each native database, the backup contains a matching pair, flat under +`backup/`: + +- `backup/.sql` — a plain `mariadb-dump`/`mysqldump` output (table structure + data). + It contains **no `CREATE DATABASE`/`USE` statement** — the target database must already exist + before this is imported. +- `backup/.conf` — a flat `key=value` text file: + ``` + accesshosts=0=localhost + db_collation=DEFAULT_CHARACTER_SET_NAME=utf8mb3&DEFAULT_COLLATION_NAME=utf8mb3_general_ci&SCHEMA_NAME= + =accesshosts=&alter_priv=Y&...&passwd=%2A&plugin=mysql_native_password&...&update_priv=Y + ``` + - Line 1: indexed list of database-level access hosts. + - Line 2: `db_collation` — charset/collation needed to `CREATE DATABASE` correctly. + - Line 3+: one line per MySQL user granted on this database, URL-encoded query-string + fields, keyed by username. Contains the real `mysql_native_password` auth hash (not a + plaintext password), the auth plugin, per-privilege `Y`/`N` flags (the same set of + privilege columns `db_privilege_profile()` in `da_restore_native_staging_to_alt_mysql.sh` + already reverse-engineers from `mysql.db`), and the granted host. + +This pair is a complete, self-contained record: charset/collation, every user's exact password +hash, full grant set, and access host(s) — sufficient to recreate the database and its users in +`alt-mariadb` without DirectAdmin's cooperation or its native engine being involved at all. + +### Restore flow + +The post-restore hook (`alt_mysql_user_restore_post_pre_cleanup.sh`) already receives the +original archive path (`$filename`/`$file`) and already checks for a plugin payload first. This +design adds a second tier before falling back to native-staging-migrate: + +1. **Plugin payload** (`backup/alt_mysql/manifest.json` present) — unchanged, existing direct + restore via `alt_mysql_restore_payload.sh`. +2. **Native `.conf`+`.sql` pairs present for a database matching the target user's prefix** — + new: extract both files for that database directly from the archive, independent of + whatever DirectAdmin itself did with its own native "database"/"database_data" restore + items. Per database: + - Parse `db_collation` → `CREATE DATABASE` with the correct charset/collation. + - Parse each per-user line → `CREATE USER ... IDENTIFIED WITH mysql_native_password AS + ''` at the parsed host, then translate the `Y`/`N` privilege flags into a `GRANT` + statement (reuse/adapt the existing privilege-flag mapping from + `db_privilege_profile()`/`rebuild_metadata_for_db()`). + - Import the `.sql` dump into the created database. + - Rebuild the plugin's ownership/grant-profile/access-host metadata tables, same as the + existing native-staging path does today. + - Verify the database exists and is non-empty before considering it restored, with the same + rollback-dump-and-restore-on-failure safety already implemented in + `alt_mysql_restore_payload.sh`/`da_restore_native_staging_to_alt_mysql.sh`. +3. **Neither available for a given database** (malformed `.conf`, missing pair, or a genuinely + foreign non-DA-native backup) — fall back to `da_restore_native_staging_to_alt_mysql.sh` for + that database only, unchanged from today. This keeps the existing staging-then-migrate logic + as a safety net, not the primary path. + +Because the hook self-serves from the archive, **no restore-initiation wrapper or forced +item-deselection (`select0=item`) is needed**. Whatever DirectAdmin does internally with its own +native "database" restore item is irrelevant to this flow — if it stages a redundant copy into +the native engine, that's harmless noise, not something the plugin needs to migrate from. + +### What this simplifies going forward + +- `da_restore_native_staging_to_alt_mysql.sh` becomes a per-database fallback, not the primary + external-backup path. It is not removed. +- CustomBuild's native MariaDB no longer needs to be usable as a restore staging target for the + common case. It only needs to keep existing/running so DirectAdmin core's own login/version + check has a legacy-safe target to talk to (per the rejected-alternative section above) — its + role in `mysql-installer/da-integration.md` Phase 7 shrinks accordingly. +- Backup creation (Phase 4) is unaffected. + +## Risks / open items for implementation planning + +- **The `.conf` format is not documented by DirectAdmin.** It was reverse-engineered from one + real backup. Treat it defensively: validate structure before trusting it, and fail closed to + the per-database native-staging fallback rather than assuming the format holds across + DirectAdmin/CustomBuild versions. +- Multiple users can be granted on one database (multiple `=...` lines) — the parser + must handle N users per `.conf`, not assume exactly one. +- The exact temporary extraction directory/working path DirectAdmin uses during restore should + still be logged and confirmed empirically on a disposable account as the first implementation + task, consistent with how the backup-side hook was already built. +- Password hashes are `mysql_native_password` in the observed example; the parser should not + assume this is the only plugin value that can appear and should fail safely (fall back) for + unrecognized auth plugins rather than guessing. + +## Explicitly out of scope + +- No `INSTALL_MODE=separated|simple` toggle in `install_db.sh`. +- No changes to `install_db.sh`'s port/socket/datadir validation. +- No rewriting of DirectAdmin's own `mysql.conf`/`my.cnf`.