Backup and restore

Complete guide for backing up and restoring a production Repod instance.

---

Overview

Since the migration to PostgreSQL (commit 6a711dc), all relational data (users, manifest index, inventory, install jobs, package search, ssh_known_hosts SSH fingerprints) lives in a single PostgreSQL database, accessible via DATABASE_URL. The /repos/ filesystem now holds only non-relational data.

Item	Location	Criticality	Notes
Application database (users, manifests, inventory, CVE, jobs)	PostgreSQL (DATABASE_URL, postgres_data volume)	🔴 Critical	Backed up via pg_dump (custom format, restorable with pg_restore)
Configuration	repos/settings.json	🔴 Critical	LDAP, OIDC, CVE policy, mirror, backup
Package manifests	repos/manifests/	🟠 Important	Metadata for each package (cache — rebuildable from pool/)
Audit logs	repos/audit/	🟠 Important	Regulatory traceability
GPG keys	repos/gnupg/	🔴 Critical	Loss means re-signing every package
Package pool	repos/pool/	🟡 Optional	.deb/.rpm/.apk — can be rebuilt from sources, but costly
CVE decisions / KEV-EPSS cache	repos/security/	🟠 Important	RSSI review decisions and CVE caches

Two backup mechanisms are available and produce the same archive format (repod_backup_TIMESTAMP.tar.gz):

1. Integrated backup (recommended) — triggerable by an admin via the API or the UI (Settings → Backups), and schedulable via the scheduler (backup_daily, cron job services/backup.py).
2. backup.sh script — at the project root, for external backups (host cron, NAS).

---

Integrated backup (admin / scheduler)

services/backup.py:create_backup() produces a .tar.gz archive in BACKUP_DIR (default /repos/backups), containing:

repod_backup_YYYYMMDD_HHMMSS/
├── postgres.dump      ← pg_dump -F c (custom format, restorable via pg_restore)
├── settings.json
├── audit/
├── security/
├── manifests/
├── gnupg/             ← permissions preserved
└── pool/

create_backup() raises an explicit error if DATABASE_URL or pg_dump is missing — there is never a silently "incomplete" archive.

Triggering a manual backup

TOKEN=$(curl -s -X POST http://localhost:8000/api/v1/auth/token \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"YourPassword"}' | jq -r .access_token)

# Create a backup
curl -X POST -H "Authorization: Bearer $TOKEN" http://localhost:8000/api/v1/backup/

# List existing backups
curl -H "Authorization: Bearer $TOKEN" http://localhost:8000/api/v1/backup/

# Download an archive
curl -H "Authorization: Bearer $TOKEN" \
  -o repod_backup_20260601_020000.tar.gz \
  http://localhost:8000/api/v1/backup/repod_backup_20260601_020000.tar.gz

# Delete an archive
curl -X DELETE -H "Authorization: Bearer $TOKEN" \
  http://localhost:8000/api/v1/backup/repod_backup_20260601_020000.tar.gz

All of these endpoints require the admin role.

Automatic scheduling

In Settings → Backups:

• backup.enabled — enables the backup_daily job (APScheduler)
• backup.hour / backup.minute — time of execution (default 04:30)
• backup.retention_count — number of archives kept (apply_retention() removes the oldest ones beyond this count)

The job is logged to the audit log (BACKUP_CREATE, SUCCESS/FAILURE).

---

backup.sh script (external backup)

For backups triggered from the host (cron, NAS), backup.sh at the project root backs up the same items via pg_dump:

# Backup into ./backups/ (default directory)
DATABASE_URL=postgresql://repod:CHANGE_ME@localhost:5432/repod ./backup.sh

# Backup to a NAS or external volume
DATABASE_URL=postgresql://... BACKUP_DIR=/mnt/nas/repod ./backup.sh

# Dry-run mode (no writes)
./backup.sh --dry-run

# Custom retention (default: 30 days)
DATABASE_URL=postgresql://... BACKUP_RETENTION_DAYS=90 BACKUP_DIR=/mnt/nas/repod ./backup.sh

backup.sh requires DATABASE_URL and pg_dump — it fails explicitly (fail) if either is missing, rather than producing an archive with no database in it.

What the script backs up

repod_backup_YYYYMMDD_HHMMSS.tar.gz
├── postgres.dump      ← pg_dump -F c (all relational data)
├── pool/               ← .deb / .rpm / .apk
├── settings.json
├── audit/              ← every *.jsonl file
├── security/           ← CVE decisions, KEV/EPSS caches
├── manifests/          ← metadata for each package
└── gnupg/              ← GPG keyring (700 permissions preserved)

Installations predating the PostgreSQL migration

If repos/auth/users.db still exists (a pre-migration installation that was never cleaned up), backup.sh backs it up too (users.db, via sqlite3 .backup if available) to avoid losing legacy data. Current installations don't have this file.

Scheduling with cron

# Daily backup at 2:00 AM with 90-day retention
0 2 * * * cd /opt/repod && DATABASE_URL=postgresql://repod:CHANGE_ME@localhost:5432/repod \
  BACKUP_DIR=/mnt/nas/repod BACKUP_RETENTION_DAYS=90 ./backup.sh >> /var/log/repod-backup.log 2>&1

Connecting to PostgreSQL from the host

The db (PostgreSQL) service is not published on the host by default. For a host-side backup.sh run to connect, either temporarily publish port 5432 of the db container, or run pg_dump from inside the repod-db container instead:

docker exec repod-db pg_dump -U repod -F c -f /tmp/postgres.dump repod
docker cp repod-db:/tmp/postgres.dump ./postgres.dump

GPG encryption of the backup (recommended in production)

# Generate a dedicated backup key (one-time setup)
gpg --batch --generate-key <<EOF
%no-protection
Key-Type: RSA
Key-Length: 4096
Name-Real: Repod Backup Key
Name-Email: [email protected]
Expire-Date: 2y
EOF

# Encrypt the archive after backup
ARCHIVE=$(ls -t backups/repod_backup_*.tar.gz | head -1)
gpg --recipient [email protected] --encrypt "$ARCHIVE"
# Result: repod_backup_TIMESTAMP.tar.gz.gpg
rm "$ARCHIVE"

# Store the private GPG key separately from the backup (vault, KMS…)
gpg --export-secret-keys [email protected] | gpg --symmetric --output backup-key.gpg.enc

---

Restore procedure

Stop the backend before any restore

pg_restore --clean drops and recreates database objects. Stop the backend-api container (or the whole stack) before restoring, to avoid concurrent writes.

Step 1 — Stop the services

cd /opt/repod
docker compose down

Step 2 — Extract the archive

ARCHIVE="repod_backup_20260601_020000.tar.gz"
BACKUP_NAME="${ARCHIVE%.tar.gz}"
RESTORE_TMP="/tmp/repod_restore"

# If GPG-encrypted, decrypt first
# gpg --decrypt "$ARCHIVE.gpg" > "$ARCHIVE"

mkdir -p "$RESTORE_TMP"
tar -xzf "$ARCHIVE" -C "$RESTORE_TMP"
ls "$RESTORE_TMP/$BACKUP_NAME/"

Step 3 — Start only PostgreSQL and restore the database

cd /opt/repod
docker compose up -d db
sleep 5   # wait for PostgreSQL to accept connections

# pg_restore --clean recreates objects; --if-exists avoids errors if the database is empty
docker exec -i repod-db pg_restore -U repod -d repod --clean --if-exists \
  < "$RESTORE_TMP/$BACKUP_NAME/postgres.dump"

Step 4 — Restore configuration and /repos/ data

REPOS_DIR="/opt/repod/repos"
RESTORE_DIR="$RESTORE_TMP/$BACKUP_NAME"

# settings.json — back up the current one as a precaution
cp "$REPOS_DIR/settings.json" "$REPOS_DIR/settings.json.before-restore" 2>/dev/null || true
cp "$RESTORE_DIR/settings.json" "$REPOS_DIR/settings.json"

# Audit logs — merge without overwriting (keep post-incident logs)
[ -d "$RESTORE_DIR/audit" ] && cp -rn "$RESTORE_DIR/audit/." "$REPOS_DIR/audit/"

# Manifests
[ -d "$RESTORE_DIR/manifests" ] && cp -r "$RESTORE_DIR/manifests/." "$REPOS_DIR/manifests/"

# CVE decisions / security caches
[ -d "$RESTORE_DIR/security" ] && cp -r "$RESTORE_DIR/security/." "$REPOS_DIR/security/"

# Package pool
[ -d "$RESTORE_DIR/pool" ] && cp -r "$RESTORE_DIR/pool/." "$REPOS_DIR/pool/"

# GPG keys — preserve permissions
if [ -d "$RESTORE_DIR/gnupg" ]; then
    cp -rp "$RESTORE_DIR/gnupg/." "$REPOS_DIR/gnupg/"
    chmod 700 "$REPOS_DIR/gnupg"
fi

Step 5 — Regenerate the repository trees

The dists/ (APT), rpm/ (RPM), and apk/ (Alpine) trees are not included in the backup — they are regenerated from pool/ and the restored PostgreSQL database:

cd /opt/repod
docker compose up -d
TOKEN=$(curl -s -X POST http://localhost:8000/api/v1/auth/token \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"YourPassword"}' | jq -r .access_token)

curl -X POST http://localhost:8000/api/v1/distributions/init \
  -H "Authorization: Bearer $TOKEN"

Step 6 — Verify

curl -s http://localhost:8000/health | python3 -m json.tool

# Verify admin login
curl -s -X POST http://localhost:8000/api/v1/auth/token \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"YourPassword"}'

---

Verifying backup integrity

Verify that a backup is valid without performing a restore:

ARCHIVE="repod_backup_20260601_020000.tar.gz"
RESTORE_TMP=$(mktemp -d)

tar -xzf "$ARCHIVE" -C "$RESTORE_TMP"
BACKUP_DIR="$RESTORE_TMP/$(basename "$ARCHIVE" .tar.gz)"

# Verify the PostgreSQL dump (lists contents without restoring)
if pg_restore --list "$BACKUP_DIR/postgres.dump" > /dev/null 2>&1; then
    echo "✅ postgres.dump: valid archive"
else
    echo "❌ postgres.dump: CORRUPTED"
fi

# settings.json
if python3 -m json.tool "$BACKUP_DIR/settings.json" > /dev/null 2>&1; then
    echo "✅ settings.json: valid JSON"
else
    echo "❌ settings.json: invalid JSON"
fi

# Manifests
MANIFEST_COUNT=$(find "$BACKUP_DIR/manifests" -name "*.json" 2>/dev/null | wc -l)
echo "📦 Manifests: $MANIFEST_COUNT files"

# Audit logs
AUDIT_COUNT=$(find "$BACKUP_DIR/audit" -name "*.jsonl" 2>/dev/null | wc -l)
echo "📋 Audit logs: $AUDIT_COUNT files"

# GPG keys
if [ -d "$BACKUP_DIR/gnupg" ]; then
    echo "🔑 GPG: directory present"
else
    echo "⚠️  GPG: missing from backup"
fi

rm -rf "$RESTORE_TMP"

---

Recommended backup strategy (3-2-1)

The 3-2-1 rule is the industry standard:

Rule	Description	Recommended implementation
3 copies	At least 3 copies of the data	Local + NAS + cloud
2 different media	On 2 distinct media types	Local disk + NAS or tape
1 off-site	At least 1 copy off-site	S3 / GPG-encrypted cloud

# Example: shipping to S3 after backup
ARCHIVE=$(ls -t backups/repod_backup_*.tar.gz | head -1)
aws s3 cp "$ARCHIVE" s3://my-backup-bucket/repod/

---

Disaster Recovery Plan (DRP)

Recovery objectives

Indicator	Target value	Notes
RPO (Recovery Point Objective)	≤ 24 hours	Daily backup recommended
RTO (Recovery Time Objective)	≤ 2 hours	Full restore time

Covered scenarios

Scenario 1 — PostgreSQL database corruption

Symptom: backend-api fails to start with a SQLAlchemy/Alembic error, or GET /health returns checks.critical.auth_db / manifest_db as failing.

ls -lt backups/repod_backup_*.tar.gz

# Follow steps 1 to 3 of the restore procedure above
# (stop services, extract, pg_restore --clean --if-exists)

Scenario 2 — Loss of GPG keys

Critical situation

Losing the GPG keys makes already-signed packages unverifiable by apt/dnf/zypper/apk clients. Restoring the keys from the backup is the only solution that avoids re-signing every package.

ARCHIVE="repod_backup_20260601_020000.tar.gz"
tar -xzf "$ARCHIVE" --strip-components=1 -C /tmp/ "*/gnupg"
cp -rp /tmp/gnupg/. repos/gnupg/
chmod 700 repos/gnupg
docker compose restart backend-api

Scenario 3 — Total disaster (server lost)

1. Provision a new server (see the deployment guide)
2. Install Docker + Docker Compose
3. Clone the Repod repository
4. Create .env and backend.env from the .example files (DATABASE_URL, POSTGRES_PASSWORD, JWT_SECRET_KEY, SETTINGS_ENCRYPTION_KEY, WEBHOOK_SECRET, REPO_FORMAT)
5. Start only db, then follow steps 2 to 5 of the restore procedure above
6. Verify the health check and admin accounts
7. Update DNS / reverse proxy if the IP address has changed
8. Notify the teams (RSSI, ops) and document the incident

Scenario 4 — Configuration rollback

ARCHIVE="repod_backup_20260601_020000.tar.gz"
tar -xzf "$ARCHIVE" --strip-components=1 -C /tmp/ "*/settings.json"
cp repos/settings.json repos/settings.json.rollback-$(date +%Y%m%d)
cp /tmp/settings.json repos/settings.json
docker compose restart backend-api

---

Restore testing

Test your backup at least once per quarter

An untested backup is a backup whose real state is unknown.

# Test procedure on a staging server
# 1. Copy the latest backup to the staging server
scp /mnt/nas/repod/repod_backup_latest.tar.gz staging:/tmp/

# 2. On staging, follow the restore procedure above

# 3. Verify the service is operational
ssh staging "curl -s http://localhost:8000/health"

# 4. Verify that an admin login works
# 5. Verify that a package is visible in the UI
# 6. Document the test (date, duration, result)

---

Backup environment variables

Variable	Default	Description
DATABASE_URL	—	Required by backup.sh and the integrated backup — PostgreSQL connection string
BACKUP_DIR	/repos/backups (integrated) / ./backups (backup.sh)	Destination directory for archives
REPOS_DIR	/repos (integrated) / ./repos (backup.sh)	Source directory for non-relational data
BACKUP_RETENTION_DAYS	30 (backup.sh)	Retention period for local archives (0 = no purge)
settings.json["backup"]["retention_count"]	7 (integrated backup)	Number of archives kept by apply_retention()

---

Operational checklist

Initial setup

• Enable backup.enabled in Settings → Backups (daily integrated backup)
• And/or configure a daily cron job for backup.sh with DATABASE_URL set
• Set BACKUP_DIR to an external volume (NAS, S3...) for backup.sh
• Configure GPG encryption of the backup
• Document the location of the backup GPG key (vault, KMS)
• Set retention according to the GDPR policy
• Test a full restore on a staging environment
• Document the actual RTO/RPO measured during the test

Monthly check

• Verify that backups (integrated and/or cron) ran successfully
• Verify the integrity of at least 1 archive (pg_restore --list)
• Verify the available disk space at the backup destination
• Verify that rotation of old archives is working

In case of an incident

• Identify the last valid backup before the incident
• Assess the RPO (data lost since that backup)
• Follow the restore procedure appropriate to the scenario
• Document the incident and the restore in the incident register
• If personal data was affected: notify the relevant DPA within 72h (GDPR Art. 33)