NixOS (10): System State, Backups and Recovery

Motivation

#

The earlier posts covered system configuration, user environments and secrets. Keeping configuration makes reinstalling a computer easier, but documents, photos, databases and application data do not return with nixos-rebuild.

This is similar to the distinction discussed in the Docker volume post: software can be deployed again, while data needs separate storage. The private decryption keys in the previous post also need protection outside configuration.

Here we will identify the state worth saving, then practice a backup and restore with restic. The example uses disposable files in a temporary directory. It copies no personal directories and changes no existing services.

System Configuration and Data

#

Different content needs different recovery methods:

The Nix store mainly holds software and build outputs, so it usually need not be the main part of a file backup. Rebuilding still depends on source availability, download locations and build resources. A lock file cannot guarantee that every input will remain available. Consider retaining required store closures and usable caches for software that is difficult to retrieve or build again.

Home directories, service data directories and parts of /var/lib may contain irreplaceable data. Treating all of /var/lib as cache is unsuitable, as is restoring an entire old system directory indiscriminately. First identify what each service writes, then choose the backup scope.

system.stateVersion selects defaults compatible with older state. It is neither a backup nor the installed release number. Keep it when updating inputs. See the NixOS stateVersion option.

Backup Methods

#

Common methods serve different purposes:

Snapshots can also be replicated to another disk or device. Important data benefits from a copy independent of the everyday computer, with offsite storage considered too. Offline or immutable copies may be useful depending on what a stolen backup account, accidental deletion or ransomware could affect.

Backing Up Files with restic

#

restic saves files in an encrypted repository, retains multiple snapshots and reuses existing data. We will use a local directory to learn the commands. The repository and source remain on one computer, so this is a practice exercise rather than a complete backup arrangement.

Preparing Demonstration Files

#

Open a shell containing restic:

1

nix shell nixpkgs#restic

In the same shell, run:

1
2
3
4
5
6
7
8

umask 077
backup_demo=$(mktemp -d)
mkdir -p "$backup_demo/source" "$backup_demo/repository"
printf 'version one\n' > "$backup_demo/source/note.txt"
printf 'disposable-demo-password\n' > "$backup_demo/password"
export RESTIC_REPOSITORY="$backup_demo/repository"
export RESTIC_PASSWORD_FILE="$backup_demo/password"
export RESTIC_CACHE_DIR="$backup_demo/cache"

backup_demo is a new temporary directory. The password is only a demonstration value. Use a separate strong password for real backups and keep recovery access in another protected location. Keeping this password alongside the repository makes the exercise convenient; do not copy that arrangement into a real backup.

RESTIC_REPOSITORY selects the repository and RESTIC_PASSWORD_FILE selects its password file. The password itself is not passed as a command argument. See the restic initialization documentation.

Creating a Backup

#

1
2
3
4

cd "$backup_demo"
restic init
restic backup --host demo --tag tutorial source
restic snapshots --host demo --tag tutorial

init creates a new repository; an existing repository needs no repeated initialization. backup saves the demonstration source directory. The host and tag distinguish these snapshots. Check command success and use snapshots to confirm the expected time and paths.

For real data, review output and exit status for missing directories, permission problems and files changing while being read. A scheduled task having run does not prove every file was saved. See the restic backup documentation.

Restoring and Comparing

#

Change the source file without making a second backup:

1

printf 'version two\n' > "$backup_demo/source/note.txt"

Restore the earlier snapshot into a new directory:

1
2
3
4
5

restic restore latest --host demo --tag tutorial \
  --target "$backup_demo/restore"
restored_note="$backup_demo/restore/source/note.txt"
printf 'version one\n' > "$backup_demo/expected.txt"
cmp "$backup_demo/expected.txt" "$restored_note"

We entered the demonstration directory and backed up the relative path source, so the restored file is at source/note.txt underneath the target. Backups of absolute paths produce a different directory hierarchy; check the paths recorded in the snapshot. No output from cmp, with exit status zero, means the restored file still contains the backed-up version one, rather than the current version two.

Here latest is filtered by host and tag, and there is only one demonstration snapshot. For a real restore, list snapshots first and confirm the exact date, paths and snapshot ID. Prepare a separate destination before restoring, rather than overwriting live documents or databases. See the restic restore documentation.

Repository Checks and Retention

#

1
2

restic check
restic check --read-data

A normal check examines repository structure. --read-data additionally reads and verifies backup data, consuming more time and bandwidth. Neither replaces a real restore and application test. See the repository checking documentation.

Keeping multiple dates requires planning storage. For example, preview a retention policy:

1
2

restic forget --host demo --tag tutorial \
  --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --dry-run

This uses --dry-run and deletes no snapshots. The numbers are illustrative, rather than requirements for everyone. forget removes snapshot records, while prune reclaims unreferenced data. Confirm scope and recovery needs before applying either. See the restic retention documentation for grouping and time calculations.

Database and Service Backups

#

The exercise backs up an ordinary text file. Copying a running database’s data directory can produce an unusable backup. Use its supported export, physical backup or consistent snapshot method. PostgreSQL, for example, provides tools such as pg_dump; see its official backup documentation.

Some services use both a database and uploaded files. Save compatible points in time and record the application version, restore order and required secrets. Restoring just the database or just the upload directory may leave records and files inconsistent.

Stopping writes before backing up may be simplest for a small service that can pause. Services that must stay available need their supported backup procedures. Filesystem snapshots can shorten a pause during file copying, but a consistent filesystem snapshot does not automatically provide a complete application backup.

Test restoration in an isolated instance and check accounts, records, attachments and main functions. Keep the test instance from sending mail, running scheduled jobs or writing back to the live system.

Scheduled Backups in NixOS

#

After manual backup and restore work, declare the task in NixOS. This fragment needs adaptation before use:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

{
  services.restic.backups.documents = {
    repository = "/mnt/backup/restic-documents";
    passwordFile = "/run/secrets/restic-password";
    paths = [ "/home/alice/Documents" ];
    initialize = false;
    timerConfig = {
      OnCalendar = "daily";
      Persistent = true;
    };
  };
}

alice is fictional. /mnt/backup represents prepared, independent backup storage, and a runtime secrets mechanism supplies the password file. The fragment does not create these conditions and cannot be applied unchanged on every computer. Check paths, mounts and secret availability in the actual configuration.

initialize = false requires an existing initialized repository. If the real storage is not mounted, the job must not quietly create a repository in an identically named directory on the system disk. Ensure the target mount is available, adding mount dependencies and checks to the generated service where needed. Remote repositories need network access, credentials and permissions instead.

OnCalendar = "daily" schedules daily runs. Persistent allows a missed calendar trigger to run when the timer becomes active again. It cannot recover the data as it existed at every missed time. Review behavior during sleep, shutdown and disconnected storage against how you use the computer.

See the NixOS restic module for its options and generated services.

After applying, inspect:

1
2
3

systemctl list-timers restic-backups-documents.timer
systemctl status restic-backups-documents.service --no-pager
sudo journalctl -u restic-backups-documents.service -b --no-pager

Check the last successful run, exit status and actual snapshots, beyond the presence of the timer. Review logs for personal paths and sensitive content before sharing. Schedule automatic cleanup and complete data checks separately once the basic job is reliable.

System Recovery

#

A useful preparation order is:

1. Confirm access to installation media, configuration, backup repositories and required keys.
2. Check disks, partitions, filesystems and mounts again; avoid copying old device paths onto a new disk.
3. Rebuild the system from suitable configuration and inputs, and prepare secrets.
4. Restore files while services are not writing, then check ownership and permissions.
5. Restore databases and other state as required by the application, start services and verify functionality.

This is a preparation outline. Individual services may require different import steps. In particular, after a newer version migrates a database, older system software may not understand the changed data. Keep a recoverable backup before upgrading and plan separately for software rollback and data restoration.

If backup passwords, SSH identities or decryption keys are accessible only from the failed computer, another recovery route is needed. Avoid storing the only key needed to decrypt a backup inside that same encrypted backup.

Recommended Usage

#

I recommend listing the directories and service state worth retaining before choosing tools and frequency. Start with a restic exercise for ordinary documents. Record separate restore procedures for databases and services with several data sources.

Recheck restoration after changing backup scope, service versions or secrets. When asking AI for help, have it identify what is saved, what is omitted and what restoration still requires. Restore and compare in a new directory or isolated instance before arranging replacement of real data.

The next post will introduce NixOS services, containers and networking, including startup relationships, data directories and access settings.