NixOS (12): Updates and Deployment Across Computers

Motivation

#

The earlier posts covered configuration, secrets, data and services. With several computers, maintenance also means knowing how far each update has progressed: whether source revisions match, whether the system was built and activated, and whether an old kernel is still running.

In practice, I organized routine system updates as a manual process: review inputs in one place, then build, apply and verify one computer at a time. Container and application source synchronization is recorded separately. A Git update alone is not reported as a completed system upgrade.

This post explains the reusable parts of that process. It assumes a working flake and Git repository and uses the fictional output laptop. Replace revisions, paths and names with your own; no existing host configuration is reproduced.

Types of Updates

#

Identify what is changing first:

Containers on these computers still use Compose. Updating one is distinct from updating NixOS. Source may be synchronized everywhere while only the deployment host recreates its container; other computers merely retain the same source files.

One lock file also does not make every system output identical. Computers share inputs but select their own modules, hardware configuration and software. Build the appropriate output for each.

Preparing an Update

#

Checking the Baseline

#

From the configuration repository, inspect changes and system state:

1
2
3
4
5
6
7
8
9

git status --short
git rev-parse HEAD
readlink -f /nix/var/nix/profiles/system
readlink -f /run/current-system
readlink -f /run/booted-system
uname -r
systemctl --failed
systemctl --user --failed
df -h / /boot

Preserve and reconcile existing edits rather than discarding them to simplify synchronization. Record pre-existing service failures so they are not mistaken for regressions. Check user services from the intended user’s session.

The Git revision, system profile, active configuration and booted configuration are separate records. Maintenance has included cases where source was updated and the system switched while an older generation remained booted. Git or generation numbers alone cannot identify the running kernel.

Scheduling Time and Build Resources

#

Routine updates use a manual maintenance window, with roughly fortnightly or monthly intervals chosen around use. This is a maintenance convention, not an enabled automatic upgrade task. Urgent fixes need separate scheduling.

Some changes regenerate configuration; others build kernels or large dependencies. An earlier kernel build took hours and failed for lack of space. Check free space, temporary storage and retained recovery generations before beginning. That observation does not impose the same resource budget on every computer.

Remote building is an option when resources are insufficient, but needs separate builder identity, platform and trust setup. It does not activate the target computer. Building, transfer and deployment are different operations. See the Nix remote build documentation.

Updating Flake Inputs

#

Updating in One Place

#

Choose a clean main checkout as the coordinator:

1
2
3
4
5

git pull --ff-only
nix flake update
git diff -- flake.lock
git diff --check
nix flake check --no-build --show-trace

nix flake update changes the lock file without applying a system. Branch selection remains in flake.nix; changing releases needs a separate upgrade review. A --no-build check cannot prove all packages build or hardware works.

For a deliberate single-input update, use:

1

nix flake update nixpkgs

This is an alternative to the full update above, not another step to execute mechanically. Dependencies can still change other nodes, so review the entire diff. See the Nix input update documentation.

Recording the Deployment Revision

#

After review, publish the prepared inputs:

1
2
3
4

git add flake.lock
git commit -m "Update reviewed flake inputs"
git push origin main
git rev-parse HEAD

If modules changed too, explicitly stage the reviewed files. The full revision printed last identifies deployment source. It does not replace evidence of target activation.

Targets synchronize this reviewed revision without running their own nix flake update. Independent updates can obtain inputs from different times and complicate comparisons.

Deploying One Computer at a Time

#

Synchronizing a Target

#

From the target’s configuration repository:

1
2
3
4

rollout_revision="REPLACE_WITH_REVIEWED_COMMIT"
test -z "$(git status --porcelain)" && \
  git pull --ff-only && \
  test "$(git rev-parse HEAD)" = "$rollout_revision"

Replace the placeholder first. This requires a clean checkout, a fast-forward update and the recorded revision. Stop and reconcile any failure. If the remote has newer commits, do not silently deploy them as though they were reviewed for this update. Prepare a checkout at the intended revision before continuing.

Deployment order considers hands-on recovery, desktop use, compute jobs and service downtime. I start with a computer that is easy to observe and recover, verify it, then continue. Avoid rebooting all computers together; a particular hardware model need not always go first.

Building and Applying

#

Build the target’s output on that computer:

1

nixos-rebuild build --flake .#laptop --no-update-lock-file

--no-update-lock-file prevents an incidental input update during the build. For a long SSH operation, an existing tmux session helps reconnect after losing the connection. It cannot solve a reboot or insufficient disk space.

After review, when ready to apply:

1

sudo nixos-rebuild switch --flake .#laptop --no-update-lock-file

switch activates the configuration and sets the later boot default. On build or activation failure, stop further deployment, retain the error and inspect current state. Activation may already have changed some services; failure does not mean nothing changed. See post five for the stages.

Rebooting and Verification

#

Deciding Whether to Reboot

#

After switching, compare boot components separately:

1
2
3
4

for component in kernel kernel-modules initrd; do
  readlink -f "/run/current-system/$component"
  readlink -f "/run/booted-system/$component"
done

When kernel or module paths change, schedule a reboot after saving work and stopping affected jobs. A changed initrd also needs a new boot to validate it. uname -r alone is insufficient: identical version strings can describe different builds.

The records also include different active and booted generations with identical kernel, module and initrd paths. Different generations alone therefore do not establish a required reboot. Other driver, service or session changes can still require a restart or new login; inspect the actual change.

Checking Target Functionality

#

1
2
3
4
5
6

readlink -f /nix/var/nix/profiles/system
readlink -f /run/current-system
readlink -f /run/booted-system
uname -r
systemctl --failed
systemctl --user --failed

If a reboot was scheduled, reconnect and inspect the system actually booted. Then test functions relevant to that computer: desktop login, SSH, hardware, compute tasks or service access. No failed units does not prove these functions were tested.

For services, check the local backend separately from the intended entry point. Desktop changes may take effect only in a new session. Record completed and pending checks before deciding to continue to the next computer.

Application Repositories and Submodules

#

Application source, Compose definitions and system configuration can live in different repositories. Some applications are submodules of a parent repository, so publication order matters:

1. Commit and push application changes in the child repository.
2. Record that published child revision in the parent and push the parent.
3. Synchronize the parent and its recorded submodule revisions on targets.
4. Prepare runtime configuration, deploy the service and verify access and function.

The parent records a child revision; it does not upload unpublished child commits automatically. Source synchronization does not rebuild a running application image either.

After preserving checkout and submodule edits and confirming updates are appropriate, use:

1
2
3

git submodule update --init --recursive
git submodule status --recursive
git status --short

Default submodule updates use the parent’s recorded revision and may leave a detached HEAD. They do not follow the child’s newest branch. Arrange branch synchronization separately when maintaining your own child branch; avoid forced updates that overwrite work. See the Git submodule documentation.

When remote access or credential tooling is unreliable, I also use Git bundles to transfer already published commits. A bundle transports commits; it does not establish clean target worktrees, correct submodules or deployed services. Verify revisions and results afterward.

Recovery and Records

#

If a new system boots but is unsuitable, the recovery plan can include returning to the previous profile generation:

1

sudo nixos-rebuild switch --rollback

The previous generation is not guaranteed usable. Retain a known-working recovery target before deployment. Restoring its kernel can require rebooting; a system that cannot boot needs a usable generation in the boot menu. Rollback neither undoes Git commits nor restores application data.

I record source revision, activation result, boot state, functional checks and outstanding work separately. Documentation-only changes are not assigned a new system generation. An unperformed reboot or test is not marked verified.

After a period of normal use, decide which old generations and unused build references to remove. Do not start troubleshooting a failed update by deleting the system retained for recovery.

Recommended Usage

#

I recommend reviewing inputs centrally and deploying computers one at a time, each from confirmed source. Separate ordinary settings, system inputs and application deployment to keep changes manageable and locate failures.

When asking AI for help, have it identify affected computers, expected time and whether it will activate or reboot. Have it report source synchronization, build, application, boot and functional verification separately. That leaves a useful record of what is complete and what has only been prepared.

The next post will reflect on maintaining NixOS with AI: which work suits an assistant and how actual results reveal its mistakes.