backup_mac

What it is

A Mac-only backup system for a single technical user: nightly rsync mirror with optional APFS snapshots, plus a bit-rot verifier that uses cshatag-compatible xattrs and can auto-recover a rotten side from the intact mirror. One unified CLI (cli.py), one TOML config per drive, one launchd job per drive. Stdlib-only Python 3.11+, plus an optional ~100 KB Swift helper (fsevents-since) for FSEvents narrowing so nightly incrementals only rsync what actually changed. If the helper is absent the system falls back to full-tree walks; nothing else changes.

Quick start

# 1. Download the latest tarball from GitHub Releases.
curl -fsSL -o /tmp/backup-mac.tar.gz \
    https://github.com/<owner>/<repo>/releases/latest/download/backup-mac.tar.gz
tar -xf /tmp/backup-mac.tar.gz -C /tmp && cd /tmp/backup-mac-*/

# 2. Install (lays down the source tree and an example config).
./install.sh

# 3. Edit the config.
$EDITOR ~/Library/Application\ Support/backup-mac/config/drive1.toml

# 4. Activate (builds the .app bundle, writes the LaunchAgent, harvests TCC).
backup-mac activate ~/Library/Application\ Support/backup-mac/config/drive1.toml

The full walkthrough is in docs/INSTALL.md. The older hand-build path is preserved in docs/MANUAL-INSTALL.md.

Architecture

+----------+        +---------------------+        +----------------+
|  cli.py  | -----> |  backup_mac.backup  | -----> | fsevents-since |  (optional)
+----------+        +---------------------+        +----------------+
     |                       |                              |
     |                       v                              v
     |               +---------------------+      ~/Library/Application
     +-------------> |  backup_mac.verify  |     Support/backup-mac/state/fsevents/
                     +---------------------+
                            ^
                            |
              both subsystems share:
                  backup_mac.common
                  (config, logging, locking,
                   volume checks, deadlines)

Module overview:

• cli.py — argparse dispatch, lock acquisition, logging setup, signal handling, exit-code mapping. No business logic — the file is thin on purpose so the moving parts live in importable modules.
• backup_mac/common/ — config loader (config.py), logging, locking, macOS volume/mount checks (volume.py), notifications, deadline arithmetic (time_utils.py). Shared by both backup and verify; nothing here knows whether it's been called from one or the other.
• backup_mac/backup/ — rsync wrapper (rsync.py), APFS snapshot management (snapshots.py), FSEvents helper integration (fsevents.py), per-pair state (state.py), FULL/INCREMENTAL mode decision (modes.py), the run_backup orchestrator (runner.py).
• backup_mac/verify/ — xattr bindings + shatag timestamps (xattr.py), SHA256 + per-file verdict (hashing.py), tree scan (scan.py), quarantine and atomic restore (recovery.py), verify-specific config validation (config.py), time budget (budget.py), the run_verify orchestrator (runner.py).
• fsevents-since — separate Swift binary, queries the macOS FSEvents API and emits one JSON object on stdout. Built and signed separately; see docs/BUILD-FSEVENTS-HELPER.md. The source lives at helper/fsevents-since.swift; the deployed binary ends up at ./fsevents-since next to cli.py.

Operational rules

• One TOML config = one drive = one launchd job.
• All subcommands share a per-config lock (/tmp/backup-<sha>.lock, derived from the config-path string as passed). Always invoke cli.py with an absolute config path so two invocations from different working directories use the same lock. backup and verify cannot overlap on the same config.
• FSEvents per-pair state lives in ~/Library/Application Support/backup-mac/state/fsevents/<key>.json. Override the directory via the BACKUP_STATE_DIR env var (used by the test suite).
• The Python side never compiles, signs, or installs the FSEvents helper. It's a one-time manual step; without it, every run falls back to FULL_DEGRADED and a full rsync --delete.

Config at a glance

One TOML file per drive. Four sections:

• Top-level — required schema_version = 1, optional deadline (HH:MM wall-clock cutoff), notify_on_failure, eject_at_end, global_excludes, verify_after_backup, fsevents_reconcile_every_n_runs.
• [snapshots] — optional. enabled = true + keep = N to take an APFS local snapshot at the end of every successful run and prune to the N newest.
• [[pair]] — one or more required. src, dst, optional name (defaults to basename(dst)), optional per-pair excludes.
• [verify] — optional. Presence means "verify is configured for this drive". interval_days, time_budget_minutes, auto_recover, suspect_retention_days, notify_on_failure. See docs/BACKUP.md for details.

Strict-schema validation: unknown keys are rejected with the offending name in the error. v0 → v1 migration table and full per-key documentation live in docs/BACKUP.md. The canonical reference config is examples/drive1.toml.

First-time bring-up

Walk one pair through the cycle manually before letting launchd loose on the whole config. The full walkthrough is in docs/MANUAL-INSTALL.md; the abbreviated form:

CFG=~/Library/Application\ Support/backup-mac/config/drive1.toml

./cli.py status "$CFG"                                 # 1. inspect config (read-only)
./cli.py backup "$CFG" --dry-run --limit-pairs Docs    # 2. dry-run one pair
./cli.py backup "$CFG" --limit-pairs Docs              # 3. first real run = FULL_INITIAL
./cli.py backup "$CFG" --limit-pairs Docs              # 4. re-run = SKIP or INCREMENTAL
./cli.py status "$CFG"                                 # 5. inspect state
./cli.py backup "$CFG"                                 # 6. all pairs
./cli.py run    "$CFG"                                 # 7. chained backup + verify (optional)

The first run of any pair is always FULL_INITIAL (no state file exists yet, so a full rsync -a --delete runs and the watermark is written). The FSEvents-narrowed INCREMENTAL path only kicks in on run two of a pair.

Documentation

Document	Contents
docs/INSTALL.md	Easy install (download tarball, run install.sh, edit TOML, backup-mac activate).
docs/MANUAL-INSTALL.md	Hand-build walkthrough: prerequisites, helper build, first config, first run, launchd activation, deactivation, troubleshooting. Authoritative reference for what backup-mac activate does under the hood.
docs/UPGRADING.md	How to re-run install.sh for a new release, handle schema bumps, diff the refreshed example config.
docs/BACKUP.md	Full TOML config schema (including [verify] keys), every subcommand and flag, exit-code table, the nine-mode auto-selection logic, log format, and manual-intervention recipes.
docs/VERIFY.md	Bit-rot subsystem deep dive: per-file SHA256 + cshatag-compatible xattrs, cross-side recovery decision matrix, quarantine + atomic-restore semantics, time budget, what verify deliberately doesn't do.
docs/BUILD-FSEVENTS-HELPER.md	Building, signing (Apple Development cert vs. ad-hoc), and deploying the optional Swift helper that bridges FSEvents to Python.
docs/LAUNCHD-PYTHON-PERMISSIONS.md	How to grant TCC permissions (Full Disk Access etc.) to a Python script via a wrapper .app bundle, without granting them to python3 globally.

Subcommand reference

cli.py [-v] backup <config> [--dry-run] [--limit-pairs N[,N,...]] [--force-full]
cli.py [-v] verify <config> [--dry-run] [--limit-pairs N[,N,...]] [--no-budget]
cli.py [-v] run    <config> [--dry-run] [--limit-pairs N[,N,...]]
cli.py      status <config>

backup performs an rsync mirror of every [[pair]] and an optional APFS snapshot; the mode for each pair (SKIP, INCREMENTAL, or one of the six FULL_* modes) is auto-derived from the helper output and the saved state. No user input picks the mode — the priority-ordered nine-way decision lives in backup_mac.backup.modes and is documented in docs/BACKUP.md. verify walks every regular file on both sides of every pair, hashes by cshatag-compatible xattrs, and (with auto_recover = true) restores one side from the other when one is rotten and the other intact; the algorithm and recovery decision matrix are in docs/VERIFY.md. run chains backup then optionally verify inside a single lock — this is what launchd should invoke. status is a read-only diagnostic that doesn't take the lock and is safe to run while a backup is in progress. The global -v / --verbose flag enables DEBUG-level logging on stderr; the log file always stays at INFO. Per-subcommand flag tables and behavior live in docs/BACKUP.md.

Exit codes

Code	Meaning
0	Success.
1	Config or CLI-argument error. Exits before the lock is taken.
2	Partial failure: at least one pair failed (backup) or at least one verify problem was reported.
3	Run aborted — deadline hit during backup, verify time budget hit, or signal received.
4	Lock already held by another process for the same config.
5	Unexpected internal error (uncaught exception).

Logging

One file handler per cli invocation (~/Library/Application Support/backup-mac/logs/<drive>-<timestamp>.log, where <drive> is the config-file stem) and one stderr handler. Format: 2026-05-13T22:14:39+0200 INFO <message>. The file handler always stays at INFO; the stderr handler escalates to DEBUG with --verbose. A best-effort 14-day retention sweep runs at startup; the precise per-run log path is appended to every failure notification.

Once-per-run banner: mode: fsevents (helper at ...) when the helper was found, mode: legacy (helper at ...) (preceded by a WARNING) when it wasn't. Per pair: a pair <name> src=... dst=... block, then the mode decision (mode=SKIP, mode=INCREMENTAL with subtree count, or mode=FULL_* with the trigger), then the rsync invocation, then a closing rsync ok: <name> (...). APFS snapshots get their own APFS snapshot created: ... + prune complete kept=N deleted=M lines. Verify emits an ERROR line per problem file (BIT ROT detected, CROSS-COMPARE DIFFER) plus an INFO scan-complete summary per side:

seen=N hashed=N cross_verified=N rotten=N cross_differ=N read_failures=N aborted=...

The closing summary is run done elapsed=Ns ok or the same with backup_failed=N verify_problems=M.

Full log-format reference and --verbose semantics: see docs/BACKUP.md (backup side) and docs/VERIFY.md (verify side).

FSEvents helper

fsevents-since is a small (~100 KB) Swift CLI that lives at ./fsevents-since next to cli.py. Given a volume path and an event-id watermark, it queries the macOS FSEvents API and emits one JSON object on stdout listing the changed subtrees since that watermark. The Python side uses this to narrow incremental rsync runs to just the subtrees that actually changed — typically seconds to minutes per pair instead of full-tree walks.

If the binary is missing or not executable, the system runs in legacy mode: every pair becomes FULL_DEGRADED (full rsync -a --delete), a WARNING is logged, a macOS notification fires (when notify_on_failure = true), and state files are deliberately not written — so a later run with the helper restored can pick up the FSEvents stream from where it left off.

The helper is optional, single-purpose, and built/signed separately. Full build instructions, code-signing trade-offs (Apple Development cert vs. ad-hoc), and deployment are in docs/BUILD-FSEVENTS-HELPER.md. Source: helper/fsevents-since.swift.

Threat model

Snapshots protect against: accidental rm, ransomware-style destination corruption, a bad rsync --delete propagating from a partly-deleted source. Rollback recipe (read-only mount + copy out) in docs/BACKUP.md. Copy-on-write, so cost scales with churn not count.

Snapshots do NOT protect against bit rot. Every snapshot points at the same underlying APFS blocks; a flipped bit reads as flipped through every snapshot that referenced it. That's what verify is for.

Verify protects against: silent data corruption on either side of a pair. Cross-side recovery picks the intact side automatically, provided both sides aren't simultaneously corrupted (independent failures on two disks within one interval_days window).

Not covered:

• Off-machine backup. Pair this tool with an offsite copy (Backblaze, rclone, etc.) — it handles "local mirror + integrity", not "the house burnt down".
• Compromised running OS. Root-level malware rewriting xattrs and file contents in lockstep is invisible to verify.
• Deliberate destruction by an attacker with shell access. tmutil deletelocalsnapshots and rm -rf are not protected.

Tests

python3 -m pytest tests/ -q

Expect 586 passing on macOS. xattr tests are macOS-only and are skipped on other platforms. One integration test exercises the real Swift helper binary; it is gated behind fsevents-since existing alongside cli.py and being executable. Build and install the helper per docs/BUILD-FSEVENTS-HELPER.md to enable it.

License

backup_mac's own code is MIT licensed — see LICENSE.

The release tarball bundles a static rsync binary at vendor/rsync, which is licensed under GPL-3.0-or-later (full text in LICENSE.rsync). backup_mac runs rsync as a separate subprocess, so bundling it is mere aggregation and does not affect the MIT license of this project. Per GPL-3.0, the corresponding rsync source is attached to every GitHub Release alongside the binary; details and a written offer are in NOTICE.