Split addy into rootless Quadlets with health checks and Traefik routing

.gitignore

@@ -1,3 +1,18 @@
-# Ignore instance specific configuration files
+# Node-specific configuration and secrets
 config.map
-secrets.yml
+secrets.yml
+*.env
+!*.env.example
+*.conf
+!redis.conf
+!*.conf.example
+# macOS / editors / local agent settings
+.DS_Store
+.AppleDouble
+._*
+.idea/
+.vscode/
+*.swp
+*.swo
+.claude/settings.local.json
+.codex/settings.local.json

README.md

@@ -1,3 +1,84 @@
-# Podman Quadlet Collection
+# Addy.io rootless Quadlets
 
-This repository contains a collection of services, modelled as podman quadlets.
+Addy.io forwards mail through aliases. This template runs addy.io, MariaDB and Redis as independent rootless systemd services. Traefik routing is an optional node-local drop-in.
+
+| Service | Image | Persistent storage |
+| --- | --- | --- |
+| addy | docker.io/anonaddy/anonaddy:latest | addy.io-data |
+| addy-db | docker.io/library/mariadb:latest | addy.io-db |
+| addy-redis | docker.io/library/redis:latest | None; disposable cache |
+
+All services intentionally follow `latest` and use `AutoUpdate=registry`. Upstream releases, including incompatible database releases, are accepted automatically by policy. MariaDB uses `MARIADB_AUTO_UPGRADE=1` for upgrades supported by the image entrypoint. Image rollback does not undo database migrations.
+
+## Layout
+
+Clone/copy this template to `~/.config/containers/systemd/quadlet-addy.io/`. Keep actual configuration outside the repository in `~/.config/addy/`. Supplied domains and env files are examples. Never commit live configuration or secrets.
+
+| File | Purpose |
+| --- | --- |
+| `addy*.container` | Independent services, startup probes and health checks |
+| `addy.network` | Application bridge with IPv6 enabled |
+| `addy*.volume` | Existing named data volumes, preserving their names |
+| `healthcheck.sh` | HTTP login endpoint and SMTP banner checks |
+| `redis.conf` | Disables persistence for the disposable Redis cache |
+| `*.env.example` | Non-secret runtime configuration templates |
+| `rootless-network.conf.example` | Experimental pesto-backed rootless bridge forwarding |
+| `addy.container.d/traefik-labels.conf.example` | Optional proxy network and routing labels |
+
+The base app joins only `addy`. MariaDB and Redis join only `addy`, with no published ports. The app publishes SMTP on host port 25. HTTP port 8000 and Rspamd port 11334 are not published. A node-local Traefik drop-in attaches the app to `webproxy` and supplies its hostname, TLS resolver and explicit upstream port. Traefik joins `webproxy` and has no direct network connection to the database/cache.
+
+## Host prerequisites
+
+Use a rootless service account with subordinate UID/GID mappings and lingering enabled. Commands below run as that account unless marked otherwise. Preserve its identity and storage when migrating.
+
+Use a Podman release supporting `rootless_port_forwarder="pasta"` and a compatible passt package supplying `pesto`. Stop all rootless workloads for the account before activating the new forwarding configuration, so its shared rootless network namespace is recreated with the pesto control socket. Copy the example to `~/.config/containers/containers.conf.d/90-pesto.conf`. The forwarding feature is experimental. Verify source IPs and a container restart on the installed versions; older passt/Podman combinations have had forwarding cleanup bugs.
+
+For direct low-port binding, the host administrator must persist and apply `net.ipv4.ip_unprivileged_port_start=25`, for example using `/etc/sysctl.d/90-rootless-container-ports.conf`. This permits all unprivileged host users to bind unused ports 25 and higher. Application configuration and units remain under the service account's home. Allow SMTP/HTTP/HTTPS on intended firewall interfaces. Remove only matching old high-port redirections after changing listeners; check IPv4 and IPv6, runtime and permanent rules. This template does not modify the firewall.
+
+## Initial deployment
+
+1. Copy `addy.env.example` and `db.env.example` to `~/.config/addy/addy.env` and `~/.config/addy/db.env`. Set real domains and database/user names. Use directory mode 0700 and file mode 0600.
+2. Create `addy-db-password`, `addy-app-key`, and `addy-reply-secret` in the service account's Podman store. Use `podman secret create NAME -` with input from a password manager or protected prompt. Preserve existing values during migration; do not enter real secrets as literal command arguments.
+3. Configure pesto as above. Deploy Traefik and its `webproxy.network` Quadlet under the same account. Traefik uses its published image with the built-in deSEC DNS provider, the existing `desec-token` Podman secret, private ACME storage, and `exposedByDefault: false`. Its Docker provider discovers this user's containers through the local Podman API socket; no compilation is needed.
+4. Copy `addy.container.d/traefik-labels.conf.example` to `addy.container.d/traefik-labels.conf` and set the real hostname. The drop-in must attach `Network=webproxy.network` AND set `traefik.docker.network=webproxy`. Labels alone cannot connect the proxy to the app. Keep live drop-ins untracked.
+5. Run `systemctl --user daemon-reload`, then `systemctl --user start addy-db.service addy-redis.service`. Wait for healthy readiness, then `systemctl --user start addy.service traefik.service`.
+6. Start generated Quadlet services; do not enable them. Their `[Install]` sections provide boot activation with lingering. Enable the vendor timer once: `systemctl --user enable --now podman-auto-update.timer`. Its default schedule is daily; check `systemctl --user list-timers podman-auto-update.timer`.
+
+## Readiness and recovery
+
+MariaDB uses `healthcheck.sh --connect --innodb_initialized`; Redis must answer PONG. The app waits for their startup jobs via Wants/After, while `Notify=healthy` gates startup on health. Wants deliberately avoids stopping the app whenever a dependency restarts.
+
+Application health requires HTTP `/login` and a local SMTP 220 banner. These do not prove end-to-end delivery or every dependency path. Startup probes run every 5 seconds, then steady-state checks every 60 seconds. Three failed steady-state checks kill the affected container and systemd recreates it. Startup permits 15 minutes for pulls/migrations. Services retry at 30-second intervals without a permanent start-limit lockout, preserving unattended recovery and future updates. Failures appear in the journal; no external alert integration is configured.
+
+## Migration from the old single-pod and Caddy layout
+
+Never run old and new database containers against the same volume concurrently.
+
+1. Record images, paths and unit configuration. Preserve the existing app encryption/reply keys and database credentials. The old `addy.io-secrets` is a kube-style multi-key secret; migrate each effective value into the three separately named secrets.
+2. Populate runtime env files from existing configuration. Change `DB_HOST=addy-db` and `REDIS_HOST=addy-redis`, retaining database names/users and other settings.
+3. Stop the old `addy.service` and Caddy; wait for shutdown. Move the old `.kube`, YAML/config files and Caddy Quadlets outside all Quadlet search paths for rollback. Preserve all named data volumes and Caddy certificate data.
+4. Activate pesto once the shared rootless network namespace is gone. Install new units and local Traefik routing drop-in. Reload systemd, start DB/cache, then app and Traefik. Traefik replaces Caddy's public listeners and obtains its own certificate using deSEC DNS-01.
+5. Verify healthy services, HTTPS, SMTP over IPv4/IPv6, preserved client addresses, auto-update labels/timer, and a restart. An app-only restart must leave the DB running.
+6. Probe external-to-external SMTP relay with MAIL/RCPT only, then RSET/QUIT. Expect rejection; never send DATA.
+
+This change intentionally installs no backup services/timers. Ensure an appropriate recovery source exists before a stateful migration. For configuration rollback, stop new services before restoring old units/config and reloading systemd. A database upgraded by `latest` may need restoration from an earlier data backup to run an older image.
+
+## Local overrides
+
+Use `<name>.container.d/*.conf` for node customization. `.conf.example` files are inert. Reset list keys with an empty assignment before replacing them (for example `PublishPort=`). Keep live env files, domains and secrets outside the template repo or ignored.
+
+## Operations
+
+```sh
+systemctl --user status addy addy-db addy-redis traefik
+podman ps
+podman healthcheck run addy
+podman healthcheck run addy-db
+podman healthcheck run addy-redis
+journalctl --user -u addy.service -u addy-db.service -u addy-redis.service
+systemctl --user start podman-auto-update.service
+podman auto-update --dry-run
+podman secret ls
+```
+
+To rotate a secret, supply its value through stdin to `podman secret create --replace NAME -`, then recreate affected containers through systemd. Updating a database secret alone does not change an existing MariaDB account password; coordinate the account update with both services. Do not regenerate encryption/reply keys as routine maintenance.

addy-data.volume

@@ -0,0 +1,3 @@
+[Volume]
+VolumeName=addy.io-data
+Label=app=addy

addy-db.container

@@ -0,0 +1,36 @@
+[Unit]
+Description=Addy MariaDB
+StartLimitIntervalSec=0
+
+[Container]
+Image=docker.io/library/mariadb:latest
+AutoUpdate=registry
+ContainerName=addy-db
+Network=addy.network
+NetworkAlias=addy-db
+Volume=addy-db.volume:/var/lib/mysql:Z
+EnvironmentFile=%h/.config/addy/db.env
+Secret=addy-db-password,type=env,target=MARIADB_PASSWORD
+Environment=MARIADB_RANDOM_ROOT_PASSWORD=yes
+Environment=MARIADB_AUTO_UPGRADE=1
+Exec=mariadbd --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
+HealthCmd=healthcheck.sh --connect --innodb_initialized
+HealthInterval=60s
+HealthRetries=3
+HealthStartPeriod=120s
+HealthTimeout=10s
+HealthStartupCmd=healthcheck.sh --connect --innodb_initialized
+HealthStartupInterval=5s
+HealthStartupRetries=60
+HealthStartupTimeout=10s
+Notify=healthy
+HealthOnFailure=kill
+
+[Service]
+Restart=always
+RestartSec=30s
+TimeoutStartSec=900
+TimeoutStopSec=120
+
+[Install]
+WantedBy=default.target

addy-db.volume

@@ -0,0 +1,3 @@
+[Volume]
+VolumeName=addy.io-db
+Label=app=addy

addy-redis.container

@@ -0,0 +1,33 @@
+[Unit]
+Description=Addy Redis cache
+StartLimitIntervalSec=0
+
+[Container]
+Image=docker.io/library/redis:latest
+AutoUpdate=registry
+ContainerName=addy-redis
+Network=addy.network
+NetworkAlias=addy-redis
+Volume=./redis.conf:/usr/local/etc/redis/redis.conf:ro,Z
+ImageVolume=ignore
+Exec=redis-server /usr/local/etc/redis/redis.conf
+HealthCmd=/bin/sh -ec 'redis-cli ping | grep -qx PONG'
+HealthInterval=60s
+HealthRetries=3
+HealthStartPeriod=15s
+HealthTimeout=5s
+HealthStartupCmd=/bin/sh -ec 'redis-cli ping | grep -qx PONG'
+HealthStartupInterval=5s
+HealthStartupRetries=12
+HealthStartupTimeout=5s
+Notify=healthy
+HealthOnFailure=kill
+
+[Service]
+Restart=always
+RestartSec=30s
+TimeoutStartSec=900
+TimeoutStopSec=60
+
+[Install]
+WantedBy=default.target

addy.container

@@ -0,0 +1,39 @@
+[Unit]
+Description=Addy email forwarding
+Wants=addy-db.service addy-redis.service
+After=addy-db.service addy-redis.service
+StartLimitIntervalSec=0
+
+[Container]
+Image=docker.io/anonaddy/anonaddy:latest
+AutoUpdate=registry
+ContainerName=addy
+Network=addy.network
+NetworkAlias=addy
+Volume=addy-data.volume:/data:Z
+Volume=./healthcheck.sh:/usr/local/bin/addy-healthcheck:ro,Z
+EnvironmentFile=%h/.config/addy/addy.env
+Secret=addy-db-password,type=env,target=DB_PASSWORD
+Secret=addy-app-key,type=env,target=APP_KEY
+Secret=addy-reply-secret,type=env,target=ANONADDY_SECRET
+PublishPort=25:25
+HealthCmd=/bin/sh /usr/local/bin/addy-healthcheck
+HealthInterval=60s
+HealthRetries=3
+HealthStartPeriod=180s
+HealthTimeout=15s
+HealthStartupCmd=/bin/sh /usr/local/bin/addy-healthcheck
+HealthStartupInterval=5s
+HealthStartupRetries=90
+HealthStartupTimeout=15s
+Notify=healthy
+HealthOnFailure=kill
+
+[Service]
+Restart=always
+RestartSec=30s
+TimeoutStartSec=900
+TimeoutStopSec=120
+
+[Install]
+WantedBy=default.target

addy.container.d/traefik-labels.conf.example

@@ -0,0 +1,11 @@
+# Copy to traefik-labels.conf on the node, then set the real hostname.
+[Container]
+Network=webproxy.network
+Label=traefik.enable=true
+Label=traefik.docker.network=webproxy
+Label=traefik.http.routers.addy.rule=Host(`addy.example.com`)
+Label=traefik.http.routers.addy.entrypoints=websecure
+Label=traefik.http.routers.addy.tls=true
+Label=traefik.http.routers.addy.tls.certresolver=letsencrypt
+Label=traefik.http.routers.addy.service=addy
+Label=traefik.http.services.addy.loadbalancer.server.port=8000

addy.env.example

@@ -0,0 +1,22 @@
+TZ=Europe/Copenhagen
+PUID=1000
+PGID=1000
+APP_URL=https://addy.example.com
+APP_DEBUG=false
+DB_HOST=addy-db
+DB_PORT=3306
+DB_DATABASE=anonaddy
+DB_USERNAME=anonaddy_user
+REDIS_HOST=addy-redis
+REDIS_PORT=6379
+ANONADDY_DOMAIN=example.com
+ANONADDY_ALL_DOMAINS=example.com
+ANONADDY_HOSTNAME=mail.example.com
+ANONADDY_RETURN_PATH=bounces@example.com
+ANONADDY_ADMIN_USERNAME=admin
+ANONADDY_ENABLE_REGISTRATION=false
+MAIL_FROM_NAME=addy.io
+MAIL_FROM_ADDRESS=addy@example.com
+# Set this to a resolver reachable from inside the container.
+ANONADDY_DNS_RESOLVER=1.1.1.1
+# Password and encryption/reply keys are supplied by Podman secrets.

addy.kube

@@ -1,19 +0,0 @@
-[Unit]
-Description=Addy.io Mail Service (Pod)
-After=network-online.target
-
-[Kube]
-Yaml=addy.yml
-AutoUpdate=registry
-
-ConfigMap=config.map
-
-PublishPort=2525:25
-PublishPort=8000:8000
-
-[Service]
-Restart=always
-RestartSec=3
-
-[Install]
-WantedBy=default.target

addy.network

@@ -0,0 +1,7 @@
+[Unit]
+Description=Addy application network
+
+[Network]
+NetworkName=addy
+IPv6=true
+Label=app=addy

addy.yml

@@ -1,97 +0,0 @@
-apiVersion: v1
-kind: Pod
-metadata:
-  name: addy.io
-spec:
-  restartPolicy: Always
-  containers:
-    - name: addy-db
-      image: docker.io/library/mariadb:12
-      args:
-        - "mariadbd"
-        - "--character-set-server=utf8mb4"
-        - "--collation-server=utf8mb4_unicode_ci"
-      env:
-        - name: MARIADB_RANDOM_ROOT_PASSWORD
-          value: "yes"
-        - name: MYSQL_DATABASE
-          valueFrom:
-            configMapKeyRef:
-                name: addy.io-config-shared
-                key: DATABASE_NAME
-        - name: MYSQL_USER
-          valueFrom:
-            configMapKeyRef:
-                name: addy.io-config-shared
-                key: DATABASE_USER
-        - name: MYSQL_PASSWORD
-          valueFrom:
-              secretKeyRef:
-                name: addy.io-secrets
-                key: db-password
-      volumeMounts:
-        - mountPath: /var/lib/mysql
-          name: addy-db-volume
-
-    # --- Redis Service ---
-    - name: addy-redis
-      image: docker.io/library/redis:8-alpine
-
-    # --- Addy Application ---
-    - name: addy-app
-      image: docker.io/anonaddy/anonaddy:latest
-      ports:
-        - containerPort: 25
-          hostPort: 2525
-          protocol: TCP
-        - containerPort: 8000
-          hostPort: 8000
-          protocol: TCP
-      envFrom:
-        - configMapRef:
-            name: addy.io-config
-      env:
-        - name: DB_HOST
-          value: "127.0.0.1"
-        - name: REDIS_HOST
-          value: "127.0.0.1"
-
-        - name: APP_KEY
-          valueFrom:
-              secretKeyRef:
-                name: addy.io-secrets
-                key: app-key
-        - name: ANONADDY_SECRET
-          valueFrom:
-              secretKeyRef:
-                name: addy.io-secrets
-                key: addy-secret
-        
-        # Database Configuration
-        - name: DB_DATABASE
-          valueFrom:
-            configMapKeyRef:
-              name: addy.io-config-shared
-              key: DATABASE_NAME
-        - name: DB_USERNAME
-          valueFrom:
-            configMapKeyRef:
-              name: addy.io-config-shared
-              key: DATABASE_USER
-        - name: DB_PASSWORD
-          valueFrom:
-              secretKeyRef:
-                name: addy.io-secrets
-                key: db-password
-      volumeMounts:
-        - mountPath: /data
-          name: addy-data-volume
-
-  # --- Volume Definitions ---
-  volumes:
-    - name: addy-db-volume
-      persistentVolumeClaim:
-        claimName: addy.io-db
-    - name: addy-data-volume
-      persistentVolumeClaim:
-        claimName: addy.io-data

config.map.tmpl

@@ -1,44 +0,0 @@
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: addy.io-config-shared
-data:
-  TZ: "Europe/Berlin"
-  DATABASE_NAME: "anonaddy"
-  DATABASE_USER: "anonaddy_user"
-
---- 
-
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: addy.io-config
-data:
-  PUID: "1000"
-  PGID: "1000"
-  MEMORY_LIMIT: "256M"
-  UPLOAD_MAX_SIZE: "16M"
-  OPCACHE_MEM_SIZE: "128"
-  REAL_IP_FROM: "0.0.0.0/32"
-  REAL_IP_HEADER: "X-Forwarded-For"
-  LOG_IP_VAR: "remote_addr"
-  APP_DEBUG: "false"
-  APP_URL: "http://127.0.0.1:8000"
-  ANONADDY_RETURN_PATH: "bounces@example.com"
-  ANONADDY_ADMIN_USERNAME: "addy"
-  ANONADDY_ENABLE_REGISTRATION: "true"
-  ANONADDY_DOMAIN: "example.com"
-  ANONADDY_ALL_DOMAINS: "example.com"
-  ANONADDY_HOSTNAME: "mail.example.com"
-  ANONADDY_DNS_RESOLVER: "127.0.0.1"
-  ANONADDY_LIMIT: "200"
-  ANONADDY_BANDWIDTH_LIMIT: "104857600"
-  ANONADDY_NEW_ALIAS_LIMIT: "10"
-  ANONADDY_ADDITIONAL_USERNAME_LIMIT: "3"
-  ANONADDY_DKIM_SIGNING_KEY: ""
-  ANONADDY_DKIM_SELECTOR: ""
-  MAIL_FROM_NAME: "addy.io"
-  MAIL_FROM_ADDRESS: "addy@example.com"
-  POSTFIX_DEBUG: "false"
-  POSTFIX_SMTPD_TLS: "false"
-  POSTFIX_SMTP_TLS: "false"

db.env.example

@@ -0,0 +1,2 @@
+MARIADB_DATABASE=anonaddy
+MARIADB_USER=anonaddy_user

healthcheck.sh

@@ -0,0 +1,4 @@
+#!/bin/sh
+set -eu
+curl --fail --silent --show-error --max-time 8 --output /dev/null http://127.0.0.1:8000/login
+php -r '$s = @fsockopen("127.0.0.1", 25, $errno, $errstr, 3); if (!$s) exit(1); stream_set_timeout($s, 3); $banner = fgets($s); $ok = is_string($banner) && strncmp($banner, "220 ", 4) === 0; fwrite($s, "QUIT\r\n"); fclose($s); exit($ok ? 0 : 1);'

redis.conf

@@ -0,0 +1,4 @@
+bind 0.0.0.0 ::
+protected-mode no
+save ""
+appendonly no

rootless-network.conf.example

@@ -0,0 +1,4 @@
+# Copy to ~/.config/containers/containers.conf.d/90-pesto.conf.
+# Requires Podman 6 with a compatible passt package supplying pesto.
+[network]
+rootless_port_forwarder="pasta"

secrets.yml.tmpl

@@ -1,9 +0,0 @@
-apiVersion: v1
-kind: Secret
-metadata:
-  name: addy.io-secrets
-type: Opaque
-data:
-  db-password: c3VwZXItc2VjcmV0LWRiLXBhc3M=
-  addy-secret: replace-me-with-a-random-string
-  app-key: base64:CN/JeFqgUyUMUfwgqZDMjOldkBt0ye/HM9esIH7diPU=