infrastructure

Everything between a cloud account and a running service: cloud resources in Terraform, host configuration in Ansible, deployments, checks, load tests and backups, each a make command run from the operator’s machine over SSH. A host exists for one working session. make down removes it, and a deadline switch terminates it if it is forgotten.

A host on demand

make up creates the network, the firewall rules, the instance, its DNS record and its Cloudflare origin certificate, then configures the host and deploys its services. A rebuild from nothing needs no step by hand. A catalog says which service runs where, under which name and port, and is checked as a whole before anything touches the host.

Closed by default

• SSH from the operator’s address only, HTTPS from Cloudflare’s address ranges only: in the AWS security group, and again in nftables on the host.
• The nftables rules judge every packet before Docker’s own rules, so a port a container publishes by mistake stays closed. Tested with a container published on 0.0.0.0.
• Outbound, only HTTPS and NTP. IPv6 off.
• TLS with a Cloudflare origin certificate and Authenticated Origin Pulls; a handshake for an unknown name is refused.
• While nothing is public, nginx answers the operator’s address only.
• No automatic updates: make updates shows what is pending and make upgrade applies it.

Releases without downtime

Every service has two colours, blue and green. A release starts the new version next to the running one, waits for its health check, points nginx at it after nginx -t, and stops the old one once its connections have drained. Clients reconnected in under a second. nginx reaches identity-service over a Unix socket, with no published port.

Checked, not assumed

make verify checks the running machines; on the last run with two machines, 69 checks: closed ports, the certificate, HSTS, request logs without query strings, automatic updates off, the services’ health. In a reboot test both machines came back, every container was healthy within 29 seconds, and all 69 checks passed with nothing started by hand.

A checklist in the repository says, item by item, what a host needs before it is exposed and while it runs for days, with its evidence or its plan. Still open before a long-running host: origin pulls with a client certificate of our own, alerting from outside the host, and a memory limit on every container.

Load tests

make bench adds a load generator next to the host and runs k6 scenarios step by step, from a plan written before the first run. Every run is recorded with a description of both machines, capacity is computed from the records, and Prometheus and Grafana watch both machines during the session. The results for identity-service are on its page.

Backups and managed services

make backup streams a pg_dump over SSH and encrypts it with age on the operator’s machine, so the host never holds a copy. make restore loads it into a new database and swaps it in only when every table’s row count matches the backup’s manifest. Both ran on AWS, also from the host’s own PostgreSQL to RDS and back.

A separate stack adds a KMS key, an S3 bucket that refuses plain HTTP and any other key, and RDS PostgreSQL reached over TLS with its certificate and name verified.

Secrets

SOPS and age, outside every repository. A pre-commit hook and CI refuse secret files by name, and gitleaks scans the whole history.