Vorras

The shirt shows four words in white capitals: I HAD POTENTIAL. Beneath them, a white curved ramp descends from an upper-left ledge to a lower-right flat. On the upper ledge sits a dashed, empty circle. On the lower flat rests a solid white ball. The ramp is curved, not a straight diagonal. That is the whole design. No arrows, no equations, no caption explaining the joke.

I keep looking at the dashed circle. In a physics diagram, a dashed outline usually marks a position that is not occupied — a ghost, a predicted location, a thing that could have been there. The solid ball is the thing that is there. The shirt’s sentence says the same thing in words: potential existed, past tense, and what remains is the object at the bottom of the ramp. The drawing and the sentence are not redundant. They are two versions of the same claim, and the drawing is the more precise one.

This is why the shirt works as an object rather than a slogan. A slogan asserts. A diagram demonstrates. The curved ramp implies a path, a direction, a loss of height. The dashed circle implies absence at the top. The solid ball implies presence at the bottom. You do not need to know the physics of rolling motion to read it; you need only notice that the empty shape is where the ball used to be and the filled shape is where it ended up. The joke is not that the ball fell. The joke is that the fall was drawn as a system.

Technical readers tend to like this kind of humor because it flatters a specific habit: reading the diagram before reading the caption. The shirt rewards that habit. If you glance at it, you see a minimalist pun. If you look at it, you see a small argument about stored versus spent energy, about the difference between a state and a trajectory. The dashed circle is the state that was possible. The solid ball is the state that is actual. The ramp is the mechanism that converted one into the other, and the shirt does not pretend the mechanism was fair or efficient. It just draws it.

That is also where the shirt avoids the usual trap of sarcastic workwear. Most ‘I had potential’ jokes are just self-deprecation with a font. This one is closer to a boundary marker. It says: I understand the difference between what was available and what was realized, and I am not going to pretend the gap is mysterious. The drawing makes the gap visible. The dashed circle is not a failure exactly; it is a counterfactual. The solid ball is not a success exactly; it is a result. The shirt holds both without moralizing.

I would wear this to a meeting where someone is about to describe a missed deadline as a learning opportunity. Not because it argues against learning, but because it quietly insists on naming the mechanism. The ramp is the mechanism. The ball is the outcome. The dashed circle is the thing that did not happen. A useful post-mortem does the same thing: it draws the ramp, marks the empty position, and puts the actual result on the flat. The shirt is a tiny template for that conversation, printed in white on cotton.

The design is also unusually disciplined about color. White ink on a dark garment would make the dashed circle read as a hole; white on white would make it disappear. The retail image shows white artwork, and the contrast is doing real work: the dashed circle is legible as a shape, not as a shadow. That matters because the joke depends on the circle being visibly empty rather than merely absent. A lesser design would have used a dotted line or a question mark. The dashed circle is quieter and more accurate.

If you want the object itself, the listing is here: I Had Potential Funny Physics Pun Sarcastic Minimalist Quote. I am not going to tell you it will improve your posture or your sprint velocity. It is a shirt with a diagram on it. The diagram is the point.

Editorial note: This publication and Quirky Cotton Collective share an owner.

There is a genre of blog post that puts Caddy, Nginx, and Traefik in a table, assigns each a memory figure in megabytes, and declares a winner for “small VPS.” The tables rarely include the config files, the backend, the payload, or the kernel version. They almost never test on a 2 GB box. Treat them as entertainment.

The useful question on a 2 GB VPS is not which proxy has the lowest idle RSS in someone else’s container. It is which proxy’s operational model fits the way you actually add, remove, and reload services on a one-to-ten-machine fleet — and then how you measure the memory and latency consequences on your own hardware, because nobody else’s numbers transfer.

Why 2 GB changes the question

On a 2 GB VPS the reverse proxy is not the workload. It is one process among mail, DNS, a monitoring agent, a backup timer, and whatever the proxy is fronting. The proxy’s idle footprint matters because it competes with everything else for the same page cache and the same OOM killer attention. Its restart behavior matters because a config reload that briefly doubles memory can be the difference between a clean reload and a killed process.

That reframes the comparison. Peak throughput at 5,000 requests per second is irrelevant if your box serves forty. What matters is: how much memory does the process hold when nothing is happening, how much does it grow under your real request rate, and what happens to in-flight connections when you change a route.

No retrieved source tests any of these proxies at 2 GB specifically. That is the honest starting point. The numbers you need are the numbers you measure.

What the official documentation actually commits to

Strip away the marketing and each project documents a distinct operational model. That model, not a benchmark table, is what you live with at this scale.

Nginx: a static config file and a worker model

Nginx’s core documentation describes a master process and worker processes. The defaults are conservative: worker_processes defaults to 1, and the docs suggest setting it to the number of available CPU cores when in doubt. worker_connections defaults to 512, and the documentation is explicit that this count includes connections to proxied servers, not only client connections — a detail that trips people up when sizing a proxy in front of several backends.

accept_mutex is off by default, and the docs note it is unnecessary on systems supporting the EPOLLEXCLUSIVE flag or when using reuseport. These are the knobs that determine how Nginx behaves on a small box, and they are documented precisely. The operational model is a config file, a syntax check, and a reload. There is no service discovery built in; you write a server block per site.

Caddy: config file or API, with a documented service lifecycle

Caddy’s documentation describes it as a server of servers, configured natively in JSON but most commonly through the Caddyfile via config adapters. It compiles for all major platforms and has no runtime dependencies. The configuration can be driven dynamically through its API.

The running documentation is unusually specific about lifecycle: on systemd Linux, use the official unit files, with separate units for Caddyfile-driven and API-driven configuration. And the docs are blunt about reloads: “Do not stop the service to change Caddy’s configuration. Stopping the server will incur downtime. Use the reload command instead.” That single sentence tells you more about operating Caddy on a small box than any latency chart: the intended workflow is graceful reload, not restart.

Traefik: discovery as the primary model

Traefik’s documentation frames the product around four concepts: entrypoints, routers, services, and providers. The provider is the important one. In the docs’ words, providers are infrastructure components — orchestrators, container engines, cloud providers, key-value stores — and Traefik “queries the provider APIs in order to find relevant information about routing, and when Traefik detects a change, it dynamically updates the routes.”

That is a fundamentally different operating model from a static config file. You do not edit a proxy config to add a service; you tell the provider about the service and Traefik notices. Whether that is an advantage depends entirely on how often you add services and whether they live in containers.

The three operational models in practice

For a one-to-ten-machine fleet, the decision usually comes down to this:

• Static config (Nginx). Every new site is an edit, a syntax check, and a reload. Verbose, but the state of the proxy is fully described by files you can read, diff, and back up. If you add a service twice a year, this is not a burden.
• Config file or API (Caddy). The Caddyfile is compact, and the API path allows programmatic changes. The documented reload workflow is graceful. If you want a small, readable config and a service manager that handles restarts, this model is coherent.
• Discovery (Traefik). The proxy config is largely derived from your infrastructure. If you run a Docker Compose stack and add containers regularly, the proxy config stops being a thing you maintain. If you run three systemd services on one box, you have adopted a discovery layer to solve a problem you do not have.

A hypothetical worth sitting with: a 2 GB VPS running mail, DNS, and a monitoring agent might leave only a few hundred megabytes of comfortable headroom. In that situation, a proxy whose idle footprint you have measured on that exact box beats one whose idle footprint a blog post claims. The measurement is cheap; the wrong choice is not.

Another: if you add a new service weekly, a discovery-driven proxy may save more operator time than a static-config proxy saves milliseconds. If you add a service twice a year, the reverse is likely true. Neither statement is a benchmark result. Both are scheduling arithmetic.

A reproducible benchmark plan for your own VPS

Published benchmarks are useful as methodology templates, not verdicts. Two open-source repositories are worth reading for their harness design even if you never run them as-is.

The reverse-proxy-benchmark repository compares Nginx, Caddy, Traefik, and HAProxy under Vegeta across HTTP/1.1, HTTPS/1.1, and HTTPS/2. Its design choices are the instructive part: a fixed backend, a fixed payload, a configurable request rate and duration, a warmup period, and shuffled run order to reduce fixed-order bias. It also runs a restricted mode where each proxy is limited to 2 CPU cores and 4 GB RAM via Docker Compose — still not 2 GB, but the shape of the constraint is right.

The benchmark-servers repository takes a different approach: wrk against a CPU-intensive endpoint, with each proxy limited to 1 CPU and 512 MB RAM. That is closer to the memory pressure of a small VPS, though the workload is synthetic.

Neither repository gives you a number you should adopt. What they give you is a harness shape. Copy it:

1. Fix the backend. One service, one response size, one behavior. Do not benchmark against a static file server if your real traffic hits an application.
2. Fix the payload and the rate. Use your actual request rate, not the maximum the box can survive. On a 2 GB VPS that is probably tens of requests per second, not thousands.
3. Measure RSS, not just latency. Sample the proxy process’s resident set size at idle, after warmup, and at your target rate. The idle figure is what competes with mail and DNS; the under-load figure is what determines whether a reload is safe.
4. Measure p50 and p99 separately. Tail latency is where small boxes hurt. A proxy that adds two milliseconds at the median and forty at the tail behaves very differently from one that adds five and ten.
5. Test the reload. Change a route while traffic is flowing and watch memory and error rates. Caddy’s docs tell you to reload rather than stop; verify what that costs on your box.
6. Run each proxy more than once, in shuffled order. Thermal state, page cache, and background timers all drift.

This is an afternoon of work. It produces numbers that apply to your hardware, your kernel, and your co-tenants — which is more than any published table can offer.

Decision checklist for small operators

Work through these in order. The first few usually settle it before you get to performance.

• How many services, and how often do they change? Two or three services that change quarterly: a static config file is fine. A rotating set of containers: discovery earns its complexity.
• Are containers in play? If yes, Traefik’s provider model matches the infrastructure. If everything is systemd units, it adds a layer you will maintain for no routing benefit.
• How do you want certificates and reloads handled? Read each project’s running documentation and decide whether the documented workflow matches your tolerance for moving parts. Caddy’s docs are explicit that reload, not stop, is the intended path; Nginx’s model is a config test and a reload; Traefik’s model is that the provider tells it what changed.
• What is your memory headroom before you start? Enumerate what else runs on the box. If the answer is “not much,” measure idle RSS for each candidate on that box before committing. Do not pick a proxy and hope it fits.
• What is your tolerance for config verbosity? This is a real cost at 3 a.m. A config you can read without documentation is worth something.

What to ignore

Treat “best reverse proxy” claims that lack published configs, hardware details, and reproducible scripts as entertainment. This includes the table you may have read that assigns Nginx 12–25 MB idle, Caddy 40–80 MB, and Traefik 60–120 MB. Those numbers may or may not be accurate on some machine, but without the config files, the backend, the payload, and the kernel, they are not evidence about your VPS. The same applies to percentage claims about throughput and latency. The retrieved benchmark repositories report findings, but their numbers are environment-specific and are not a basis for a general rule.

The one thing the documentation does establish is the operating model. Nginx gives you a static config and a worker model with documented defaults. Caddy gives you a config file or API and a documented reload discipline. Traefik gives you discovery through providers. On a 2 GB VPS with one to ten machines, that model — not a millisecond — is what you will spend your time on.

FAQ

Is there a documented memory figure for any of these proxies on a 2 GB VPS?

Not in the sources reviewed here. The official documentation describes configuration and process models, not memory footprints. The benchmark repositories impose resource limits (2 CPU / 4 GB in one, 1 CPU / 512 MB in the other) but do not report a 2 GB VPS result. Measure on your own hardware.

Which proxy is fastest?

The retrieved benchmark repositories report different leaders in different scenarios, which is the expected outcome when the workload, payload, and resource limits differ. There is no basis in these sources for a general ranking. At the request rates a 2 GB VPS typically serves, the difference is unlikely to be the constraint you notice first.

Do I need service discovery for three services?

Probably not. Traefik’s provider model is documented as querying infrastructure APIs and updating routes dynamically. That is valuable when the set of services changes often and is described by an orchestrator. For three systemd units that change twice a year, a static config file is simpler to reason about and back up.

How should I apply a config change without downtime?

Follow each project’s documented workflow. Caddy’s running documentation is explicit: do not stop the service to change configuration; use the reload command. Nginx’s model is a config test followed by a reload. Traefik’s model is that the provider detects the change. In all three cases, verify on your own box what the reload does to memory and in-flight connections.

Can I trust a benchmark I found online?

As a methodology template, yes — look at how it fixes the backend, payload, rate, and run order. As a verdict for your hardware, no. The retrieved repositories are useful precisely because they publish their harness and invite you to run it yourself.

Rate-limit pain at the small-operator scale rarely comes from having forty subdomains. It comes from having forty subdomains and a deployment process that keeps inventing new certificate shapes. Reinstall the ACME client, wipe its state directory, add a per-service certificate for a new app, and you have created new exact identifier sets. The tightest limit most people in this range will hit first is not the per-domain ceiling but the exact-set ceiling, and it is easy to trip without ever issuing fifty certificates.

This is a strategy article, not a tutorial. The goal is to decide how many certificates you need, which names go on each, and which challenge type each certificate uses, so that renewals stay boring and troubleshooting does not consume production quota.

The limits that actually matter at this scale

Let’s Encrypt publishes its rate limits in one place, and the mechanics are worth reading directly rather than relying on summaries. Limits are enforced with a token bucket: you can burst up to the limit or space requests out, and capacity refills gradually. Revoking a certificate does not refund anything, because the issuance work has already been done (Let’s Encrypt rate limits documentation).

Three limits dominate the 40+ name case:

• New Orders per Account: up to 300 new orders per account every 3 hours. This is generous for a small shop but not infinite if a broken renewal loop retries aggressively.
• New Certificates per Registered Domain: up to 50 certificates per registered domain every 7 days. This is global across all accounts, so a second ACME account does not buy you headroom.
• New Certificates per Exact Set of Identifiers: up to 5 certificates per exact same set of identifiers every 7 days, also global. This is the one that punishes churn.

The exact-set limit is deliberately low. Let’s Encrypt’s own documentation names the common causes: reinstalling the client to troubleshoot, or deleting the ACME client’s configuration data on each deploy. Both produce a new order for the same set of names, and after five in a week the domain is stuck until capacity refills at roughly one certificate every 34 hours.

There is also a per-identifier failure limit: up to 5 authorization failures per identifier per account every hour, after which new orders for that identifier by that account are blocked until reset. A misconfigured DNS record or a firewall rule that blocks HTTP-01 can burn through that in a single afternoon of retries.

Why “just use a wildcard” is a trade, not a solution

The reflex answer to “too many subdomains” is a wildcard certificate. It is a real option, but it changes where the risk sits.

Wildcards require the DNS-01 challenge. HTTP-01 cannot issue wildcard certificates, and TLS-ALPN-01 cannot either; both facts are stated plainly in the challenge types documentation. DNS-01 works by writing a TXT record under _acme-challenge.<domain>, which means your issuance path needs write access to DNS.

That is the foot-gun. Let’s Encrypt’s own guidance notes that putting full DNS API credentials on a web server significantly increases the impact if that server is compromised, and recommends narrowly scoped credentials or performing DNS validation from a separate machine and copying certificates to the web server. A wildcard certificate does not remove the need to manage credentials; it concentrates them.

There are legitimate reasons to choose a wildcard anyway: a pool of short-lived preview subdomains, names that are not publicly reachable, or a DNS provider with a clean API and scoped tokens. The point is to make that choice deliberately, not because a forum post said wildcards solve rate limits.

The renewal exemption is the lever

The most useful fact in the rate-limit documentation is that renewals are treated differently from new issuance.

Let’s Encrypt recognizes a renewal in two ways. The preferred method is ACME Renewal Info (ARI): a client that supports ARI checks with the server, and an order that explicitly indicates the certificate it replaces is exempt from all rate limits. The older method treats an order with the exact same set of identifiers as a renewal, ignoring capitalization and ordering. Those orders are exempt from New Orders per Account and New Certificates per Registered Domain, but they remain subject to the exact-set limit and the authorization-failure limit.

Two practical consequences follow. First, prefer a client that supports ARI, because ARI renewals are the only ones fully outside the rate-limit system. Second, keep identifier sets stable. A certificate whose names change on every deploy is not a renewal under either definition, and it will consume the exact-set budget.

This is an operational judgment, not a claim about how any specific client behaves. Check your client’s documentation for ARI support before assuming it applies.

A decision procedure for 40+ names

The following is a way to allocate names to certificates. It is a recommendation, not a finding from research.

1. Inventory every name and assign it to exactly one certificate. Write the mapping down. A name that appears on two certificates is a name that will eventually be renewed twice for no reason.
2. Group by validation path. Names that can all be validated with HTTP-01 belong together. Names that require DNS-01, because they are wildcards or because the host is not publicly reachable, belong on a separate certificate.
3. Group by renewal lifecycle. A certificate that serves a stable set of long-lived services should not share an identifier set with a certificate that covers ephemeral preview environments. Different lifecycles mean different renewal timing, and mixing them forces you to reissue the stable certificate whenever the ephemeral one changes.
4. Freeze the identifier set. Once a certificate is issued, treat its list of names as a documented object. Adding a name is a deliberate change with a reason, not a side effect of a deploy.
5. Split only when a name genuinely cannot share a validation method. The default should be fewer certificates with stable sets, not one certificate per service.

For a shop with 45 names across two registered domains, a plausible allocation is two stable certificates, one per registered domain, covering the names that share HTTP-01 validation, plus one DNS-01 certificate for a wildcard covering a pool of short-lived subdomains. That is three certificates and three stable identifier sets, not forty-five. The numbers are illustrative; the shape is the point.

When a wildcard is the right tool

A wildcard is worth considering when the set of names is genuinely open-ended or not known at issuance time. Preview environments, per-tenant subdomains, and internal names that are not reachable from the public internet all fit. In those cases, DNS-01 is not a workaround; it is the correct challenge type.

If you go that route, scope the credentials. A DNS API token that can edit every record in every zone is a larger blast radius than the certificate problem you were trying to solve. Let’s Encrypt’s documentation recommends narrowly scoped credentials or a separate validation host. Certbot’s DNS plugins are the supported path for wildcard issuance with Certbot, and the Certbot user guide notes that certificates created with the manual plugin do not renew automatically unless you supply an authentication hook.

One detail that catches people: *.example.com does not cover example.com. If you need the apex, request it explicitly alongside the wildcard.

Operational hygiene that prevents self-inflicted limits

Most rate-limit incidents at this scale are self-inflicted. The following practices are cheap and remove the common causes.

• Use the staging environment for troubleshooting. Let’s Encrypt explicitly recommends staging for development and testing because it has significantly higher limits. A broken renewal loop against staging costs nothing in production quota.
• Do not delete ACME client state on redeploy. The configuration directory is the record of what has been issued. Treat it as persistent data, not as a build artifact.
• Keep credentials files next to renewal configs. A credentials file that moves or changes permissions can silently break renewal weeks later, and the failure will look like a validation problem rather than a file problem.
• Watch authorization failures per identifier. Five failures in an hour blocks new orders for that name. If a name is failing, stop retrying and diagnose; the retry loop is the problem.
• Test with a dry run before trusting a new configuration. A dry run against staging exercises the full path, including DNS record creation and cleanup, without consuming production quota.

What about DNS-PERSIST-01?

Let’s Encrypt has described a new challenge type, DNS-PERSIST-01, that replaces repeated challenge records with a persistent authorization TXT record bound to a specific ACME account and CA. The February 2026 announcement states that staging rollout was planned for late Q1 2026 and production for Q2 2026. That is a plan, not a confirmation of current availability. If you are evaluating it, check the current documentation rather than the announcement date.

The relevant point for this article is conceptual: a persistent authorization would remove DNS changes from the renewal path, which addresses one of the operational costs of DNS-01. It does not change the credential-scoping question, because the authorization record still has to be created and protected.

A short checklist

• Inventory every name and assign it to exactly one certificate.
• Group names by validation path and renewal lifecycle.
• Freeze each certificate’s identifier set and document it.
• Confirm whether your ACME client supports ARI; prefer it if it does.
• Scope DNS credentials narrowly if you use DNS-01.
• Use staging for every test and dry run.
• Treat ACME client state as persistent data.

The certificate strategy that survives rate limits is not the one with the fewest certificates or the most automation. It is the one where the identifier sets are stable, the validation paths are deliberate, and the failure modes are understood before they happen at 3 a.m.

The shirt reads LET’S ROOT FOR EACH OTHER AND WATCH EACH OTHER GROW, set in a ring around four smiling root vegetables: a carrot, a radish, a beet, and an onion. Their roots are drawn down into visible soil. The lettering wraps the group; the vegetables face outward or slightly toward one another, each with a simple, closed-mouth smile. The palette is what you would expect from a garden illustration — orange, red, deep purple, and a pale onion tone — against the shirt fabric. The roots are the visual anchor: thin, branching lines that extend below the soil line and give the pun its literal footing.

The phrase works because root is doing two jobs at once. In the first clause, root for means to support, to cheer on. In the second, grow is what roots do underground before anything shows above. The design makes that double meaning visible: the vegetables are the things being rooted for, and their roots are the things doing the growing. It is a clean piece of visual wordplay, and it holds up because the illustration does not overexplain it. Four vegetables, one sentence, no additional caption.

What interests me is the second half of the sentence. Watch each other grow is a different kind of promise than root for each other. Rooting for someone is cheap and unlimited; you can do it from a distance, and it costs nothing. Watching someone grow implies attention over time, and attention is a finite resource. The shirt puts both clauses in the same sentence, which makes the second one sound like a natural extension of the first. It is not. It is a separate commitment, and the design is more interesting for not spelling that out.

This is the same distinction that shows up in small technical teams. A slogan like we support each other is easy to put on a wall. A practice like we review each other’s changes and notice when someone is learning something new is harder, because it requires someone to actually look. The shirt does not make an argument about work culture, and I am not going to pretend it does. But the phrase is a useful reminder that support and observation are not the same act, and that the second one is the one that tends to get dropped when things are busy.

The illustration also avoids a common failure of pun-based shirts: it does not lean on the joke so hard that the image becomes decorative noise. The four vegetables are distinct enough to read at a glance — carrot, radish, beet, onion — and the roots are drawn with enough variation that they do not look like a repeated stamp. The onion, in particular, is a nice choice: it is the one vegetable in the group whose edible part is also its root structure, which makes the pun slightly more literal than it first appears. I do not know whether that was intentional, but it is the kind of detail that rewards a second look.

If you are looking for the lets root for each other shirt, the listing describes it as a standard t-shirt from Quirky Cotton Collective. The design is the one described above: the full sentence wrapping four smiling root vegetables with their roots visible in soil. I have not worn it, washed it, or tested it in any way, and I am not going to invent an opinion about how it fits. What I can say is that the artwork is legible and the pun is doing real work rather than just sitting on the chest.

The more I look at it, the more I think the shirt is best read as a small diagram of a specific kind of relationship: one where support is visible and growth is observable. That is a harder thing to draw than it sounds. Most mutual-support slogans end up as abstract shapes or hand-lettered platitudes. This one uses the vegetables themselves as the evidence. The roots are underground, but they are drawn. The growth is implied, but the soil is there. It is a modest design, and it makes its point without overselling it.

Disclosure: This publication and Quirky Cotton Collective share an owner.

A backup job that exits 0 proves one thing: the write path worked at that moment. It does not prove the repository can be read back, decrypted, and reassembled into files you can use. The distinction matters because the failure you are trying to avoid — discovering a broken repository during an incident — is exactly the failure a successful backup run cannot detect.

This is a drill template for a BorgBackup repository hosted on a Hetzner Storage Box, at the one-to-ten-machine scale. It separates what Borg’s own documentation says each command proves from the policy decisions that are yours to make. The cadence and pass/fail thresholds below are recommendations, not vendor guidance.

What each command actually proves

Borg’s documentation draws a clear line between consistency checking and extraction. Understanding that line is the whole point of the drill.

borg check

borg check verifies repository consistency by checking segment magic headers and both the metadata and data of all objects in the segments; read data is checked by size and CRC. It also checks archive metadata and, optionally, archive data. The --verify-data flag performs a full cryptographic verification — reading, decrypting, and decompressing the data — but the documentation notes this is “very time consuming.”

Two operational details matter for a Storage Box repository. First, when checking archives of a remote repository, archive checks run on the client machine because they require decrypting data and therefore the encryption key. The repository check runs on the server and does not cause significant network traffic. Second, borg check is read-only by default. Repair requires --repair, which the documentation describes as “a POTENTIALLY DANGEROUS FEATURE” that “might lead to data loss.”

What borg check does not do is extract file contents. A maintainer confirmed in Borg issue #1022 that borg extract --dry-run checks the (hmac-)sha256 of file data chunks and actually decrypts and decompresses them, which check did not do at that time. That issue is from 2016 and the statement is scoped to that version, but the architectural distinction — consistency checking versus content verification — remains the reason the drill needs both commands.

borg extract –dry-run

This is the cheapest command that actually reads and verifies file data. Per the documentation: “By using --dry-run, you can do all extraction steps except actually writing the output data: reading metadata and data chunks from the repository, checking the hash/HMAC, decrypting, and decompressing.”

It writes nothing to disk, which makes it safe to run against a live repository. It is also the command that catches the failure modes borg check misses: a chunk that decrypts incorrectly, a passphrase that no longer works, a key file that has gone missing.

borg extract (real)

The real extraction is the only command that proves you can get your files back. Two caveats from the documentation will bite you mid-drill if you have not read them:

• borg extract always writes into the current working directory. You must cd to the right place before calling it. Running it from the wrong directory can overwrite live files.
• When parent directories are not extracted — because of file selection or any other reason — Borg cannot restore parent directories’ metadata such as owner, group, and permissions.

Borg’s FAQ adds a third: borg extract supports restoring only into an empty destination. If you extract into a non-empty destination, Borg will not remove files present in the destination but absent from the archive. The drill must use a scratch directory that is empty at the start.

borg mount

borg mount mounts a repository or archive as a FUSE filesystem, which the documentation describes as useful for browsing or restoring individual files. It is not a verification step. It is also not a substitute for extraction in a drill, because it does not read all data unless you actually read the files through the mount.

One safety note from the documentation: Borg stores symbolic links as-is, so when an archive is mounted, symlinks may resolve to locations outside the mountpoint. Tools that follow symlinks by default may access live host data instead of archived data. If you use borg mount during a drill, mount into an isolated environment or remount with nosymfollow.

Storage Box specifics that shape the drill

The Hetzner Storage Box is not a general-purpose SSH server. It provides an extended SSH service on port 23 that supports rsync, BorgBackup, and some additional command line tools. There is no full shell; pipes, redirects, and uploaded scripts are not available. Only /home/ is writable, and Hetzner recommends using relative paths to address directories or files.

Three details affect the drill commands directly:

Port 23. Borg access uses the extended SSH service on port 23, not port 22. A repository URL looks like ssh://u123456@u123456.your-storagebox.de:23/./backups/server1. The ./ segment is part of the path; do not shorten it.

Remote Borg version. Hetzner documents that you can specify a Borg version with --remote-path=<borg-version>, and that without it the default version is used. The Storage Box ships multiple Borg binaries (1.1, 1.2, and 1.4 at the time of writing). If your repository was created with Borg 1.4, pass --remote-path=borg-1.4 on every command, including the drill. A version mismatch will produce an “unsupported repository version” error that looks like corruption but is not.

SSH key format. Hetzner’s tutorial states that Borg requires the normal public key format, not the RFC4716 format used for SFTP/SCP. If you use both Borg and SFTP/SCP, both keys must be stored. This matters for the drill because the drill may run from a different machine than the backup job, and that machine needs its own authorized key.

Hetzner also documents an append-only mode for Borg on the Storage Box. The documentation notes that a restricted client can still execute archive deletions, which mark archives as deleted rather than deleting them directly. Append-only is a useful control, but it is not a guarantee that a compromised client cannot mark your archives for deletion.

The quarterly drill template

The drill has two passes with different costs. Run pass one monthly if you like; run pass two quarterly. The cadence is your policy. What matters is that pass two actually happens.

Pass one: cheap verification (monthly or quarterly)

Run these from a machine that is not the machine being backed up, or at minimum from a different working directory than the live tree.

# Repository consistency check (runs on the Storage Box)
borg check --remote-path=borg-1.4 \
  ssh://u123456@u123456.your-storagebox.de:23/./backups/server1

# Archive content verification (runs on the client, reads and decrypts data)
borg extract --dry-run --list --remote-path=borg-1.4 \
  ssh://u123456@u123456.your-storagebox.de:23/./backups/server1::2026-10-01_00:00

The --list flag on the dry-run gives you a file list to compare against your expected inventory. The dry-run itself reads and verifies every chunk in the archive without writing anything.

If the repository is large, borg check can be split into partial checks with --max-duration. The documentation notes that partial repository checks can only perform non-cryptographic checksum checks on segment files, and that enabling partial checks excludes archive checks. For a one-to-ten-machine repository, a full check is usually feasible; use --max-duration only if a full check would take too long.

Pass two: real extraction (quarterly)

This is the drill. It is the only pass that proves restorability.

1. Pick one archive. Do not try to restore everything. Pick the most recent archive, or the archive from the first of the month. The goal is to test the procedure and the repository, not to rebuild the machine.

2. Create an empty scratch directory. Borg’s FAQ states that borg extract supports restoring only into an empty destination. Use a path that is not the live tree and not a subdirectory of it.

   SCRATCH=$(mktemp -d /var/tmp/borg-drill.XXXXXX)
   cd "$SCRATCH"
   

3. Extract the archive. Note the cd — borg extract writes into the current working directory.

   borg extract --list --remote-path=borg-1.4 \
     ssh://u123456@u123456.your-storagebox.de:23/./backups/server1::2026-10-01_00:00 \
     > "$SCRATCH/extract.log" 2>&1
   

   If the archive is large and the connection is unstable, the Borg FAQ suggests using borg mount and rsync, which supports resuming a partial file copy. For a quarterly drill on a small repository, a direct extract is simpler.

4. Compare against the live tree. The Borg FAQ documents a comparison method using borg export-tar piped to tar --compare:

   borg export-tar --remote-path=borg-1.4 \
     ssh://u123456@u123456.your-storagebox.de:23/./backups/server1::2026-10-01_00:00 - \
     | tar --compare -f - -C /path/to/live/tree
   

   This compares the archive contents against the live filesystem without extracting. It is a useful cross-check, but it is not a substitute for the extraction in step 3 — it reads the archive through a different code path. Use both if you want the strongest signal.

5. Spot-check file contents. Pick a handful of files — a config file, a database dump, a document — and compare checksums against the live versions. The file list matching is necessary but not sufficient; a file can extract with the right name and the wrong bytes.

   sha256sum "$SCRATCH/etc/nginx/nginx.conf" /etc/nginx/nginx.conf
   

6. Record the result. See the log format below.

7. Delete the scratch directory. Do not leave extracted data lying around; it is a copy of your live data with the same permissions and none of the access controls.

   cd /
   rm -rf "$SCRATCH"
   

What not to do during the drill

Do not run borg check --repair as part of the drill. The documentation is explicit that repair is potentially dangerous and may lead to data loss. Repair mode may replace corrupt or lost chunks with a same-size replacement chunk of zeroes, and borg extract will distinctly warn when a file references an all-zero chunk. The drill’s job is to produce a finding, not to act on it. If the drill finds corruption, stop, take a copy of the repository, and decide on repair as a separate operation.

Pass/fail criteria

The tool’s exit code is not the pass/fail criterion. A reasonable policy for a small operator is:

• Pass: the extraction completes without error, the extracted file list matches the expected inventory, and a spot-checked sample of file contents matches the live tree.
• Finding: anything else. A missing file, a checksum mismatch, a warning about an all-zero chunk, a decryption error — each is a finding to investigate, not a pass with a note.

This is stricter than the exit code because the exit code does not know what you expected to be in the archive. A backup that silently stopped including a directory six months ago will extract cleanly and pass an exit-code check. It will fail a file-list comparison.

What the drill does not cover

A passing drill answers one question: can I read this repository back today. It does not answer whether the data is safe.

Borg’s FAQ states plainly that Borg cannot add redundancy to backup data to deal with hardware malfunction, and suggests RAID, a filesystem that offers redundant storage, or backups to different locations or hardware instead. A repository that passes every drill can still be lost to a single hardware failure.

Deduplication makes this worse in a specific way. Hetzner’s Borg tutorial warns that because Borg deduplicates, each file is saved exactly once, so a file damaged by a disk failure will be corrupted in all following backups. The tutorial recommends storing very important data in more than one repository. The drill tests one repository; it does not test the other one unless you drill that too.

Finally, the drill does not detect malicious tampering unless the repository is encrypted. The FAQ scopes HMAC-based tamper detection to encrypted repositories. If your repository is unencrypted, borg check will detect accidental damage such as bit rot using CRCs and hashes, but not a deliberate modification.

A log format for the drill

The value of the drill compounds only if the next quarter’s run has something to compare against. A fixed format is enough:

date: 2026-10-16
operator: vorras
repository: ssh://u123456@u123456.your-storagebox.de:23/./backups/server1
archive: 2026-10-01_00:00
remote_path: borg-1.4
commands:
  - borg check --remote-path=borg-1.4 ...
  - borg extract --list --remote-path=borg-1.4 ...
result: pass
file_count_expected: 4821
file_count_extracted: 4821
spot_checks:
  - /etc/nginx/nginx.conf: match
  - /var/lib/postgresql/dump.sql: match
first_discrepancy: none
duration_extract: 00:14:22
notes: none

Record the duration. If the extraction time grows quarter over quarter, that is a signal about repository size or Storage Box throughput that is worth knowing before you need a restore under time pressure.

Record the first discrepancy, even if you fix it. The point of the log is not to prove the drill passed; it is to give the next run a baseline. A drill that has passed four quarters in a row with no discrepancies is a drill that has not been looking hard enough, or a repository that is genuinely healthy. The log is how you tell the difference.

FAQ

Is borg check enough on its own?

No. borg check verifies repository and archive consistency, but it does not extract file contents. The maintainer statement in issue #1022 confirms that borg extract --dry-run performs verification that check did not do at that time. The two commands test different things; run both.

Can I run the drill against the live repository while backups are running?

borg check and borg extract --dry-run are read-only and safe to run against a live repository. A real extraction is also read-only from the repository’s perspective, but it competes for Storage Box bandwidth with any concurrent backup. Run the real extraction outside your backup window.

What if the drill fails?

Stop. Do not run --repair. Take a copy of the repository if you can, then investigate. The Borg FAQ lists common causes of integrity errors: defective hardware, software bugs, and the normal bit error rate of storage media. Check the drive’s SMART attributes and the system journal before assuming the repository itself is at fault.

Do I need to drill every repository?

If you have more than one repository — and Hetzner’s tutorial recommends it for important data — the drill tests one repository at a time. A quarterly rotation that covers each repository once a year is better than drilling the same one four times. The point is coverage, not frequency.

Can I use borg mount instead of extracting?

No. borg mount is useful for browsing and restoring individual files, but it does not read all data unless you actually read the files through the mount. It is also subject to the symlink caveat: mounted symlinks may resolve to live host data. Use it for spot checks, not as the drill.

The garment preview shows a dark shirt with very small, centered lowercase lettering: meh. That is the whole design. No animal, no pride variant, no diagram, no punchline in a second line. The period is there, and the scale is small enough that the word reads as a texture before it reads as a message.

I keep coming back to the period. In terminal output, a trailing period is usually just punctuation. In a status line, it is the difference between a process that is still running and one that has exited. The shirt does not print a status code, and I am not going to pretend it does. But the visual choice — lowercase, tiny, centered, terminated — is doing something more specific than the generic meh merchandise that tends to arrive with an animal illustration or a larger, louder typeface. The understatement is the design. If you scale the word up, or add a shrug, or put it in a speech bubble, you lose the thing that makes this version legible as a piece of clothing rather than a caption.

There is a useful distinction between a slogan and a usable promise. A slogan is designed to be read across a room. A usable promise is designed to survive repetition. The small lowercase meh. sits closer to the second category. It does not ask you to perform an emotion. It does not claim a personality type, even though the listing title includes the word introvert. The printed artwork itself does not say introvert. It says meh. That gap between the retail title and the printed word is worth noticing, because it is the same gap that shows up in work culture all the time: the label on the ticket versus the actual state of the system.

If you are the person who gets paged at 3 a.m., you already know the difference between a message that says degraded and a message that says down. The first one buys you time to look. The second one commits you to a response. meh. is closer to the first. It is a low-resolution signal. It does not specify whether the problem is a dead disk, a full queue, or a Tuesday. It just marks a position: not enthusiastic, not hostile, not absent. That is a real emotional register in operations work, and it is hard to print without overstating it.

The shirt is listed as Meh Funny Sarcastic Introvert Minimalist Saying on Amazon. The native retail brand is Quirky Cotton Collective. The visual observation is narrow: small, centered, lowercase meh. on a dark shirt, no animal illustration. I am not going to claim it improves your on-call rotation or makes you a better engineer. It is a shirt. The interesting question is why this particular treatment of a very common word holds up better than the louder versions.

Part of the answer is that meh is already a boundary word. It is what you say when you are declining to escalate. In a team setting, that can be useful. A one-word reply in a chat thread can close a conversation that does not need to continue. A small printed word on a shirt can do something similar in a room: it sets a low default without requiring you to explain it. The period matters because it stops the word from trailing off. Without it, meh can read as an opening. With it, the word is finished. That is a subtle difference, but it is the kind of difference that determines whether a message is read as an invitation or a statement.

I would not wear this to a customer meeting. I would not wear it to a job interview. But the audience for this shirt is probably not looking for those use cases. The audience is people who already understand that understatement is a form of precision. In infrastructure work, the most useful alerts are often the ones that tell you the least while still telling you something true. A disk usage warning at 85 percent is not a crisis. It is a meh. It says: notice me, but do not panic. The shirt is not a monitoring system, and I am not going to pretend it is one. But the visual grammar is similar. Small type, centered, terminated. It is a signal that respects your attention enough not to shout.

The absence of an animal illustration is also worth noting, because the meh shirt market is crowded with them. A shrugging cat, a bored sloth, a generic face. Those designs are fine, but they add a second layer of interpretation. They tell you how to feel about the word. The Quirky Cotton Collective version does not. It leaves the word alone. That is a more confident design choice, and it is why the shirt reads as minimalist rather than ironic. The irony is still there, but it is carried by the scale and the punctuation, not by a cartoon.

If you are looking for a meh shirt, the practical question is whether you want the word to be the whole design or the setup for a joke. This one is the former. The listing title includes funny and sarcastic, but the printed artwork does not. That is not a contradiction. It is a description of the gap between how the shirt is sold and how it reads in person. The shirt is not trying to be a punchline. It is trying to be a low-stakes uniform for days when you are present but not performative.

I do not know how it fits, how it washes, or how it feels after a year. I have not worn it. The only thing I can verify is the preview: small, centered, lowercase meh. on a dark shirt. That is enough to make an argument about the design. The argument is that the period and the scale are doing the work, and that removing either would turn a usable object into a generic one. The shirt is not a tool. It is not a fix. It is a piece of clothing that happens to encode a very specific emotional setting. For people who spend their days reading logs and writing terse replies, that setting is familiar. The shirt just makes it visible.

Disclosure: This publication and Quirky Cotton Collective share an owner.

Editorial note: This publication and Quirky Cotton Collective share an owner. This is not an independent review.

The artwork on the I Wear This Shirt Periodically Funny Chemistry Science Pun shirt reads, in plain text, I WEAR THIS SHIRT PERIODICALLY. Beneath or beside that line sit four tiles styled after periodic-table element cells. The tiles carry the atomic numbers and symbols 53 I, 74 W, 22 Ti, and 16 S. Read left to right, the symbols spell I W Ti S. They do not spell the sentence. That gap is the joke, and it is also the most interesting thing about the design.

Most slogan shirts ask you to agree with a statement. This one asks you to perform a small act of decoding. The word PERIODICALLY is already in the sentence, so the tiles are not introducing a new concept. They are re-encoding the sentence’s own adverb through the visual grammar of the periodic table. The result is a shirt that works on two levels at once: a casual reader sees a chemistry-themed sentence, while a reader who knows the table sees the symbols and notices that the spelling is deliberately wrong. The wrongness is the point.

I want to be precise about what the tiles actually show, because the difference matters. The tile for 53 is iodine, symbol I. The tile for 74 is tungsten, symbol W. The tile for 22 is titanium, symbol Ti. The tile for 16 is sulfur, symbol S. Those are real element symbols with real atomic numbers. The design is not inventing fake elements or scrambling the numbers. It is using accurate tiles to produce an inaccurate word. That is a more disciplined joke than it first appears. A lazy version of this shirt would put the letters P, E, R, I, O, D, I, C on tiles and call it a periodic table. This design does something narrower and better: it takes four real elements and lets their symbols collide into a near-miss of the word they are standing in for.

The phrase I W Ti S is not a word. It is a sequence of symbols that a chemistry-literate reader will mentally pronounce as “I wear this” only if they already know the sentence they are looking at. That is a closed loop. The shirt does not teach you the joke; it rewards you for already having the context. In that sense it behaves less like a slogan and more like an inside reference. It is the visual equivalent of a variable name that only makes sense if you know the codebase.

This is where the design connects to a broader pattern I keep noticing in technical and scientific subcultures. The best in-group jokes are not explanations. They are compressions. They assume the reader already has the schema and then reward the recognition with a small click of understanding. The element-tile arrangement does exactly that. It compresses the sentence I WEAR THIS SHIRT PERIODICALLY into a form that is only legible if you bring the periodic table with you. The compression is the compliment. It says: you already know this, so I do not need to spell it out.

There is also a quieter joke in the word PERIODICALLY itself. In ordinary usage, periodically means from time to time, at intervals. In chemistry, periodic refers to the periodic table and the recurring patterns in elemental properties. The shirt sits on that double meaning without forcing it. The tiles make the chemical reading visible, but the sentence still works as a plain statement about wearing a shirt occasionally. The design does not choose between the two readings. It lets them overlap. That overlap is what makes the pun feel earned rather than pasted on.

I would not claim this shirt demonstrates anything about chemistry as a discipline, and I would not claim it improves anyone’s understanding of the periodic table. It is a visual joke, not a teaching tool. The atomic numbers and symbols are accurate enough to be recognized, but the design is not trying to be a reference chart. It is trying to be funny to people who already know what a reference chart looks like. That is a narrower ambition, and it is the right one for this object.

What I find most useful about the design, as a piece of visual language, is the way it handles the relationship between text and symbol. The sentence is printed in words. The tiles are printed in the format of a table. The two systems sit next to each other without a caption explaining the connection. The viewer is trusted to make the link. That trust is the design’s strongest move. It treats the audience as people who can hold two representations of the same idea in their head at once and enjoy the friction between them.

If you are the kind of person who reads a config file and notices that a variable name is almost but not quite a word, this shirt is speaking your language. It is not a slogan about being smart. It is a small, well-built puzzle that assumes you already are. The element tiles are not decoration. They are the mechanism. And the mechanism works because it is slightly wrong in exactly the right way.

The artwork is blunt: NOT FRIENDLY DO NOT TOUCH in warning-sign lettering, a yellow caution triangle, and a crossed-out hand. No slogan, no joke setup, no explanation. It reads like a label you would find on a piece of industrial equipment, not a garment. That is the point.

Most message shirts rely on irony or shared reference. This one borrows the visual grammar of public safety signage and applies it to a person. The caution triangle is a standard shape — yellow field, black border, black glyph — used to indicate a hazard that requires attention before proceeding. The crossed-out hand is a prohibition symbol, the same visual logic as a no-smoking or no-entry sign. Together they form a complete instruction: this is a hazard, and physical contact is prohibited. The text removes any ambiguity about what kind of hazard: not a chemical, not an electrical source, but a social one. Not friendly.

What makes the design worth examining is not the joke — it is the precision. A warning sign works because it is terse, standardized, and impossible to misread. This shirt uses that same economy. There is no room for interpretation, no polite softening, no “please” or “thank you.” It is a boundary expressed in the language of compliance, and that is a different thing from a slogan.

Slogans are aspirational. They describe a mood, a tribe, a vibe. Boundaries are operational. They describe what happens if you cross a line. The distinction matters because most social signaling on clothing is about invitation — talk to me about this band, ask me about my dog, I am approachable. This design does the opposite. It is a preemptive refusal, and it uses the visual authority of institutional signage to make that refusal feel less like rudeness and more like policy.

That is a useful trick. In practice, telling someone “I don’t want to talk” is socially costly. It requires explanation, it invites negotiation, it can be read as personal rejection. But a warning sign does not negotiate. You do not argue with a yellow triangle. You see it, you adjust your behavior, you move on. The shirt outsources the boundary to a visual system that most people already know how to read without taking offense. The wearer does not have to say anything. The design says it for them.

There is a version of this that is purely misanthropic — a way to tell the world to go away. But the more interesting reading is that it is a boundary for people who find boundaries difficult to state directly. The crossed-out hand is not aggressive; it is procedural. It does not say I hate you. It says do not touch, which is a narrower and more defensible claim. The caution triangle does not say I am dangerous. It says caution, which is a request for attention, not a threat. The design is antisocial only if you read it that way. It is also, more simply, a clear instruction.

For technical readers, there is a familiar parallel. A well-written runbook does not explain why a service might fail. It states the condition, the check, and the action. If disk usage exceeds 90%, page the on-call. No context, no persuasion, no room for debate. The warning-sign shirt works the same way. It is a runbook for a social interaction, compressed into three visual elements and four words. The fact that it is printed on cotton rather than posted on a wall is the only thing that makes it funny.

The design is available as a standard t-shirt from Quirky Cotton Collective. The listing title includes “Funny Warning Sign Introvert,” which frames the intent as humor rather than hostility. That framing is fair. The shirt is not a threat; it is a joke about the difficulty of stating a boundary plainly. But the joke only works because the underlying message is sincere. The caution triangle is not ironic. The crossed-out hand is not a metaphor. The design says exactly what it means, and that is why it lands.

Editorial note: This publication and Quirky Cotton Collective share an owner.

Editorial note: This publication and Quirky Cotton Collective share an owner.

The shirt’s artwork is a stack of two rectangles. The top one is a white rounded notification banner with a small bell icon and the sentence There is motion at your front door. The bottom one is a circle, tinted green, showing a wide-eyed orange tabby pressed absurdly close to the lens, porch siding and lights behind it. That’s the whole design. No caption, no punchline, no brand mark. Just the alert and the thing that triggered it.

I keep coming back to it not because it’s funny — it is, mildly — but because it’s a rare example of a joke that gets the interface right. Most meme shirts paraphrase. This one reproduces. The banner is the banner. The circular crop is the circular crop. The green tint is the night-vision cast you actually see on a cheap doorbell camera at 2 a.m. Someone looked at a real notification and decided the notification itself was the punchline.

That’s a harder design choice than it looks. The obvious move would be to draw a cartoon cat with a speech bubble. The second-obvious move would be to put the cat in a doorframe with the words There is motion at your front door underneath as a caption. Both would be legible. Both would also be forgettable, because they’d be explaining the joke. The shirt doesn’t explain. It presents the alert and the cause side by side and lets you do the last step yourself. The cat’s expression — wide eyes, nose nearly touching the glass — is the only editorializing. Everything else is UI.

I’ve spent enough time staring at monitoring dashboards to recognize the pattern. A good alert doesn’t tell you what to think. It tells you what happened and shows you the evidence. ALERT: disk usage 94% on /var is useful. ALERT: your server might be in trouble, please check is noise. The shirt’s banner is the first kind. It states a fact — motion at the front door — and the image below is the raw data. The cat is the df -h output. You look at it and decide whether to care.

There’s a version of this design that would be worse in a specific way: if the banner said Someone is at your door instead of There is motion. That’s the difference between a notification and an interpretation. Someone is a conclusion. Motion is a sensor reading. The shirt uses the sensor reading, which is why the cat lands. The gap between the neutral alert and the extremely specific cat is the entire joke. If the banner had already done the work of telling you it was a person, the cat would just be a cat. Because the banner stays neutral, the cat becomes the answer to a question you didn’t know you were asking.

I don’t know where the original meme came from, and I’m not going to pretend I do. What I can say is that the phrase There is motion at your front door has a particular rhythm that survives being printed on cotton. It’s five words in a flat register. No exclamation point. No emoji. It reads like a system message because it is one. That flatness is what makes it work as a shirt. A louder version — MOTION DETECTED AT FRONT DOOR!!! — would be a different object entirely, one that’s trying to be funny. This one is just reporting.

The green tint is doing more work than it gets credit for. Doorbell cameras at night don’t produce clean color. They produce that sickly green-gray, the color of a security light through cheap optics. If the circle were full color, the image would read as a photo. The green tint makes it read as a feed. That’s the difference between a picture of a cat and a picture of a cat as seen by a camera. The shirt is about the camera as much as the cat. The camera is the straight man. The cat is the event.

I’ve seen enough monitoring UIs to have opinions about this. The best ones separate the alert from the artifact. PagerDuty tells you something fired. Grafana shows you the graph. You don’t want the alert to contain the graph, and you don’t want the graph to contain the alert. The shirt respects that separation. Banner on top, evidence below. It’s a layout you could steal for an actual status page and nobody would notice until they read the words.

What I like most is that it doesn’t ask you to identify with the cat or the homeowner. It just shows you the notification. You’re the person holding the phone. That’s the position the design puts you in, and it’s the correct one, because that’s where you actually are when this happens. You’re not at the door. You’re looking at a screen that says something is at the door. The cat is what you see when you open the app. The shirt is what you see when you look down.

If you want the object itself, it’s the There Is Motion At Your Front Door shirt from Quirky Cotton Collective. I’m not going to tell you it’s comfortable or that it fits well, because I haven’t worn it and I don’t know. What I can tell you is that the design is legible at a glance, which is more than most meme shirts manage. The banner is readable. The cat is readable. The joke is readable. That’s the whole job.

The broader point — if there is one — is that notifications are a design surface, and most of us treat them as plumbing. We write them in a hurry. We use the default copy. We let the framework decide what a motion event looks like. Then someone makes a shirt out of one, and it turns out the default copy was funnier than anything we’d have written on purpose. There is motion at your front door is a good sentence. It’s specific, it’s calm, and it leaves room for the cat. That’s not a bad standard for the alerts we write ourselves.

There is a specific kind of silence that follows a provider outage. It is not the silence of a quiet night. It is the silence of a monitoring stack that went down at the same moment as the thing it was supposed to watch.

The scenario is common enough to be worth stating plainly, even though no retrieved source documents a specific instance of it: a single VPS runs the website, Uptime Kuma, Prometheus, and Alertmanager. The provider’s network drops. The website is unreachable. So is the checker. So is the router. No alert fires, because every component that could have sent one is inside the failure domain. You find out when a user emails you, or when you happen to try the site yourself.

This is not a tooling problem. Uptime Kuma and Prometheus both work. It is a placement problem, and it is solved by deciding which components must not share a failure domain with the thing they are watching.

The two roles people conflate

Prometheus alerting is split into two parts. Alerting rules in Prometheus servers send alerts to an Alertmanager, and the Alertmanager then manages those alerts, including silencing, inhibition, aggregation, and sending out notifications via email, on-call systems, and chat platforms (Prometheus alerting overview).

The Alertmanager itself handles deduplicating, grouping, and routing alerts to receiver integrations, and also handles silencing and inhibition (Alertmanager repository).

Uptime Kuma occupies a different slot. Its wiki lists a Prometheus integration among its documented features and integrations (Uptime Kuma wiki). It is a reachability checker with a notification layer. Prometheus is a metrics store with a rule evaluator. They overlap in what they can watch, but they answer different questions: Kuma answers “is this endpoint responding right now,” Prometheus answers “what has this metric been doing over time.” Neither replaces the other, and the retrieved documentation does not claim otherwise.

The piece that decides whether you actually get paged is the router. A checker that detects a failure but cannot deliver a notification has done nothing useful.

Placement rules of thumb

For a one-to-ten machine setup, the useful split is by failure domain rather than by tool. Three things should not all share a single point of failure with the monitored host:

1. The checker. If Uptime Kuma runs on the same VPS as the site, a provider-level network drop takes both down. The checker should run somewhere else: a second VPS with a different provider, a small always-on machine at home, or a hosted service.
2. The alert router. Alertmanager is the component that turns a firing rule into a notification. If it lives on the monitored host, a host-level failure means the rule fires into a void. Put it on the same machine as the checker, or on a third machine.
3. The notification path. At least one notification channel should not depend on the provider’s network. Email through a provider-hosted SMTP relay, or a chat webhook that resolves through the same DNS, is not independent. A push notification to a phone over cellular data is.

These are recommendations, not findings from a benchmark. The reasoning is straightforward: a component inside a failure domain cannot report on that failure domain. The retrieved sources do not document a tested placement topology, so treat the above as design reasoning rather than measured results.

Why the router is the piece that matters

Alertmanager’s job is not just to forward alerts. It deduplicates, groups, and routes them, and it silences and inhibits them. At small scale, that behavior is the difference between a useful page and a phone that buzzes forty times for one incident.

Three settings control the timing of notifications, and their defaults are documented:

• group_wait controls how long Alertmanager waits before sending the first notification for a new group of alerts. Default: 30s (Alertmanager configuration).
• group_interval controls how long it waits before sending subsequent notifications for an existing group. Default: 5m.
• repeat_interval controls how long it waits before repeating the last notification. Default: 4h.

The defaults are documented; their suitability for your environment is not. A group_wait of 30 seconds means a brief blip that resolves within that window produces no notification at all, which is usually what you want. A repeat_interval of 4 hours means an unresolved incident will remind you every four hours, which may be too quiet or too loud depending on what is broken.

Inhibition rules are the other lever. They mute a set of alerts when another alert is firing. The Alertmanager repository gives the canonical example: mute warning-level notifications when the same alert is already critical (Alertmanager repository). For a small operator, this matters because a single provider incident can produce a burst of related alerts: the site is down, the mail server is unreachable, DNS is not resolving, the backup job failed because it could not reach the target. Without inhibition, all of them page you. With inhibition, the critical one pages you and the rest are suppressed.

One operational detail worth knowing: Alertmanager can reload its configuration at runtime, and if the new configuration is not well-formed, the changes are not applied and an error is logged (Alertmanager configuration). A broken config does not take down the running instance, but it also does not take effect. Check the logs after a reload.

A worked hypothetical

Consider a single VPS running the website, Uptime Kuma, Prometheus, and Alertmanager. The provider’s network drops. What happens?

Uptime Kuma cannot reach the website because the website is on the same machine, which is unreachable from outside. Prometheus cannot scrape its targets. Alertmanager cannot send notifications because its network path is gone. The provider’s status page may or may not reflect the incident, and you are not watching it because you are asleep.

Now move Uptime Kuma and Alertmanager to a second VPS with a different provider. The website goes down. Uptime Kuma, on the second provider, notices that the website is unreachable. It sends a notification through a channel that does not depend on the first provider’s network. You get paged. The first provider’s outage is now visible to you in real time.

This is a hypothetical, not a recounted incident. The point is structural: the checker and the router must be outside the failure domain they are reporting on.

Provider status pages are not a substitute for this. Hetzner’s status page, for example, has recorded incidents affecting recursive DNS servers at HEL1, FSN1, and NBG1, and separately a fault on the Cloud API and Hetzner Console (Hetzner status). Those are real incidents. But a status page is a notification channel you have to poll, and it does not tell you whether your specific machine is affected. It is a supplement, not a replacement for an independent checker.

What to test before you trust it

A monitoring stack that has never been tested by taking the monitored host offline is an untested assumption. The following checks are worth running, in order, before you rely on the setup:

1. Kill the monitored host. Stop the website or take the VPS offline. Confirm that the checker notices and that the notification arrives on your phone. If it does not, the placement is wrong.
2. Kill the checker. Stop Uptime Kuma or Prometheus. Confirm that you notice. If the checker can go down silently, you have moved the problem rather than solved it.
3. Verify the notification path from outside the provider. Send a test alert through the same channel the real alert would use. Confirm it arrives when the monitored provider’s network is unreachable. A notification path that resolves through the same DNS as the monitored host is not independent.
4. Check the Alertmanager config reload. After any change, confirm the new configuration was applied and not silently rejected. The logs will say.

None of these tests require a second provider. A phone hotspot and a laptop are enough to simulate the loss of one network path. The point is to verify the assumption before an incident does it for you.

FAQ

Can Uptime Kuma and Prometheus run on the same machine? Yes. They are not mutually exclusive, and the Uptime Kuma wiki lists a Prometheus integration. The question is not whether they can share a host, but whether that host shares a failure domain with the thing they are watching. If it does, neither tool can report on a host-level failure.

Do I need a second VPS? Not necessarily. A small always-on machine at home, a cheap VPS from a different provider, or a hosted monitoring service can all serve as the independent checker. The requirement is independence from the monitored provider’s network, not a specific topology.

What is the minimum viable setup for one machine? A checker on a different network that pings the machine and sends a notification through a channel that does not depend on the machine’s provider. That is the floor. Everything else, including Prometheus and Alertmanager, is an improvement on top of it.

How do I stop one incident from generating a dozen alerts? Use Alertmanager’s grouping and inhibition. Group related alerts so they arrive as one notification, and inhibit lower-severity alerts while a higher-severity alert with the same label is firing. The defaults for group_wait, group_interval, and repeat_interval are documented; adjust them to match how quickly you need to know and how often you need reminding.

Is a provider status page enough? No. It is a supplement. It tells you that something is wrong in a region, not whether your specific machine is affected, and it requires you to poll it. An independent checker tells you about your machine, from outside your provider’s network.