Mine stopped being that. Photos, documents, the family tree, the budget, the camera feeds, the music, this website — all of it lives at home now, and other people in the house notice when it is down. That changes the rules. Uptime matters, restores have to work, and "I clicked something in a UI six months ago" is no longer an acceptable description of how a service is configured.

So I started calling it homeprod, and I rebuilt it around one requirement: if a machine dies, I should be able to bring it back from a git repo and a backup, without remembering anything.

The shape of it

Two Proxmox nodes in one cluster, a Raspberry Pi 5 that does all the Docker work, a Pi 5 for Home Assistant OS, an ancient Pi 2B running Pi-hole, and a separate physical box for Proxmox Backup Server. The backup server is deliberately not part of the cluster — a backup that shares a failure domain with the thing it backs up is not a backup.

Stateful things get an LXC, stateless things get a container

The split I settled on is boring and has held up well.

Anything with a large data directory — Immich, Jellyfin, Frigate, Paperless, the file server, Syncthing, the private CA — runs as its own LXC on the Proxmox node with the ZFS pool. Those containers are pets. They get snapshots, they get backed up by the cluster backup job, and I do not move them around.

Anything that is just a process in front of some config — the reverse proxy, this website, the dashboard, the dynamic DNS clients — runs as a Docker stack on a single Raspberry Pi 5. Those are cattle. Their entire definition is a compose file in git, so the Pi itself holds nothing I would miss.

That second half is the part worth writing about, because it is where I made the biggest change this year.

Git is the control plane

Docker on that Pi used to be managed through Portainer. It worked, but a good chunk of the setup existed only inside Portainer's database: stacks pasted into a textarea, environment variables typed into a form, edits made at 11pm and never written down. The compose files in my repo had quietly drifted into fiction.

I replaced it with Komodo. The topology is a Core and Peripheries: Core runs as a Docker LXC on the Proxmox node, each Docker host runs a small Periphery agent that connects outbound to Core over the LAN. The agent dials out, so the host needs no inbound ports and no credentials stored on Core's side.

What makes it stick is that the stacks are declared in the repo, not in the UI:

[[stack]]
name = "traefik"
[stack.config]
server = "docker-pi"
project_name = "traefik"
git_provider = "forgejo.internal"
repo = "k/homelab"
branch = "main"
run_directory = "stacks/traefik"
file_paths = ["compose.yaml"]
environment = """
ACME_EMAIL=[[ACME_EMAIL]]
CF_API_EMAIL=[[CF_API_EMAIL]]
CF_DNS_API_TOKEN=[[CF_DNS_API_TOKEN]]
"""

One resources.toml lists every stack, which host it belongs to, and where its compose file lives. Komodo reads it as a resource sync, so adding a service is a pull request, not a session with a UI. Secrets never enter the repo: [[VAR]] references a variable held by Komodo, which it expands into the stack's .env at deploy time.

The loop closes with a webhook. I push to my self-hosted Forgejo, Forgejo posts to Komodo Core, Core tells the Periphery on the Pi to pull and redeploy that one stack. Editing a compose file and pushing it is the deploy.

One snag worth knowing if you self-host Forgejo: its ALLOWED_HOST_LIST for webhooks defaults to external, which silently blocks private addresses. Every delivery to a LAN-hosted Komodo fails until you add the host.

The edge

Traefik v3 terminates everything. The router forwards 80 and 443 to the Pi, port 80 redirects to 443, and services opt in with labels rather than config files.

The certificate setup is the part that took the longest to get right, because three different kinds of name need three different challenges:

DNS-01 is what makes the middle row possible: I get a real Let's Encrypt certificate for a hostname that resolves only inside the house, because the challenge is answered in DNS and never needs an inbound connection. The third row is a private CA running as its own LXC, issuing certificates over ACME just like Let's Encrypt does — the same Traefik config, a different directory URL.

Two ddclient containers keep the A records pointed at my residential IP, one per registrar, because my domains are split across Cloudflare and Namecheap.

Nothing administrative is on the internet

The only things exposed publicly are the two websites Traefik serves. Every admin interface — Komodo, Proxmox, Forgejo, the dashboards — is reachable only over Tailscale.

The trick I like most here is tailscale serve. Rather than putting Komodo's UI behind the reverse proxy and inventing an auth story for it, the LXC runs:

tailscale serve --bg 9120

That publishes the UI at an HTTPS URL with a valid tailnet certificate, reachable only by my devices. No port forward, no proxy rule, no extra login. Tailscale SSH covers shell access the same way, which means the Pi has password authentication disabled entirely and I do not manage authorized_keys by hand — with one LAN fallback key, because I have locked myself out of a machine by trusting an overlay network before.

Pi-hole is LAN DNS, so the internal names resolve in the house and the ad blocking is free.

Rebuilding a host is a playbook

Earlier this month I reinstalled the Docker Pi from scratch — new OS release, new hostname, same hardware. The point of the exercise was to find out whether the rest of this was true.

Host setup is one Ansible playbook: static IP, SSH hardening, Docker from the upstream repo, the shared Docker network, Grafana Alloy for logs and metrics, and the Komodo Periphery agent. Run it against a freshly flashed card and the host is ready to be handed stacks.

Two things in it exist only because the rebuild taught me they were needed:

- name:
    Periphery waits for the network (it does not retry a failed first connect)
  ansible.builtin.copy:
    dest: /etc/systemd/system/periphery.service.d/10-network-online.conf
    content: |
      [Unit]
      Wants=network-online.target
      After=network-online.target

- name: Periphery starts on boot (installer does not enable it)
  ansible.builtin.systemd_service:
    name: periphery
    state: started
    enabled: true

The agent does not retry its first connection, so on a cold boot it raced the network and lost, and the installer never enabled the unit. Both are the kind of detail you discover once and then encode, which is the whole argument for the playbook.

The backup is the part you test

Before wiping the Pi I took a Proxmox Backup Server snapshot of the whole host and marked it protected so no retention job could eat it.

Then I did the thing that matters: mounted the snapshot and compared checksums of the files I actually cared about — the Docker volumes, the Traefik acme.json, the config directories — against the live host. A backup job that exits 0 tells you a job ran. Reading the bytes back and finding them identical tells you that you have a backup.

They matched, and the restore afterwards was uneventful. Certificates came back without being reissued, and the Apple TV pairing in an anonymous volume survived because I had gone looking for it beforehand.

What I would tell past me

State that lives in a UI does not exist. The single biggest improvement was moving stack definitions out of a management tool's database and into a git repo that the management tool reads. The tool became replaceable, which is exactly what you want from a tool.

Write the migration down before running it. My reinstall was a document with numbered phases, a check per service, and a rollback per step. It turned a scary evening into a checklist, and when something did go wrong I knew which step I was on.

Keep an explicit list of what is not in code. The router's port forwards, DHCP reservations, the CA's provisioner, the Home Assistant configuration, the Komodo variables, the DNS records at two registrars. None of it is in the repo, so the repo documents where each one lives. That list is the real disaster recovery plan.

Delete the experiments. My repo had Kubernetes manifests for a cluster that no longer exists, Ceph notes for a thumb drive I pulled out a year ago, and compose files for four services I stopped running. Every one of them was a trap for future me, reading a file and assuming it described reality. Homeprod means the repo contains what runs, and nothing else — the experiments live in a tag if I ever want them back.

The lab was more fun. This is better.

Note that this method works if you instlled Proxmox using ZFS.

Setup

Get your new drive ready, and make sure it is the same size or larger than the old one. Plug in your drive into an empty bay in the Proxmox machine, or attach it with an enclosure using USB. The idea is to have both drives available to Proxmox.

Let's assume

Old drive: /dev/sda New drive: /dev/sdb

Step 1: Prepare partitions on the new drive

The easy way to do this is to copy over the partitions using sfdisk

sfdisk -d /dev/sda > pve-boot-partitions.txt

cat pve-boot-partitions.txt

label: gpt
label-id: FDB673E1-0941-43AB-BB02-B8A6C3D0D0F8
device: /dev/sda
unit: sectors
first-lba: 34
last-lba: 122142686
sector-size: 512

/dev/sad1 : start=          34, size=        2014, type=21686148-6449-6E6F-744E-656564454649, uuid=4CC31FA5-A6FC-42DB-9071-DF76FBA08F60
/dev/sad2 : start=        2048, size=     1048576, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=B0560706-84DB-45B1-AF0D-BB32DD7F4476
/dev/sda3 : start=     1050624, size=   121085953, type=6A898CC3-1DD2-11B2-99A6-080020736631, uuid=D8BA81D1-0C84-49A8-99B6-F32C041A2FF3

This is also the input to sfdisk. By modifying the following in this file, it can be used to create partitions on the new drive.

1. Remove last-blk line. This is tell sfdisk to use the entire disk.
2. Remove UUIDs for all partition. This will let sfdisk generate new UUIDs for the new drive.
3. Replace size with - for the last partition. This will let sfdisk use the remaining space on the disk for the last partition.

It should look like this:

# cat pve-boot-partitions.txt

label: gpt
label-id: FDB673E1-0941-43AB-BB02-B8A6C3D0D0F8
device: /dev/sda
unit: sectors
first-lba: 34
sector-size: 512

/dev/sad1 : start=          34, size=        2014, type=21686148-6449-6E6F-744E-656564454649
/dev/sad2 : start=        2048, size=     1048576, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B
/dev/sda3 : start=     1050624, size=   -, type=6A898CC3-1DD2-11B2-99A6-080020736631

Use this file to create the partitions on the new drive:

# sfdisk /dev/sdb < pve-boot-partitions.txt

Check that the partitions are created correctly:

# lsblk /dev/sda /dev/sdb

Learn more about the partitions in Proxmox in the Proxmox documentation.

Step 2: Copy the partitions

Now that the partitions are created, we can copy the data.

First up is the BIOS boot partition.

# dd if=/dev/sda1 of=/dev/sdb1

Next, the EFI system partition.

Create the filesystem on the new drive, by simply copying over from the old drive

# dd if=/dev/sda2 of=/dev/sdb2

Check if you are using EFI or Grub bootloader

# proxmox-boot-tool status

If you are using EFI, you can skip the grub at the end of the command.

proxmox-boot-tool init /dev/sdb2 [grub]

Step 3: Create a mirrored ZFS pool

The idea is to create a mirrored ZFS pool with the two drives and detach the old drive.

# zpool replace -f rpool /dev/sda3 /dev/sdb3

This will start the resilvering process, which will copy the data from the old drive to the new drive. You can check the status of the resilvering process with:

# zpool status rpool

Once the resilvering is complete, you can boot from the new drive.

Conculsion

Always use ZFS during installation of Proxmox. By using ZFS, you can easily replace drives without data loss.

References

• Proxmox VE—ZFS on Linux
• ZFS—zpool replace

Typically, you are going to submit code and review code.

As a reviewer

Categorised comments

Categorize the comments based on their importance. This will help the author to prioritize the comments and take action accordingly.

Question

A Question might not be actionable in terms of updating the code.

But it is important to get the author's attention, so that you as a reviewer and peers who might look at the pull request understand better.

Suggestion

A Suggestion also might not be actionable in terms of updating the code. It could be something that is stylistic or something that is not a blocker for the pull request to be merged.

However, it is important to overcommunicate why a suggesion is made, along with an example. Eg, this if condition could be more readable if it is extracted to a variable/function.

Action required

These comments require action before the pull request can be merged. It could be a bug, a security issue, or a performance issue which might impact user experience or other key metrics of the application.

Again, explaining how the issue might impact users or the application is important.

Blocking approval

These comments are blockers for the pull request to be merged. For example, then the pull request does not have tests, or the code written is a duplicate and the author can reuse an existing function/library to do the same.

Another case, is when the code becomes irrelevant due to changing requirements.

Require offline communication

This is a special case where the reviewer, the author, and maybe more team members need to have a discussion offline to come to a consensus.

This could be a design decision, or a decision to refactor a large part of the codebase. In other cases, it could be a complex feature that needs to be broken down into smaller pieces or to discuss some missing pieces only discovered after writing the code.

Ensure to approve or reject the pull request

Do not leave the pull request hanging. As long as there is no blocking approval or require offline communication comment, the pull request should be approved. We trust that the author to make the changes required, and reply to relevant comments before they merge the code.

If we fail to approve or reject the pull request, the author will be slowed down and in turn the team's performance is impacted.

As an author

Work in progress

We will use openssl to create a keystore in PKCS12 format from certificate and key files in pem format, and use this to configure WebClient to use mTLS.

Setup

Obtain the client certificate and key, this will usually be in pem format.

Now we can convert this into a PKCS12 keystore. This keystore can hold multiple certificates and keys, identified by an alias/friedly name. The whole keystore is password protected, and each key entry may also be optionally password protected.

openssl pkcs12 -in client_cert.pem -inkey client_key.pem -export -out client.p12 -name my-client

Configure WebClient

val keyStore = KeyStore.getInstance("PKCS12")

// read a file as inputStream from path /etc/ssl/certs/client.p12
File("/etc/ssl/certs/client.p12").inputStream().use { inputStream ->
    keyStore.load(inputStream, "password".toCharArray())
}

// read the client certificate and key from the keystore using `my-client` alias
val clientCrt: X509Certificate = keyStore.getCertificate("my-client") as X509Certificate
val keyStoreEntry = keyStore.getEntry(
    "my-client",
    KeyStore.PasswordProtection("password".toCharArray())
)
val clientKey = (keyStoreEntry as KeyStore.PrivateKeyEntry).privateKey


val httpClient: HttpClient =
    HttpClient.create()
        .wiretap("reactor.netty.http.client.HttpClient", LogLevel.DEBUG, AdvancedByteBufFormat.TEXTUAL)
        .secure { builder ->
            // Use this even though it is deprecated so that the TLS version can be negotiated and changed during runtime
            builder.sslContext(
                SslContextBuilder.forClient().clientAuth(ClientAuth.REQUIRE).keyManager(clientKey, clientCrt)
            )
        }
val httpConnector: ClientHttpConnector = ReactorClientHttpConnector(httpClient)


return WebClient.builder()
    .clientConnector(httpConnector)
    .baseUrl("https://mtls-secured-server.com")
    .build()