feat(paperless): provision RBD volume to move data off CephFS (phase 1)

paperless-data is declared ReadWriteMany on ceph-filesystem, but only ONE pod
mounts it -- the paperless Deployment itself. Its contents are all single-writer
state: the 50 MB Whoosh index, classification_model.pickle,
celerybeat-schedule.db, log/, migration_lock. 75 files, nothing shared. Only
`consume` and `media` genuinely need RWX, because the ftp-scanner writes to those
from another node.

It is on CephFS by default, not by requirement, and every access goes through the
MDS. Measured 2026-08-20 with ZERO CI pipelines running:

  paperless-data client   request_load_avg 22   (all other sessions 0-1)
  MDS over 180s           4,052 client requests = 22.5/s
                          2,718 journal events  = 15/s
                              2 dir commits     <- not file writes

Cap and inode churn from repeated access, not data being written. That accounts
for most of the constant 7-8 GB/day on cephfs-metadata, and since that pool is
size 3, roughly 21-24 GB/day reaches the drives.

cephfs-metadata deliberately STAYS at size 3. Losing it destroys the whole
filesystem rather than the objects on one OSD, and the 275 GiB in cephfs-data0
rides on it. Cutting writes at source is the fix; cutting replication is not.

PHASE 1 ONLY, and inert: the PVC is created empty and nothing mounts it. The data
copy and the deployment repoint are separate steps, documented in
MIGRATION-data-to-block.md along with why the copy needs paperless stopped (the
Whoosh index and celerybeat schedule are both live-written, so copying under an
active writer gives a torn read).

Honest sizing: ~21-24 GB/day off the drives against ~432 GB/day cluster-wide, so
about 5%. This is a correctness fix -- a single-writer volume does not belong on
a shared filesystem -- more than a large win. It also removes 22 req/sec of MDS
load, which benefits Woodpecker CI workspaces sharing the same MDS.

Refs #434

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/apps/authentik/cluster.yaml

@@ -43,27 +43,6 @@ spec:
     parameters:
       shared_buffers: "256MB"
       max_connections: "400"
-      # Write-reduction tuning. These databases are largely idle, and the
-      # Postgres defaults favour throughput over storage that wears out.
-      #
-      # checkpoint_timeout is the lever. The 5min default pushes every dirty
-      # page to disk 288 times a day, so a small database whose same few pages
-      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
-      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
-      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
-      # day. At 15min those pages coalesce and are written a third as often.
-      #
-      # max_wal_size must rise with it, or checkpoints trigger on size instead
-      # and the timeout change does nothing. It also bounds crash recovery to
-      # at most 2GB of WAL replay.
-      #
-      # NOT changed: synchronous_commit stays at the default `on`. It is off for
-      # ha-pg because a state recorder is telemetry; these hold credentials,
-      # documents and job state, where losing the last few hundred ms of commits
-      # on an unclean shutdown is not an acceptable trade.
-      checkpoint_timeout: "15min"
-      max_wal_size: "2GB"
-      wal_compression: "on"
     # Async replication makes CNPG switchover slow because the standby has
     # to chase missing WAL via barman-cloud and replay queued records
     # before it can promote. With one synchronous standby, the standby is

kubernetes/apps/grafana-db/postgres.yaml

@@ -29,27 +29,6 @@ spec:
     parameters:
       shared_buffers: "128MB"
       max_connections: "50"
-      # Write-reduction tuning. These databases are largely idle, and the
-      # Postgres defaults favour throughput over storage that wears out.
-      #
-      # checkpoint_timeout is the lever. The 5min default pushes every dirty
-      # page to disk 288 times a day, so a small database whose same few pages
-      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
-      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
-      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
-      # day. At 15min those pages coalesce and are written a third as often.
-      #
-      # max_wal_size must rise with it, or checkpoints trigger on size instead
-      # and the timeout change does nothing. It also bounds crash recovery to
-      # at most 2GB of WAL replay.
-      #
-      # NOT changed: synchronous_commit stays at the default `on`. It is off for
-      # ha-pg because a state recorder is telemetry; these hold credentials,
-      # documents and job state, where losing the last few hundred ms of commits
-      # on an unclean shutdown is not an acceptable trade.
-      checkpoint_timeout: "15min"
-      max_wal_size: "2GB"
-      wal_compression: "on"
   resources:
     requests:
       cpu: 100m

kubernetes/apps/home-assistant/ha-pg-cluster.yaml

@@ -55,21 +55,7 @@ spec:
       owner: homeassistant
   postgresql:
     parameters:
-      # 512MB, not the 256MB this shipped with. Measured after 15h on Postgres:
-      # buffers_backend was 629,704 of 939,025 total page writes — 67% written
-      # by backend processes rather than the checkpointer or bgwriter. That
-      # ratio means the working set does not fit in shared_buffers, so backends
-      # evict dirty pages themselves and the same pages are written out
-      # repeatedly instead of coalescing in memory.
-      #
-      # The recorder touches a wide set of pages continuously (~30 state
-      # changes/sec across states, state_attributes and five indexes), which is
-      # exactly the workload that punishes a small buffer pool.
-      #
-      # Memory headroom checked before raising: the instance was using 383Mi of
-      # a 1Gi limit with 256MB of that being shared_buffers, so the limit goes
-      # to 1.5Gi to keep the same margin.
-      shared_buffers: "512MB"
+      shared_buffers: "256MB"
       max_connections: "50"
       # The rest of this block is tuned to MINIMISE WRITE VOLUME, which is the
       # entire reason for the migration. Defaults would give a correct database
@@ -101,4 +87,4 @@ spec:
       cpu: 200m
       memory: 512Mi
     limits:
-      memory: 1536Mi
+      memory: 1Gi

kubernetes/apps/immich/cluster.yaml

@@ -43,25 +43,6 @@ spec:
       shared_buffers: "512MB"
       max_wal_size: "2GB"
       wal_compression: "on"
-      # Write-reduction tuning. These databases are largely idle, and the
-      # Postgres defaults favour throughput over storage that wears out.
-      #
-      # checkpoint_timeout is the lever. The 5min default pushes every dirty
-      # page to disk 288 times a day, so a small database whose same few pages
-      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
-      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
-      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
-      # day. At 15min those pages coalesce and are written a third as often.
-      #
-      # max_wal_size must rise with it, or checkpoints trigger on size instead
-      # and the timeout change does nothing. It also bounds crash recovery to
-      # at most 2GB of WAL replay.
-      #
-      # NOT changed: synchronous_commit stays at the default `on`. It is off for
-      # ha-pg because a state recorder is telemetry; these hold credentials,
-      # documents and job state, where losing the last few hundred ms of commits
-      # on an unclean shutdown is not an acceptable trade.
-      checkpoint_timeout: "15min"
   resources:
     requests:
       cpu: 500m

kubernetes/apps/paperless/MIGRATION-data-to-block.md

@@ -0,0 +1,97 @@
+# paperless-data: CephFS → RBD
+
+## Why
+
+`paperless-data` is `ReadWriteMany` on `ceph-filesystem`, but **only the paperless
+Deployment mounts it**. Verified 2026-08-20:
+
+```
+$ pods mounting paperless-data
+paperless-68cb8b8744-7hjb8      <- one, and only one
+
+paperless-data     ReadWriteMany  ceph-filesystem
+paperless-media    ReadWriteMany  ceph-filesystem   <- genuinely shared (ftp-scanner)
+paperless-consume  ReadWriteMany  ceph-filesystem   <- genuinely shared (ftp-scanner)
+```
+
+Its contents are single-writer state — the 50 MB Whoosh index,
+`classification_model.pickle`, `celerybeat-schedule.db`, `log/`,
+`migration_lock`, `protonmail.pem`. 75 files. Nothing else reads or writes them.
+
+It is on CephFS by default, not by requirement.
+
+## What it costs
+
+Every access goes through the MDS. Measured with **zero CI pipelines running**:
+
+```
+paperless-data client   request_load_avg 22   (all other sessions 0-1)
+MDS over 180s           4,052 client requests  = 22.5/s
+                        2,718 journal events   = 15/s
+                            2 dir commits      <- almost none: not file writes
+```
+
+So it is cap/inode churn from repeated access, not data being written. That is
+most of the constant **7-8 GB/day** written to `cephfs-metadata`, and since that
+pool is `size: 3`, roughly **21-24 GB/day** reaches the drives.
+
+`cephfs-metadata` deliberately stays at size 3. Losing it destroys the entire
+filesystem rather than the objects on one OSD, and the 275 GiB in `cephfs-data0`
+depends on it. Cutting the writes at source is correct; cutting the replication
+is not.
+
+## Sequence
+
+Staged deliberately: the PVC lands first and empty, so merging it changes
+nothing. The copy and the repoint are separate.
+
+### Phase 1 — provision (this PR)
+
+`paperless-data-block` (RWO, `ceph-block`, 10Gi) is created and mounted by
+nothing. Inert.
+
+Verify: `kubectl -n paperless get pvc paperless-data-block` → Bound.
+
+### Phase 2 — copy
+
+The Whoosh index must come across intact. Paperless *can* rebuild it
+(`document_index reindex`), but that reprocesses every document, so copying is
+both faster and less disruptive.
+
+1. Scale paperless to 0. The source must be quiescent — `celerybeat-schedule.db`
+   and the Whoosh index are both live-written, and copying them under an active
+   writer gives a torn read.
+2. Run a one-off pod that mounts BOTH PVCs and copies:
+   ```
+   cp -a /src/. /dst/
+   ```
+   The RWO target pins the pod to one node; the RWX source will follow.
+3. Verify file count and index size match:
+   ```
+   find /src -type f | wc -l   ==   find /dst -type f | wc -l
+   du -sh /src/index           ==   du -sh /dst/index
+   ```
+
+### Phase 3 — repoint
+
+Change the Deployment's `data` volume to `claimName: paperless-data-block`,
+merge, let it roll.
+
+Verify afterwards:
+- paperless starts, search returns results (proves the Whoosh index survived)
+- `ceph tell mds.cephfs-b session ls` no longer shows a busy `paperless-data`
+  client
+- `rate(ceph_pool_wr_bytes{pool_id="3"})` drops from its 7-8 GB/day baseline
+
+**Do not delete `paperless-data` until search has been confirmed working.** It is
+the rollback, and it is `prune: disabled` so Flux will not remove it.
+
+## Expected effect
+
+~7-8 GB/day off `cephfs-metadata`, so ~21-24 GB/day off the drives once
+replication is counted. Against ~432 GB/day cluster-wide that is roughly 5% —
+worth doing, but it is a correctness fix (a single-writer volume does not belong
+on a shared filesystem) more than a large win.
+
+It also removes 22 req/sec of MDS load, which benefits the workloads that
+genuinely need CephFS — Woodpecker CI workspaces share the same MDS.

kubernetes/apps/paperless/cluster.yaml

@@ -34,27 +34,6 @@ spec:
     parameters:
       shared_buffers: "256MB"
       max_connections: "200"
-      # Write-reduction tuning. These databases are largely idle, and the
-      # Postgres defaults favour throughput over storage that wears out.
-      #
-      # checkpoint_timeout is the lever. The 5min default pushes every dirty
-      # page to disk 288 times a day, so a small database whose same few pages
-      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
-      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
-      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
-      # day. At 15min those pages coalesce and are written a third as often.
-      #
-      # max_wal_size must rise with it, or checkpoints trigger on size instead
-      # and the timeout change does nothing. It also bounds crash recovery to
-      # at most 2GB of WAL replay.
-      #
-      # NOT changed: synchronous_commit stays at the default `on`. It is off for
-      # ha-pg because a state recorder is telemetry; these hold credentials,
-      # documents and job state, where losing the last few hundred ms of commits
-      # on an unclean shutdown is not an acceptable trade.
-      checkpoint_timeout: "15min"
-      max_wal_size: "2GB"
-      wal_compression: "on"
     # See ak-pg cluster.yaml for the rationale on synchronous replication.
     synchronous:
       method: any

kubernetes/apps/paperless/pvcs.yaml

@@ -18,6 +18,45 @@ spec:
     requests:
       storage: 10Gi
 ---
+# Replacement for paperless-data, on RBD instead of CephFS.
+#
+# WHY: paperless-data is declared ReadWriteMany on ceph-filesystem, but only ONE
+# pod ever mounts it — the paperless Deployment itself. Its contents are all
+# single-writer state: the 50 MB Whoosh index, classification_model.pickle,
+# celerybeat-schedule.db, logs, migration_lock. Nothing shares it. Only `consume`
+# and `media` genuinely need RWX, because the FTP scanner writes to those from a
+# different node.
+#
+# COST OF LEAVING IT: every access goes through the MDS. Measured 2026-08-20 with
+# zero CI running, this volume's client was doing 22 MDS requests/sec sustained,
+# and the MDS logged ~15 journal events/sec against only 2 directory commits per
+# 180s — i.e. cap and inode churn from repeated access, not file writes. That is
+# most of the constant 7-8 GB/day written to cephfs-metadata, which is a size 3
+# pool, so ~21-24 GB/day actually hits the drives.
+#
+# cephfs-metadata deliberately STAYS at size 3 (see storageclass.yaml): losing
+# the metadata pool destroys the whole filesystem, not just the objects on one
+# OSD, so the 275 GiB in cephfs-data0 rides on it. Reducing the writes at source
+# is the right fix; reducing the replication is not.
+#
+# MIGRATION: this PVC is created EMPTY and nothing mounts it yet. The data copy
+# and the deployment repoint are a separate, staged step — see
+# MIGRATION-data-to-block.md. Merging this alone is inert.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: paperless-data-block
+  namespace: paperless
+  annotations:
+    kustomize.toolkit.fluxcd.io/prune: disabled
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: ceph-block
+  resources:
+    requests:
+      storage: 10Gi
+---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:

kubernetes/apps/trmnl/cluster.yaml

@@ -33,26 +33,6 @@ spec:
       shared_buffers: "128MB"
       max_connections: "50"
       wal_compression: "on"
-      # Write-reduction tuning. These databases are largely idle, and the
-      # Postgres defaults favour throughput over storage that wears out.
-      #
-      # checkpoint_timeout is the lever. The 5min default pushes every dirty
-      # page to disk 288 times a day, so a small database whose same few pages
-      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
-      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
-      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
-      # day. At 15min those pages coalesce and are written a third as often.
-      #
-      # max_wal_size must rise with it, or checkpoints trigger on size instead
-      # and the timeout change does nothing. It also bounds crash recovery to
-      # at most 2GB of WAL replay.
-      #
-      # NOT changed: synchronous_commit stays at the default `on`. It is off for
-      # ha-pg because a state recorder is telemetry; these hold credentials,
-      # documents and job state, where losing the last few hundred ms of commits
-      # on an unclean shutdown is not an acceptable trade.
-      checkpoint_timeout: "15min"
-      max_wal_size: "2GB"
   resources:
     requests:
       cpu: 100m

kubernetes/apps/woodpecker/cluster.yaml

@@ -56,27 +56,6 @@ spec:
     parameters:
       shared_buffers: "128MB"
       max_connections: "50"
-      # Write-reduction tuning. These databases are largely idle, and the
-      # Postgres defaults favour throughput over storage that wears out.
-      #
-      # checkpoint_timeout is the lever. The 5min default pushes every dirty
-      # page to disk 288 times a day, so a small database whose same few pages
-      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
-      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
-      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
-      # day. At 15min those pages coalesce and are written a third as often.
-      #
-      # max_wal_size must rise with it, or checkpoints trigger on size instead
-      # and the timeout change does nothing. It also bounds crash recovery to
-      # at most 2GB of WAL replay.
-      #
-      # NOT changed: synchronous_commit stays at the default `on`. It is off for
-      # ha-pg because a state recorder is telemetry; these hold credentials,
-      # documents and job state, where losing the last few hundred ms of commits
-      # on an unclean shutdown is not an acceptable trade.
-      checkpoint_timeout: "15min"
-      max_wal_size: "2GB"
-      wal_compression: "on"
   resources:
     requests:
       cpu: 100m