perf(cnpg): roll archive_timeout out to the remaining six clusters #541

2026-08-20T13:36:54Z

claude-bot commented

2026-08-20 13:36:54 +00:00

Verified on gf-pg first

The staged rollout is complete on its first target and the result matched the prediction:

	before	after
volume writes	211 MB/hour	41 MB/hour
segments archived	~12/hour	~1.5/hour

81% reduction, against ~35 MB/hour predicted beforehand. Both signals moved together, which is what must happen if the volume was segment padding rather than data.

Mechanism

A WAL segment is a fixed 16 MiB file. archive_timeout forces a switch on a timer, and the switched-out segment is written and archived at full size no matter how little real WAL it holds. CNPG defaults to 5min = up to 288 segments/day per cluster.

What this rolls out

cluster	instances	value	note
wp-pg	1	30min
tr-pg	1	30min
im-pg	1	30min	photos live on the library PVC, not here
pprl-pg	2	30min	documents live on the media PVC, not here
ha-pg	1	30min	saves least: segments already 63% full of real WAL
ak-pg	2	15min	credentials — see below

ak-pg deliberately gets 15min, not 30min. It holds credentials, sessions and enrolment state, and the extra 15min buys only a further 0.3 GB/day. That does not justify doubling the worst-case loss window on that data. It does carry a synchronous standby, so realistic exposure is losing both instances — but the conservative value is nearly free here.

Expected effect

~18 GB/day of logical writes, which at size=2 with ~1.27x BlueStore overhead is ~45 GB/day at the drives — about 14% of the 328 GB/day the three OSD NVMes currently take. I will measure and report rather than assume, as with gf-pg.

Also: two corrections this work forced

This PR fixes two comments in ha-pg-cluster.yaml that my own measurements falsified.

wal_compression — I claimed these clusters "were not getting compression". That was wrong. "on" is pglz, which was already compressing. Switching to zstd moved the WAL rate not at all: 4911 and 4976 MB/day before, 5002 MB/day after, by pg_current_wal_lsn advance over 20min windows.

Full-page images genuinely are ~62% of WAL here — 61.9% measured as a steady-state delta over one checkpoint cycle, against 62.4% lifetime. So the migration did not skew that statistic, which was my first explanation and it was also wrong. There was simply no headroom between pglz and zstd on this data. zstd stays because it is not worse and costs nothing, but it is not a lever.

checkpoint_timeout — 15min to 30min likewise did not move WAL. Probable reason: a recorder is append-mostly, so a page is dirtied while it fills and then not touched again, paying its full-page image once regardless of checkpoint interval. Interval only helps pages re-dirtied across many cycles.

Kept, but now flagged in-file as unjustified pending a data-file measurement — this volume takes ~11 GB/day against 4.9 GB/day of WAL, so most of its traffic is outside what a WAL measurement can see. It should not be treated as validated until that is measured.

Context

Drive-endurance work on three QLC NVMe OSDs at 81-87% wear. Verified separately today: the earlier Ceph fixes (#519, #520) held at 24h — cluster writes 1019 to 507.5 GB/day, mon writes 80 to 8.03 GB/day/mon, mon store stable at 145 MB.

## Verified on gf-pg first The staged rollout is complete on its first target and the result matched the prediction: | | before | after | |---|---|---| | volume writes | 211 MB/hour | **41 MB/hour** | | segments archived | ~12/hour | ~1.5/hour | **81% reduction**, against ~35 MB/hour predicted beforehand. Both signals moved together, which is what must happen if the volume was segment padding rather than data. ## Mechanism A WAL segment is a fixed **16 MiB file**. `archive_timeout` forces a switch on a timer, and the switched-out segment is written and archived at full size no matter how little real WAL it holds. CNPG defaults to `5min` = up to 288 segments/day per cluster. ## What this rolls out | cluster | instances | value | note | |---|---|---|---| | wp-pg | 1 | 30min | | | tr-pg | 1 | 30min | | | im-pg | 1 | 30min | photos live on the library PVC, not here | | pprl-pg | 2 | 30min | documents live on the media PVC, not here | | ha-pg | 1 | 30min | saves least: segments already 63% full of real WAL | | **ak-pg** | 2 | **15min** | **credentials — see below** | **ak-pg deliberately gets 15min, not 30min.** It holds credentials, sessions and enrolment state, and the extra 15min buys only a further 0.3 GB/day. That does not justify doubling the worst-case loss window on that data. It does carry a synchronous standby, so realistic exposure is losing both instances — but the conservative value is nearly free here. ## Expected effect ~18 GB/day of logical writes, which at `size=2` with ~1.27x BlueStore overhead is **~45 GB/day at the drives** — about 14% of the 328 GB/day the three OSD NVMes currently take. I will measure and report rather than assume, as with gf-pg. ## Also: two corrections this work forced This PR fixes two comments in `ha-pg-cluster.yaml` that my own measurements falsified. **`wal_compression`** — I claimed these clusters "were not getting compression". That was wrong. `"on"` is pglz, which was already compressing. Switching to zstd moved the WAL rate **not at all**: 4911 and 4976 MB/day before, 5002 MB/day after, by `pg_current_wal_lsn` advance over 20min windows. Full-page images genuinely are ~62% of WAL here — 61.9% measured as a steady-state delta over one checkpoint cycle, against 62.4% lifetime. So the migration did **not** skew that statistic, which was my first explanation and it was also wrong. There was simply no headroom between pglz and zstd on this data. zstd stays because it is not worse and costs nothing, but it is not a lever. **`checkpoint_timeout`** — 15min to 30min likewise did not move WAL. Probable reason: a recorder is append-mostly, so a page is dirtied while it fills and then not touched again, paying its full-page image once regardless of checkpoint interval. Interval only helps pages re-dirtied across many cycles. Kept, but now flagged in-file as **unjustified pending a data-file measurement** — this volume takes ~11 GB/day against 4.9 GB/day of WAL, so most of its traffic is outside what a WAL measurement can see. It should not be treated as validated until that is measured. ## Context Drive-endurance work on three QLC NVMe OSDs at 81-87% wear. Verified separately today: the earlier Ceph fixes (#519, #520) held at 24h — cluster writes 1019 to 507.5 GB/day, mon writes 80 to 8.03 GB/day/mon, mon store stable at 145 MB.

claude-bot added 1 commit 2026-08-20 13:36:54 +00:00

perf(cnpg): roll archive_timeout out to the remaining six clusters

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

119b2c77ef

A WAL segment is a fixed 16 MiB file. archive_timeout forces a switch on a
timer, and the switched-out segment is written and archived at its full size
no matter how little real WAL it holds. CNPG defaults to 5min, permitting
288 segments/day per cluster of largely padding.

gf-pg went first as a staged rollout and is now verified: 211 -> 41 MB/hour,
an 81% cut, with archived segments falling from ~12/hour to ~1.5/hour in
step. Two independent signals moving together is what confirms the volume
was padding rather than data, and the measured result matched the ~35
MB/hour predicted beforehand.

Fleet-wide this is worth roughly 18 GB/day of logical writes, which at
size=2 with ~1.27x BlueStore overhead is ~45 GB/day at the drives -- about
14% of the 328 GB/day the three OSD NVMes currently take.

ak-pg gets 15min rather than 30min. It holds credentials, sessions and
enrolment state, and the extra 15min would save only a further 0.3 GB/day,
which does not justify doubling the worst-case loss window on that data.

ha-pg saves least in relative terms: its segments are already 63% full of
real WAL (4.9 GB/day), so the timer is not what drives most of its switches.

Also corrects two comments in ha-pg-cluster.yaml that this work falsified.

wal_compression: the claim that these clusters "were not getting compression"
was wrong -- "on" is pglz, which was already compressing. Switching to zstd
moved the WAL rate not at all (4911/4976 -> 5002 MB/day, by LSN advance over
20min windows). Full-page images genuinely are ~62% of WAL here (61.9%
measured as a steady-state delta, against 62.4% lifetime -- so the migration
did not skew it, which was my first guess and it was wrong). There was simply
no headroom between pglz and zstd on this data. zstd is kept because it is
not worse and costs nothing, but it is not a lever.

checkpoint_timeout: 15min -> 30min likewise did not move WAL. Probable reason
is that a recorder is append-mostly -- a page is dirtied while it fills and
then not touched again, so it pays its full-page image once whatever the
checkpoint interval. Kept, but flagged as unjustified pending a data-file
measurement: this volume takes ~11 GB/day against 4.9 GB/day of WAL, so most
of its traffic is outside what a WAL measurement can see.

Context: drive-endurance work on three QLC NVMe OSDs at 81-87% wear.

k.renner merged commit 96387108ff into master

2026-08-20 13:39:01 +00:00

k.renner referenced this issue from a commit

2026-08-20 13:39:02 +00:00

Merge pull request 'perf(cnpg): roll archive_timeout out to the remaining six clusters' (#541) from rollout-archive-timeout into master

review-bot reviewed

2026-08-20 13:40:26 +00:00

review-bot left a comment

🤖 Automated review — model claude-sonnet-5 (complexity: normal), commit 119b2c77efcb150b4f95211e3874c838039eb927.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR sets archive_timeout on the six remaining CNPG clusters (ak-pg, ha-pg, im-pg, pprl-pg, tr-pg, wp-pg), following the staged verification already done on gf-pg, and rewrites two comment blocks in ha-pg-cluster.yaml to correct claims about wal_compression and checkpoint_timeout that the author's own measurements disproved. It is a comment-heavy, low-risk change: one new string parameter per cluster plus documentation. I found no blocking issues.

🟡 Suggestions

kubernetes/apps/authentik/cluster.yaml:12 (comment) — The padding estimate for ak-pg reads "up to 288 segments/day = 3.2 GB/day here", while the identical 288 segments/day cap is given as 4.6 GB/day in immich, trmnl, woodpecker, and ha-pg, and 4.1 GB/day in paperless. If these are all meant to be the same theoretical ceiling (288 × 16 MiB ≈ 4.5–4.8 GB), the 3.2/4.1 figures look low; if instead they're per-cluster estimates netting out real WAL already occupying part of each segment (which would explain the variance), it's worth a one-line note saying so, since as written it reads like a plain unit conversion that doesn't match itself across files. This is a comment-only inconsistency with no functional effect.
kubernetes/apps/woodpecker/cluster.yaml:96 — This is the CI system's own database (wp-pg), and it now carries a 30-minute RPO like the other "low-value" clusters. That may well be an intentional/acceptable tradeoff (build history is generally not update-critical), but it's the one target in this batch where "up to 30min of WAL is unrecoverable" affects the pipeline's own state rather than an end-user app; worth a conscious nod if not already considered.

Tests

No automated tests apply to this kind of change; the manifests are plain CNPG Cluster CRs with string parameters. .woodpecker/ci-manifests-test.yml presumably runs schema/kustomize validation over the repo, which is the right gate here (archive_timeout accepts standard Postgres time-unit strings like "15min"/"30min", matching the syntax already live on gf-pg in kubernetes/apps/grafana-db/postgres.yaml:77). I verified by reading the checkout that:

All six postgresql.parameters blocks are correctly indented and the new keys sit inside the existing parameters: map, consistent with the already-merged gf-pg pattern.
instances: counts in each file match the PR's table (ak-pg: 2, pprl-pg: 2, ha-pg/im-pg/tr-pg/wp-pg: 1), so the RPO framing in each comment ("single-instance cluster" vs "carries a synchronous standby") is accurate to the actual manifest.
ak-pg is the only cluster given 15min instead of 30min, matching the PR's stated rationale (credentials/session data).

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `claude-sonnet-5` (complexity: normal), commit `119b2c77efcb150b4f95211e3874c838039eb927`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR sets `archive_timeout` on the six remaining CNPG clusters (ak-pg, ha-pg, im-pg, pprl-pg, tr-pg, wp-pg), following the staged verification already done on gf-pg, and rewrites two comment blocks in `ha-pg-cluster.yaml` to correct claims about `wal_compression` and `checkpoint_timeout` that the author's own measurements disproved. It is a comment-heavy, low-risk change: one new string parameter per cluster plus documentation. I found no blocking issues. ## 🟡 Suggestions - `kubernetes/apps/authentik/cluster.yaml:12` (comment) — The padding estimate for ak-pg reads "up to 288 segments/day = 3.2 GB/day here", while the identical `288 segments/day` cap is given as 4.6 GB/day in `immich`, `trmnl`, `woodpecker`, and `ha-pg`, and 4.1 GB/day in `paperless`. If these are all meant to be the same theoretical ceiling (288 × 16 MiB ≈ 4.5–4.8 GB), the 3.2/4.1 figures look low; if instead they're per-cluster estimates netting out real WAL already occupying part of each segment (which would explain the variance), it's worth a one-line note saying so, since as written it reads like a plain unit conversion that doesn't match itself across files. This is a comment-only inconsistency with no functional effect. - `kubernetes/apps/woodpecker/cluster.yaml:96` — This is the CI system's own database (wp-pg), and it now carries a 30-minute RPO like the other "low-value" clusters. That may well be an intentional/acceptable tradeoff (build history is generally not update-critical), but it's the one target in this batch where "up to 30min of WAL is unrecoverable" affects the pipeline's own state rather than an end-user app; worth a conscious nod if not already considered. ## Tests No automated tests apply to this kind of change; the manifests are plain CNPG `Cluster` CRs with string parameters. `.woodpecker/ci-manifests-test.yml` presumably runs schema/kustomize validation over the repo, which is the right gate here (`archive_timeout` accepts standard Postgres time-unit strings like `"15min"`/`"30min"`, matching the syntax already live on `gf-pg` in `kubernetes/apps/grafana-db/postgres.yaml:77`). I verified by reading the checkout that: - All six `postgresql.parameters` blocks are correctly indented and the new keys sit inside the existing `parameters:` map, consistent with the already-merged `gf-pg` pattern. - `instances:` counts in each file match the PR's table (ak-pg: 2, pprl-pg: 2, ha-pg/im-pg/tr-pg/wp-pg: 1), so the RPO framing in each comment ("single-instance cluster" vs "carries a synchronous standby") is accurate to the actual manifest. - `ak-pg` is the only cluster given `15min` instead of `30min`, matching the PR's stated rationale (credentials/session data). ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#541