perf(grafana-db): raise archive_timeout to 30min on gf-pg #539

2026-08-20T12:21:25Z

claude-bot commented

2026-08-20 12:21:25 +00:00

The finding

This is the answer to the anomaly that has been open since the start of the endurance work: gf-pg appeared to write gigabytes per day for a 21 MB database, and neither WAL volume nor data changes could account for it.

A WAL segment is a fixed 16 MiB file. archive_timeout forces a segment switch on a timer so the archive never trails reality by more than that interval — and the switched-out segment is written and archived at its full 16 MiB regardless of how few bytes of real WAL it contains.

CNPG defaults this to 5min, permitting up to 288 segments/day = 4.6 GB/day, per cluster.

Measured, not inferred

	gf-pg
database size	21 MB
actual WAL generated	4.5 MB/day
RBD volume writes	211 MB/hour
predicted from 12 switches x 16 MiB	192 MB/hour

Prediction and measurement agree within 10%. Essentially every byte written to this volume is padding — roughly 1100x amplification.

Why gf-pg alone

Staged rollout. gf-pg is the lowest-risk cluster and gives the cleanest possible signal: its real WAL is small enough that any change in volume is unambiguously attributable to this parameter and nothing else.

The same 5min default affects all seven clusters and is worth an estimated ~24 GB/day of logical writes fleet-wide — but that only goes ahead after this one is verified. I will report the measured before/after on gf-pg rather than assuming it generalises.

The cost

RPO. On total loss of the cluster, up to 30min of Grafana state becomes unrecoverable instead of 5min. That state is dashboards, annotations and preferences, changing at ~4.5 MB/day, so the window is nearly always empty.

Clusters holding credentials or job state (ak-pg, wp-pg) get evaluated individually rather than inheriting this value by default.

Verification

archive_timeout is in CNPG GlobalDefaultSettings, not FixedConfigurationParameters — so it is user-overridable. Checked against pkg/postgres/configuration.go, not assumed. (archive_mode and archive_command are fixed; archive_timeout is not.)

Note on base

Based on reduce-pg-wal-fpi (#538), which touches the same parameter block and is not yet merged. Merge #538 first, or retarget this to master afterwards.

Context

Drive-endurance work on three QLC NVMe OSDs at 81-87% wear. Current: 270 GB/day at the drives, 82 GB/day of logical client writes into RBD (confirmed independently by kernel device counters and Ceph pool stats). Segment padding is the single largest identified component of that.

## The finding This is the answer to the anomaly that has been open since the start of the endurance work: **gf-pg appeared to write gigabytes per day for a 21 MB database**, and neither WAL volume nor data changes could account for it. A WAL segment is a fixed **16 MiB file**. `archive_timeout` forces a segment switch on a timer so the archive never trails reality by more than that interval — and the switched-out segment is written and archived **at its full 16 MiB regardless of how few bytes of real WAL it contains**. CNPG defaults this to `5min`, permitting up to 288 segments/day = 4.6 GB/day, per cluster. ## Measured, not inferred | | gf-pg | |---|---| | database size | 21 MB | | actual WAL generated | 4.5 MB/day | | RBD volume writes | **211 MB/hour** | | predicted from 12 switches x 16 MiB | 192 MB/hour | Prediction and measurement agree within 10%. Essentially every byte written to this volume is padding — roughly **1100x amplification**. ## Why gf-pg alone Staged rollout. gf-pg is the lowest-risk cluster and gives the cleanest possible signal: its real WAL is small enough that any change in volume is unambiguously attributable to this parameter and nothing else. The same 5min default affects **all seven clusters** and is worth an estimated **~24 GB/day of logical writes fleet-wide** — but that only goes ahead after this one is verified. I will report the measured before/after on gf-pg rather than assuming it generalises. ## The cost RPO. On total loss of the cluster, up to 30min of Grafana state becomes unrecoverable instead of 5min. That state is dashboards, annotations and preferences, changing at ~4.5 MB/day, so the window is nearly always empty. Clusters holding credentials or job state (ak-pg, wp-pg) get evaluated individually rather than inheriting this value by default. ## Verification `archive_timeout` is in CNPG `GlobalDefaultSettings`, **not** `FixedConfigurationParameters` — so it is user-overridable. Checked against `pkg/postgres/configuration.go`, not assumed. (`archive_mode` and `archive_command` *are* fixed; `archive_timeout` is not.) ## Note on base Based on `reduce-pg-wal-fpi` (#538), which touches the same parameter block and is not yet merged. Merge #538 first, or retarget this to master afterwards. ## Context Drive-endurance work on three QLC NVMe OSDs at 81-87% wear. Current: 270 GB/day at the drives, 82 GB/day of logical client writes into RBD (confirmed independently by kernel device counters and Ceph pool stats). Segment padding is the single largest identified component of that.

claude-bot added 1 commit 2026-08-20 12:21:25 +00:00

perf(grafana-db): raise archive_timeout to 30min on gf-pg

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

d15a4654a0

A WAL segment is a fixed 16 MiB file. archive_timeout forces a segment
switch on a timer so the archive never trails reality by more than that
interval, and the switched-out segment is written and archived at its full
16 MiB no matter how few bytes of real WAL it holds. CNPG defaults this to
5min, which permits up to 288 segments/day = 4.6 GB/day per cluster.

Measured on gf-pg: the database is 21 MB and generates 4.5 MB/day of actual
WAL, while its RBD volume takes 211 MB/hour -- against 192 MB/hour predicted
from 12 switches of 16 MiB. Essentially every byte written to this volume is
padding, roughly 1100x amplification.

This resolves an anomaly that has been open since the start of the endurance
work: gf-pg appeared to write gigabytes per day for a 21 MB database, and
neither WAL volume nor data changes could account for it. Segment padding
does, exactly.

Applied to gf-pg alone as a staged rollout. It is the lowest-risk cluster
and gives the cleanest signal -- its real WAL is small enough that any change
in volume is unambiguously attributable to this parameter. The same default
affects all seven clusters and is worth an estimated ~24 GB/day of logical
writes fleet-wide, but that only goes ahead once this one is verified.

The cost is RPO. On total loss of the cluster, up to 30min of Grafana state
becomes unrecoverable instead of 5min. That state is dashboards, annotations
and preferences, changing at ~4.5 MB/day, so the window is nearly always
empty. Clusters holding credentials or job state get evaluated separately
rather than inheriting this value by default.

archive_timeout is in CNPG's GlobalDefaultSettings, not its
FixedConfigurationParameters, so it is user-overridable -- verified against
pkg/postgres/configuration.go rather than assumed.

Context: drive-endurance work on three QLC NVMe OSDs at 81-87% wear.

review-bot reviewed

2026-08-20 12:25:08 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit d15a4654a0421bb490356e9e3b69fa2d0b430604.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR increases PostgreSQL's archive_timeout parameter from the CNPG default of 5min to 30min on the gf-pg (Grafana) cluster. The change is well-measured and justified: gf-pg generates ~4.5 MB/day of actual WAL but sees 211 MB/hour of volume writes (~5x daily amplification) due to forced segment switches. Reducing switches from 288/day to 48/day trades off RPO from 5min to 30min—acceptable for Grafana's slowly-changing state. The staged rollout on the lowest-risk cluster is a sound approach.

🔴 Blocking

None identified.

🟡 Suggestions

1. Clarify verification reference — postgres.yaml:65 (and the commit message) reference pkg/postgres/configuration.go to verify that archive_timeout is in "GlobalDefaultSettings, not FixedConfigurationParameters." This file doesn't exist in the current repository. If this is a reference to CNPG's external documentation/codebase, that's fine but should be clarified (e.g., "CNPG's CRD defaults" or a link). If it should exist in this repo, that's a problem.

2. Archive behavior on gf-pg — gf-pg has barman-cloud.cloudnative-pg.io explicitly disabled. The PR's explanatory text assumes WAL segments are "written and archived," but with archiving disabled, the practical effect of archive_timeout is only to control segment-switching frequency for crash recovery purposes, not archival cadence. This isn't a defect (the write reduction still holds), but the comment should either clarify this distinction or verify that archiving is actually enabled on this cluster.

🟢 Nits

The commit message and inline comments are thorough and well-structured; no changes needed.

Tests

No tests expected for a configuration parameter change. The staged rollout approach is the verification mechanism: gf-pg's small real-WAL generation means any volume change will be unambiguously attributable to this parameter. Before applying the same change to the other six clusters, monitoring should confirm that volume write reduction matches the ~6x prediction (from ~4.6 GB/day to ~770 MB/day on this cluster's footprint).

Recommendation

Looks reasonable — no blocking issues; suggestions above are optional. Verify that base branch PR #538 is merged or ready, since this PR targets reduce-pg-wal-fpi, not main.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `d15a4654a0421bb490356e9e3b69fa2d0b430604`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR increases PostgreSQL's `archive_timeout` parameter from the CNPG default of 5min to 30min on the gf-pg (Grafana) cluster. The change is well-measured and justified: gf-pg generates ~4.5 MB/day of actual WAL but sees 211 MB/hour of volume writes (~5x daily amplification) due to forced segment switches. Reducing switches from 288/day to 48/day trades off RPO from 5min to 30min—acceptable for Grafana's slowly-changing state. The staged rollout on the lowest-risk cluster is a sound approach. ## 🔴 Blocking None identified. ## 🟡 Suggestions **1. Clarify verification reference** — `postgres.yaml:65` (and the commit message) reference `pkg/postgres/configuration.go` to verify that `archive_timeout` is in "GlobalDefaultSettings, not FixedConfigurationParameters." This file doesn't exist in the current repository. If this is a reference to CNPG's external documentation/codebase, that's fine but should be clarified (e.g., "CNPG's CRD defaults" or a link). If it should exist in this repo, that's a problem. **2. Archive behavior on gf-pg** — gf-pg has `barman-cloud.cloudnative-pg.io` explicitly disabled. The PR's explanatory text assumes WAL segments are "written and archived," but with archiving disabled, the practical effect of `archive_timeout` is only to control segment-switching frequency for crash recovery purposes, not archival cadence. This isn't a defect (the write reduction still holds), but the comment should either clarify this distinction or verify that archiving is actually enabled on this cluster. ## 🟢 Nits - The commit message and inline comments are thorough and well-structured; no changes needed. ## Tests No tests expected for a configuration parameter change. The staged rollout approach is the verification mechanism: gf-pg's small real-WAL generation means any volume change will be unambiguously attributable to this parameter. Before applying the same change to the other six clusters, monitoring should confirm that volume write reduction matches the ~6x prediction (from ~4.6 GB/day to ~770 MB/day on this cluster's footprint). ## Recommendation Looks reasonable — no blocking issues; suggestions above are optional. Verify that base branch PR #538 is merged or ready, since this PR targets `reduce-pg-wal-fpi`, not `main`.

k.renner merged commit f5ad8988d9 into reduce-pg-wal-fpi

2026-08-20 12:30:31 +00:00

k.renner referenced this issue from a commit

2026-08-20 12:30:32 +00:00

Merge pull request 'perf(grafana-db): raise archive_timeout to 30min on gf-pg' (#539) from reduce-wal-segment-padding into reduce-pg-wal-fpi

claude-bot referenced this pull request

2026-08-20 12:38:16 +00:00

perf(grafana-db): raise archive_timeout to 30min on gf-pg (re-target of #539) #540

k.renner referenced this issue from a commit

2026-08-20 12:49:43 +00:00

Merge pull request 'perf(grafana-db): raise archive_timeout to 30min on gf-pg (re-target of #539)' (#540) from archive-timeout-gf-pg into master

Sign in to join this conversation.