PostgreSQL WAL archive failures caused by incomplete copies and full filesystems

Problem:

A production PostgreSQL 15 instance experienced a Severity‑1 outage after the filesystem containing WAL files reached 100% usage. The customer observed failed Commvault log-backup jobs and that WAL segments were not being archived: SHOW archive_command returned an external script invocation (/var/lib/pgsql/15/scripts/copy_wal.sh %p %f). The shipped copy script used a “test ! -f … && cp” pattern. Both the database data-area (pg_wal) and the archive staging directory became full; manual cleanup temporarily freed space but backups continued to fail for one or two attempts afterwards.

Process:

Step 1: Confirm configuration and symptom timeline

Observed archive_command and inspected the helper script content reported by the customer. Verified that archive_command invoked a shell script with positional parameters (%p %f) and that the script used a conditional test to avoid overwriting an existing destination file. This established the exact mechanism PostgreSQL used to hand off WAL segments to the archive staging area.

Step 2: Examine PostgreSQL archive status and filesystem layout

Reviewed pg_stat_archiver and the pg_wal/archive_status directory to measure backlog and failure counts. Ran df -h and readlink -f on the data and archive paths to determine whether the archive staging directory shared the same filesystem as pg_wal. Found a backlog of .ready flags and a nonzero failed_count, and confirmed that accumulation had consumed space on the same filesystem in the deployed layout—explaining how archiving pressure could cause the data filesystem to fill.

Step 3: Inspect logs and the copy script behaviour under outage

Checked the archive helper’s log file and PostgreSQL archive-related log entries for “archive command failed” occurrences. Found repeated copy failures concurrent with out‑of‑space conditions and evidence of partially written destination files in the archive directory. These partial files caused the script’s “test ! -f” guard to treat the segment as already present, so retries never re-copied a now-correct source file.

Step 4: Correlate with backup appliance behaviour

Reviewed the backup sweep behaviour (as reported) and noted that the backup job had failed to remove or sweep archived files while the copy operations were failing, allowing the archive staging to accumulate excess data. The customer had observed that manually moving older WALs freed space and allowed temporary recovery, and that a later change to the archive format made Commvault succeed—confirming the problem was the archive publishing behavior rather than a PostgreSQL path-handling bug.

Step 5: Evaluate proposed changes and define safe requirements

Assessed two customer-proposed fixes: hard‑coding the data directory into archive_command and replacing the script with a plain cp. Both were rejected: hard-coded paths reduce future portability and the plain cp risks silently overwriting existing archive files (potentially breaking PITR). Identified the key requirements: avoid leaving partial files under the final name, refuse to overwrite files with differing content, keep idempotent success for identical files, and maintain logging for auditing.

Step 6: Implement and validate a robust archive helper script

Replaced the script file (no PostgreSQL configuration change required) with a safe implementation that copies to a temporary file, performs a file-level flush, then publishes atomically via a hard link. The script compares an existing destination against the source and treats identical files as success while refusing to overwrite differing files. It logs every attempt and removes temporary files on exit. After deployment, a controlled manual archive of the oldest .ready segment returned success and idempotent SKIP on a second attempt; pg_stat_archiver observed advancing last_archived_wal and failed_count stopped increasing, allowing Commvault to resume successful log-backups. This final step introduced the implemented fix and transitions to the Solution below.

Solution:

PostgreSQL continued to use the existing archive_command, but the external helper script was replaced with a robust implementation: copy to a temporary file, sync the temporary file, atomically publish it with a hard link, compare existing files to avoid destructive overwrites, and log all outcomes. No changes to PostgreSQL configuration were required.

Architecturally, the fix prevents partial files from appearing under the final archive name (atomic publish), ensures durability before the file is visible (file-level sync), and enforces idempotent behavior for retries (cmp-based acceptance). Together these properties prevent an out-of-space interrupted copy from permanently blocking WAL archiving and replay.

Conclusion:

After deploying the script, WAL archiving resumed and the pg_stat_archiver metrics advanced; the archive backlog drained and Commvault log-backups returned to success. The change reduces the operational risk from full filesystems and interrupted copies by making archive publishing atomic, durable and idempotent.