10.08.2026 Problem:
A production PostgreSQL cluster (community PostgreSQL 15.8, managed by Patroni v2.1.4) experienced an unplanned automatic switchover. Symptoms reported included an unexpected promotion of the standby after the primary stopped responding to health checks. Monitoring and OS traces showed memory usage climbing to ~95% on the primary node, PostgreSQL failing to respond to Patroni probe requests, and a rapid, concurrent spike in query and monitoring activity during the incident window.
Relevant configuration and observed signals: max_connections configured as 6000, work_mem = 20 MB, six parallel workers allowed per query, roughly 150 GiB reserved via HugePages on the host, Grafana dashboard/monitoring queries that read approximately 110 GiB within 45 seconds, and two exporter instances running concurrently. The customer also reported concurrent NFT testing and maintenance activity during the timeframe.
Process:
Step 1: Verify incident timeline and cross-reference logs
Observed Postgres logs, Patroni agent logs and OS-level SAR snapshots for the reported window to construct the timeline. The logs showed PostgreSQL stopped responding to Patroni health checks at the point memory usage peaked; Patroni then promoted the standby. This established that the failover was an automatic response to service unavailability rather than a manual operation.
Step 2: Correlate memory metrics with process-level usage
Reviewed host memory reporting, page-cache statistics and per-process RSS/committed numbers from the attached OS logs. Discovery: total memory pressure near 95% comprised reserved HugePages (~150 GiB), kernel page cache, and large cumulative memory from PostgreSQL backend processes. This explained why overall memory appeared saturated even before monitoring queries intensified.
Step 3: Identify high-impact queries and concurrent consumers
Analyzed query samples, Grafana dashboard query patterns and exporter activity in the incident window. Found heavy Grafana queries and two exporter instances reading large volumes (roughly 110 GiB over ~45s). These monitoring queries contributed a high, short-lived I/O and buffer-cache pressure but did not alone account for the full memory spike.
Step 4: Inspect connection behaviour and application activity
Checked connection-rate indicators from available logs and PostgreSQL connection metadata. The application account (anonymized application role) exhibited a rapid connection surge: from ~3 connections/sec to ~72 connections/sec, opening ~1,350 new connections in one minute. The surge began prior to the first memory allocation errors, indicating connection growth was a primary driver of memory consumption. This pattern also aligned with the customer’s concurrent NFT testing activity.
Step 5: Quantify memory reduction impact after service restart
Compared committed process memory before and after a controlled PostgreSQL restart. After restart committed Postgres memory decreased by ~237 GiB, confirming a large portion of the exhausted memory had been held by active backend processes and their working buffers rather than kernel-only allocations. This validated that reducing uncontrolled connections would materially reduce peak memory usage.
Step 6: Synthesize contributing factors into a remediation plan and implement immediate controls
Combined configuration review and observed metrics to produce targeted mitigations: limit uncontrolled client connections with a connection pooler, reduce per-connection memory reservation and parallelism, eliminate redundant monitoring exporters, and adjust HugePages reservation to match expected buffer usage. These mitigations were presented and applied as the immediate fix to prevent repeated failovers while longer-term tuning was planned.
Solution:
Implemented changes focused on limiting connection-driven memory growth and reducing per-query memory pressure in PostgreSQL. A transaction-mode PgBouncer pooler was deployed in front of the application to cap concurrent server connections, application connection pooling and retry behaviour were corrected, max_connections was reduced to remove excessive connection reservation, work_mem and parallel-worker limits were lowered to constrain per-backend memory use, duplicate exporter instances were removed, and HugePages reservations were rebalanced to the actual shared_buffers and OS page-cache requirements.
These changes work because PostgreSQL allocates significant memory per active backend and per-query working area (work_mem and parallel workers). Limiting the number of physical backends with PgBouncer prevents a rapid, multiplicative increase of memory commitments; tuning work_mem and parallelism reduces worst-case per-query allocations; aligning HugePages prevents large amounts of effectively unusable reserved memory from contributing to overall pressure. Together these changes restored predictability of memory headroom and prevented health-check timeouts that triggered Patroni failover.
Conclusion:
Operational outcome: immediate reduction in transient memory peaks and elimination of repeat automatic failovers caused by connection storms. Post-change monitoring showed substantially lower peak committed Postgres memory (consistent with PgBouncer-limited connections) and reduced risk of service unavailability during heavy query or monitoring activity. Risk mitigation: stricter connection control and tuned per-query memory parameters significantly lower probability of future memory-exhaustion induced failovers.