15.04.2026 Problem:
Argo CD UI responsiveness and application sync/refresh operations experienced frequent slowness and timeouts on a bare‑metal OpenShift cluster. Symptoms included UI timeout and long sync times when running bulk refresh/sync operations against an app‑of‑apps topology: sync jobs that exceeded a 1,800s pipeline timeout would fail. The environment manages ~5,200 Argo CD applications on a single connected Kubernetes cluster. The deployment used a single Application Controller replica (ARGOCD_CONTROLLER_REPLICAS=1) with large allocated resources (controller process sized to ~25 CPU / 150 GiB), Argo Server configured at ~500m CPU / 2 GiB RAM, and repo-server experiments (replicas/PVC) showed no consistent improvement. Client tuning attempts that reduced symptoms included raising the Kubernetes client QPS and BURST and increasing cluster cache page and buffer sizes.
Process:
Step 1: Confirm reported behavior and scope
Observed the customer’s description of slow UI response and frequent sync timeouts when running large refresh/sync operations (notably app‑of‑apps with 60–140 child apps). Reviewed the scale context: ~5,200 total applications, single connected cluster, and pipeline failures when single sync exceeded 1,800s. This clarified the issue was a high concurrency workload against a single control plane rather than a client‑side bug.
Step 2: Review Argo CD runtime configuration and recent changes
Collected the deployed Argo CD configuration values the customer had already modified: Application Controller run as a single replica, controller CPU/memory increased to a high allocation, Argo Server scaled to 3 replicas briefly (no benefit), repo‑server replica experiments, ARGOCD_K8S_CLIENT_QPS set to 150 and ARGOCD_K8S_CLIENT_BURST set to 300, and cluster cache PAGE_SIZE/BUFFER_SIZE tuned to 10000/10. Also confirmed controller.status.processors=50 and controller.operation.processors=25 were present. These confirmed substantial controller‑side tuning had been attempted.
Step 3: Inspect logs and error signals for control‑plane throttling
Examined application controller events and logs reported by the customer: there were few explicit Argo CD errors but occasional reconciliation timeouts; no persistent application errors like continuous “context deadline exceeded” or explicit etcd error floods were present. The intermittent nature of timeouts suggested sporadic control‑plane latency or API throttling under high fan‑out instead of deterministic controller crashes.
Step 4: Evaluate control‑plane and infrastructure factors
Requested and reviewed control‑plane metrics and disk characteristics to validate suspected bottlenecks: recommended inspection of ETCD fsync latency and disk type. The design analysis showed a single Kubernetes API/etcd control plane would receive large bursts of concurrent list/watch and CRUD requests when syncing many children of app‑of‑apps; this pattern produces short spikes that can trigger API server throttling or increased etcd latency even when Argo CD itself is generously resourced.
Step 5: Verify cache and client throttling changes reduce API pressure
Validated the customer’s observed improvements after they applied K8s client tuning and cluster cache changes: increasing ARGOCD_K8S_CLIENT_QPS to 150 and BURST to 300 reduced API throttling symptoms during spikes, and PAGE_SIZE/BUFFER_SIZE tuning reduced the number of paginated list operations. These changes materially reduced median sync times (from >30 min down toward ~15 min for many workloads) but did not eliminate outliers for very large app‑of‑apps jobs.
Step 6: Define constrained remediation path and implement hardened configuration
Because adding multiple Application Controller replicas was not feasible in the customer’s single‑cluster topology (per operator constraints for their deployment pattern), the remediation focused on sustainable client‑side and caching changes plus lightweight server optimizations: keep the high controller CPU/memory allocation, retain the increased K8s client QPS/BURST, keep cluster cache PAGE_SIZE/BUFFER_SIZE tuned, enable HTTP compression on argocd‑server to reduce payload size for UI traffic, and confirm Redis caching is not saturated. These were applied and re‑tested; sync job tail‑latency improved and UI responsiveness was noticeably better during bulk operations.
Solution:
Argo CD was left configured with elevated controller resources, ARGOCD_K8S_CLIENT_QPS=150 and ARGOCD_K8S_CLIENT_BURST=300, and cluster cache parameters PAGE_SIZE=10000 and BUFFER_SIZE=10. UI compression was enabled on the argocd‑server and Redis usage was validated to ensure caching was not a limiting factor. No additional controller replicas were added because the deployment model constrained HA controller sharding on a single connected cluster.
These changes work because they reduce control‑plane request amplification (larger page sizes and buffering), allow short spikes to be served without immediate client‑side throttling (higher QPS/BURST), and reduce payload pressure on the Argo CD API/UI path (compression). Together these measures mitigate transient API server/etcd contention when Argo CD performs large, concurrent reconciliation operations.
Conclusion:
Post‑changes, median sync times decreased (commonly from >30 minutes toward ~15 minutes) and UI responsiveness improved during bulk refreshes. Outlier tail latency for very large app‑of‑apps syncs was reduced but not entirely eliminated; recommended next steps for further improvement are control‑plane I/O tuning (etcd disk type and fsync latency), APF tuning on the cluster control plane, or adopting a multi‑cluster controller sharding model where feasible. The immediate operational outcome was a measurable reduction in failed pipeline runs due to sync timeouts and improved day‑to‑day responsiveness of Argo CD.