04.08.2026 Problem:
An OpenSearch 2.9 cluster (3 nodes) generated a critical alert: one node reported swap utilization at ~80%, with the OpenSearch process identified as the primary swap consumer. The node’s runtime Java command line showed -Xms10g -Xmx10g and -XX:MaxDirectMemorySize=5368709120 (5GB). The environment reported ~23 GB total RAM. The customer asked how to reduce swap use and improve predictability.
Process:
Step 1: Confirm alert details and collect targeted outputs
Alert payload included a single-node swap utilization metric (~80%). Requested outputs were vm.swappiness (/proc/sys/vm/swappiness), free -g, OpenSearch node listing (_cat/nodes) and cluster health, and the running Java startup parameters plus the node’s configuration directory. These artifacts were needed to correlate OS memory policy, JVM sizing and cluster role.
Step 2: Inspect JVM process parameters
ps output contained the OpenSearch bootstrap command line showing a fixed JVM heap of 10g (-Xms/-Xmx) plus a 5GB MaxDirectMemorySize. This indicated significant native memory alongside the JVM heap. That combination materially increases the process resident set size compared with heap-only assumptions, which directly affects how close the process is to available physical RAM.
Step 3: Review OS memory state and swappiness
System metrics returned total RAM ≈23 GB and vm.swappiness = 30. The free/memory snapshot showed nontrivial free memory but persistent swap usage. This pattern revealed that the kernel was allowed to swap under moderate memory pressure due to the swappiness setting and the large native+heap footprint of the OpenSearch process.
Step 4: Analyze OpenSearch configuration
Collected opensearch.yml and jvm.options confirmed default-like configuration with a 10 GB heap on a modest node. Cluster health and node lists were reviewed to ensure the node was not running unexpected roles. The configuration confirmed the node was a data node expected to hold normal indexing/search workload, so predictable memory behavior was important for latency and stability.
Step 5: Map findings to risk and options
The combination of: 23 GB total RAM, 10 GB JVM heap, 5 GB configured direct memory, and vm.swappiness=30 explained why the process was putting pressure on RAM and the kernel was swapping. Swapping of JVM pages causes unpredictable latency spikes and degradation for OpenSearch; keeping hot JVM memory resident is therefore the primary goal. Options considered were: reduce swappiness, disable swap, change heap sizing, or increase physical RAM.
Step 6: Provide targeted remediation and operational guidance
Recommended concrete changes were prepared and delivered to the customer: (a) set vm.swappiness=0 via sysctl (and persist in /etc/sysctl.conf) or disable swap (swapoff -a and remove swap entries from fstab) for production OpenSearch nodes, (b) align JVM heap with available RAM — either increase heap to 16–18 GB only if the node has sufficient dedicated RAM or reduce heap if other services share the host, and (c) plan a controlled OpenSearch restart to apply heap changes. Additional notes covered OOM risk if swap is removed without correct heap sizing and the need to schedule a restart during a maintenance window. The customer confirmed they would implement the recommendations and asked the case to be closed.
Solution:
Provide and document two principal configuration changes for the OpenSearch node: set vm.swappiness=0 (sysctl -w vm.swappiness=0 and persist) or disable swap entirely, and adjust OpenSearch JVM heap (-Xms and -Xmx) so aggregate process memory stays comfortably below physical RAM. Restart OpenSearch to apply JVM heap changes. These changes prevent the kernel from moving hot OpenSearch pages to swap and ensure the JVM and its native buffers remain resident, reducing unpredictable latency and improving resource predictability.
Conclusion:
After guidance, the node was left with an actionable remediation plan to remove swap pressure and align JVM sizing with host capacity. Expected operational improvements: lower swap utilization, more consistent query latency, and reduced risk of performance degradation caused by kernel swapping. A follow-up support engagement was recommended if RAM upgrades or heap-sizing adjustments are to be performed across the cluster.