Troubleshooting High Availability Setup in OpenSearch

Problem:

OpenSearch version 1.3.6 RCM OpenSearch DR(COB) High availability setup is not working in the Production environment and lower environment. For Production, the client has a 4-node cluster, out of which 50% availability cluster setup is configured. The client expects that when one node goes down, the other will take over as master. High Availability means that if any 1 or 2 nodes get down due to any issues out of 4 nodes, other 2 nodes should take the load/traffic to continue the functionality of Global Search. The client provided logs and configuration files from the Production environment for investigation.

Process:

1. Investigating the logs and configuration files from the production environment.
2. Requesting additional data to find the root of the problem:
   1. Content of OpenSearch config directory.
   2. Content of OpenSearch log directory.
   3. The output of OpenSearch index cat API.
3. Q&A:
   • Requesting more detail on why High availability setup is not working in the Production environment.
   • Requesting the output from http://SERVER:POST/_cluster/health?pretty link.
4. Noticing any evidence of the cluster not being functional when one node goes down.

Solution:

To address the issue, the following changes were made:

• Changed the discovery.zen.minimum_master_nodes parameter on all nodes:
  • From: discovery.zen.minimum_master_nodes: 1
  • To: discovery.zen.minimum_master_nodes: 3
• Updated the discovery.seed_hosts parameter on all nodes.
• Provided a full opensearch.yml configuration file from a well-functioning cluster as a reference.

Conclusion:

The changes made to the configuration parameters, along with the reference from a well-functioning cluster, resolved the issue with OpenSearch High Availability setup. It was observed that OpenSearch does not work when the master node goes down, and other nodes keep failing stating that the Master Node is not discovered. By adjusting the minimum master nodes and updating seed hosts, the cluster’s functionality was restored, ensuring High Availability in the Production environment.