Case Studies

Problem: The production OpenSearch cluster encountered frequent shutdowns due to Out of Memory (OOM) errors across all four nodes. The exact cause of the OOM issue needed further investigation, requiring specific data and logs to diagnose the root cause. Process: Resource Assessment: Evaluate RAM, CPU, and disk usage on each node to identify potential resource […]

Problem: The client’s billing system experienced around 10 billing cycles each month. During each billing cycle, the billing process was initiated and connected to the database cluster with 20 concurrent streams. However, upon starting the process, both the replication and the billing process failed, displaying the following error: “Could not send data to the client: […]

Problem: The client seeks to configure SSL communication within an existing Patroni/etcd/Postgres cluster, specifically aiming to switch to HTTPS in the Patroni configuration file to secure communication between components. Solution: After a thorough analysis, the following recommendations were made: Certificate Generation Utilize OpenSSL or obtain a certificate from a trusted Certificate Authority (CA). For self-signed […]

Problem: The client is experiencing high read latency in their production cluster monitoring. They are seeking assistance in identifying the cause of this latency and resolving it to prevent potential outages. Process: Steps and measures undertaken to investigate the issue: Initial Assessment: Requested logs/config from all nodes. Observed server overload or potential network issues. Configuration […]

Problem: Java client for Apache Kafka is missing error logs when attempting to publish messages to a Kafka topic that is full on the broker side. Despite expecting error logs in such cases, the logs only show warnings and traces, hindering quick identification of the root cause. Solution: Main solution steps suggested to resolve the […]

Problem: The client’s Prometheus pod, despite having substantial memory resources, is experiencing prolonged startup times, likely due to extended WAL (Write-Ahead Logging) loading durations. This delay leads to readiness probe failures and leaves the pod in a failed state. The client seeks a resolution to mitigate this performance issue and ensure prompt pod initialization. Solution: […]

Problem: After a successful installation of OpenSearch on a Red Hat Enterprise Linux 8.9 system, attempts to start the service fail with the error message: “Could not initialize class com.sun.jna.Native.” Additionally, there is a warning indicating the inability to load JNA (Java Native Access) native support library, resulting in disabled native methods. Solution: The following […]

Problem: A scheduled job did not trigger at its designated time, and upon attempting to run the job manually, the pods failed to come up in the backend. Reviewing the scheduler logs, the client did not find any specific errors. Process: Our expert investigated the following logs and data: Airflow Configuration: Relevant details of the […]

In iTop a high severity vulnerability CVE-2023-47626 was detected. When viewing or editing personal tokens, users should be aware that cross-site scripting attacks could occur. This vulnerability is resolved in version 3.1.1. For more information, visit https://avd.aquasec.com/nvd/2023/cve-2023-47626/.