Data Analytics

Problem: A client with an Elasticsearch cluster consisting of three nodes was experiencing a recurring issue where one of the nodes is disconnecting from the cluster automatically. This disruption was resulting in numerous unassigned shards, impacting the overall stability and performance of the Elasticsearch environment. Process: Data Collection for Further Analysis: Request: Provide detailed system […]

Problem: The client’s Elastic server is experiencing sporadic crashes, evident from StackOverflowError logs linked to “GraphTokenStreamFiniteStrings.” While a potential fix exists in Lucene 9.7, the product’s certification with Elastic 7.17 using Lucene 8.11.1 complicates the implementation of the fix. The client seeks assistance to evaluate the possibility of backporting Lucene 9.7 changes to Lucene 8.11.1. […]

Problem: The problem identified in Apache Airflow version 2.5.0 was the lack of password confirmation during password changes, posing a significant security risk to users. This vulnerability could potentially lead to unauthorized access and session hijacking within the Airflow application. Solution: Based on the client’s request and the provided information, the recommended solution steps to […]

Problem: The client was facing a situation where fluent-bit containers jeopardized the k8s host by overusing the temporary space which is on the host. Solution: After the investigation, the expert team suggested the next solution to the client: Resource Limitations: Checked for resource limitations on the fluent-bit containers. Insufficient CPU or memory limits could have […]

Problem: The production OpenSearch cluster encountered frequent shutdowns due to Out of Memory (OOM) errors across all four nodes. The exact cause of the OOM issue needed further investigation, requiring specific data and logs to diagnose the root cause. Process: Resource Assessment: Evaluate RAM, CPU, and disk usage on each node to identify potential resource […]

Problem: Java client for Apache Kafka is missing error logs when attempting to publish messages to a Kafka topic that is full on the broker side. Despite expecting error logs in such cases, the logs only show warnings and traces, hindering quick identification of the root cause. Solution: Main solution steps suggested to resolve the […]

Problem: The client’s Prometheus pod, despite having substantial memory resources, is experiencing prolonged startup times, likely due to extended WAL (Write-Ahead Logging) loading durations. This delay leads to readiness probe failures and leaves the pod in a failed state. The client seeks a resolution to mitigate this performance issue and ensure prompt pod initialization. Solution: […]

Problem: After a successful installation of OpenSearch on a Red Hat Enterprise Linux 8.9 system, attempts to start the service fail with the error message: “Could not initialize class com.sun.jna.Native.” Additionally, there is a warning indicating the inability to load JNA (Java Native Access) native support library, resulting in disabled native methods. Solution: The following […]

Problem: A scheduled job did not trigger at its designated time, and upon attempting to run the job manually, the pods failed to come up in the backend. Reviewing the scheduler logs, the client did not find any specific errors. Process: Our expert investigated the following logs and data: Airflow Configuration: Relevant details of the […]