Data Analytics

Problem: In the production environment of a multi-node OpenSearch cluster, the nodes frequently crashed due to Out-of-Memory (OOM) errors. Initially, the heap size was increased from 16 GB to 30 GB, but the issue persisted. The client then tried increasing the number of shards from 16 to 64 to reduce memory pressure. However, they sought […]

Problem: The client encountered a critical issue in their production environment involving HBase regions stuck in a transition state. This problem resulted in service disruptions within their Hadoop cluster. The issue was exacerbated by file system permission changes following a cold restart of the cluster, leading to difficulties in accessing data and managing HBase operations. […]

Problem: The client requested assistance with upgrading their Elasticsearch installation from version 7.15 to 7.17. The client sought a detailed step-by-step guide and expressed the need for a meeting to clarify the upgrade process. Process: Upon receiving the request, the expert requested additional details about the client’s current Elasticsearch setup, including information on the cluster, […]

Problem: The client experienced issues setting up a second node for high availability in their OpenSearch cluster. While the first node worked properly, the second node encountered SSL handshake errors when both nodes were used simultaneously. Process: The expert reviewed the client’s YML configurations and SSL certificate setups. They provided detailed feedback on correcting SSL […]

Problem: The client reported an issue and requested support to address potential vulnerabilities. A security issue was identified in an Apache Airflow instance, version 2.5.0, involving the absence of an account lockout mechanism. Solution: Our expert identified two key measures to enhance Airflow security and responded to the client’s request with the following recommendations: Implementing […]

Problem: Our customer, using Opensearch (1 node) with SSL in their production environment, faced challenges while implementing a new node to ensure high availability. When individually pointing our application (Actone) to the new node, indexing and searching worked as expected. However, integrating both nodes simultaneously resulted in a – javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown error […]

Problem: The client faced Spark issues during job submissions through Airflow. Out of 15 max DAGs that got triggered, there were always 3-4 dags getting triggered in the Initial state and never moving to the running state. A few DAGs also got stuck in the ContainerCreate state and never came up and ran. Process: Step […]

Problem: The client seeks guidance on passing multiple configuration files to Prometheus Adapter (P8s) version v0.11.1 without encountering size limitations or performance issues. They aim to aggregate custom metrics efficiently across microservices in Kubernetes. Process: The experts discussed various aspects of configuring Prometheus Adapter, addressing concerns such as size limitations with config maps and optimizing […]

Problem: The client, operating a Kafka Streams application version 3.3.1, encountered significant issues with managing changelog topics. Despite configuring the application for automatic record cleanup, the changelog topics exhibited unchecked growth. This accumulation of records risked deteriorating system performance and stability, potentially impacting the application’s reliability. Process: Initial Assessment: Issue Reporting: The client highlighted problems […]