Data Analytics

Problem: A Prometheus deployment failed a security compliance scan when Prisma Cloud flagged a high-severity vulnerability in the sidecar container image kiwigrid/k8s-sidecar:1.30.9. The issue was linked to the pip package, which was identified as vulnerable to arbitrary code execution. This raised concerns about both compliance and the security of the monitoring stack. Process: Step 1: […]

Problem: The client reported that their Apache Airflow Scheduler pod was not automatically detecting updates from a Persistent Volume Claim (PVC) named airflow-dags. Meanwhile, a related airflow-triggerer pod using the same PVC was updating files as expected. To temporarily resolve the issue, the client had to manually restart (“bounce”) the scheduler pod to force it to recognize new […]

Problem: A financial services client encountered critical SSL-related errors while deploying a two-node OpenSearch 1.3.6 cluster for high availability. Despite both nodes appearing operational, accessing indices or interacting with the cluster through a Java application resulted in errors such as: SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment CertPathBuilderException: unable to find valid certification path […]

Problem: The client faced recurring Kafka sink connector failures (e.g., chf-cdr-sftp-sink-connector) in a Kubernetes environment (Kafka 3.2.0 with three brokers and ZooKeeper). The failures were caused by corrupt messages at specific offsets, leading to task crashes. Despite skipping corrupt offsets and restarting connectors, the issue persisted, requiring a more permanent solution. Process: Step 1: Environment […]

Problem: The client reported continuous heartbeat issues in the Airflow scheduler, causing failure to generate controller DAGs in a production environment. This critical issue impacted job execution, especially when multiple jobs were triggered simultaneously, leading to timeouts and job failures. Process: Step 1: Initial Identification The error message displayed in the logs indicated that the […]

Problem: The client encountered an issue in their Elasticsearch setup where search results did not return exact matches when the search phrase included special characters, such as “:” (colon). This problem persisted despite using a custom indexing configuration with the `index_word_delimiter_graph_filter`. The client needed a solution to preserve special characters for exact matches while maintaining […]

Problem: The client implemented a 4-node OpenSearch cluster to ensure high availability for their application. When all four nodes were operational, both indexing and searching worked seamlessly. However, during a high availability test where two nodes were intentionally turned off, the indexing process stalled, and no documents were processed. Indexing resumed only after the two […]

Problem: The client’s operations team reported issues with triggering jobs via Apache Airflow, specifically through a custom solution, the dag_factory. While jobs triggered outside of the dag_factory worked without problems, those initiated through it were not being processed as expected. Attempts to gather logs in the Airflow UI yielded no entries, as the DAG triggering […]

Problem: The client encountered issues with the data-explorer view functionality in their CKAN environment. While resources could be downloaded manually, the data-explorer view was unable to load. During the initial investigation, it was found that while the “datastore” plugin was enabled in the ckan.ini file, the ckan.datastore.write_url and ckan.datastore.read_url were not configured. The client was […]