Case Studies

Problem: The client experienced repeated failures in two Apache Airflow DAGs responsible for launching Kubernetes Jobs. Each DAG followed a delete-and-recreate pattern using a fixed Kubernetes Job name. Although the Airflow task responsible for deleting the Job reported success, the subsequent Job creation consistently failed with a Kubernetes conflict error indicating that the Job already […]

Problem: The client reported recurring kernel messages appearing in the system logs while running OpenZFS on a virtualized Linux server. Using dmesg -T, the customer observed approximately five identical stack traces per second, raising concerns about potential impact on production stability and data integrity. The log messages consistently referenced memory allocation paths within ZFS and […]

Problem: The client experienced an intermittent disappearance of a critical Apache Airflow DAG — Dag-Factory. The DAG would appear in the Airflow UI and CLI, then vanish from the DagBag and UI for periods of time before reappearing. Each disappearance caused pipeline failures (notably the need to re-trigger the BONG initial load), interrupting production workflows. […]

Problem: The client experienced an unexpected failover in a production PostgreSQL Patroni cluster running PostgreSQL Community version 15.8 and Patroni version 2.1.4 configured in asynchronous replication mode. Node A unexpectedly went offline and Patroni automatically promoted Node B to the primary node. The client needed to determine: The reason for the failover. Whether any transactions […]

Problem: After restoring a PostgreSQL 15.8 database from Commvault, the client noticed multiple errors during startup. Autovacuum began trying to remove temporary tables, but instead reported issues such as: “Dropping orphan temp table” “pg_attribute catalog is missing attributes” Indexes and constraints appearing only partially restored About 30 temp tables were affected, and several indexes or […]

Problem: The client, reported repeated network timeouts and instability in their Apache Kafka environment, resulting in missing Call Detail Records for specific time periods. The issue manifested as persistent connectivity errors between Kafka brokers and 100% CPU utilization on the node hosting critical services (UMS, App, and ODF pods). Process: Step 1: Initial Identification The […]

Problem: The client operates a 6-node multi-DC replication setup for Cassandra, consisting of 3 nodes in the PROD datacenter and 3 nodes in the DR datacenter. They are planning to perform DDL changes, including altering tables to adjust the default_time_to_live parameter and dropping and recreating a table with a new definition. However, they are unsure […]

Problem: The client encountered a critical issue while starting one of their production pods during a Cassandra migration. Although the PostgreSQL migration completed successfully, the Cassandra migration failed with a com.datastax.oss.driver.api.core.AllNodesFailedException, indicating that the driver could not connect to any Cassandra nodes. This blocked the production deployment. Process: Step 1: Initial Analysis The logs revealed […]

Problem: The client using Argo Workflows identified a compliance issue during a Twistlock (Prisma Cloud) scan. The image argoproj/argoexec:v3.7.1 failed the CIS Docker Benchmark 4.1 rule, which requires containers to run as a non-root user. Running containers as root posed a risk of privilege escalation and potential account takeover. The client wanted to resolve the […]