Case Studies

Problem: The client reported a recurring issue with the application, which consistently threw an error: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend. He was uncertain about the root cause of this error and sought assistance to diagnose and resolve the issue. The client’s initial request was to understand what should be checked […]

Problem: The client, deploying a production Cassandra cluster on OpenShift, encountered persistent communication issues between two datacenters, each hosting three Cassandra nodes. Despite initial configuration and verification, nodes in Datacenter 1 (“RCKL”) and Datacenter 2 (“CLSP”) were unable to establish consistent communication. SSL handshake failures and inconsistent OpenSSL connectivity tests exacerbated the problem, indicating potential […]

Problem: Airflow login failed with an “Invalid login” prompt for the user “admin” in production. The environment details are as follows: Helm version 1.9.0 and Airflow Image version 2.6.3. The expected outcome was a successful login. Solution: Identify the Issue: The problem was related to login issues with the Airflow UI using the “admin” user. […]

Problem: The client had a 6-node multi-DC setup for Cassandra (3 nodes in PROD – East US2 and 3 nodes in DR – West US2) and needed to alter a keyspace. The keyspace was initially defined as follows: CREATE KEYSPACE bulk_api_management WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'} AND durable_writes = true; The client wanted […]

Problem: A client faced difficulties downloading images from their local Nexus repository using Podman. Despite several troubleshooting attempts, including adding the registry to insecure registries and adding the certificate locally, the issue persisted. The specific error encountered was related to certificate validation. Process: Initial Troubleshooting: The client added the Nexus registry to the list of […]

Problem: A client was experiencing data inconsistencies with the Lucene index on Cassandra and suspected improper shutdown procedures as the root cause. The kill -9 command was used to shut down Cassandra, which led to concerns that data was not being written to disk properly. The client sought guidance on the best way to shut […]

Problem: The customer upgraded their Kubernetes cluster from version 1.19 to 1.24.8. Following this upgrade, they lost access to the PodPreset feature, which was removed in Kubernetes version 1.20. The customer needed a replacement for this functionality and identified Admission Webhooks as a potential solution. However, despite following RedHat’s procedure for implementing Admission Webhooks, the […]

Problem: The client faced significant delays in executing Elasticsearch queries within their production environment. A particular query, which involved a simple numeric account identifier, took an alarming 68 seconds to execute, despite returning only six hits. The total size of the query output was 583KB, yet the Elasticsearch profiler indicated that 67 seconds of this […]

Problem: The client managed a 5-node Cassandra cluster across two data centers (DC1 and DR1), each containing 5 nodes. The data_file_directories were distributed across multiple mount points. On one node, the mount point /cassandra/data2 was nearly full due to a large table in the “jesi” keyspace, specifically the “service_monitoring_payload” table. This resulted in significant storage […]