Case Studies

Problem: The issue stems from confusion surrounding the Replication Factor (RF) in Apache Cassandra, where the documentation implies RF determines the quantity of data copies, but clarity on this matter is lacking. Process: To address this issue, in-depth investigation took place. The team explored the information provided by the client: Cassandra Configurations: Cassandra Version: 4.0.1 […]

Problem: The system is generating alerts indicating replication lag on a PostgreSQL instance. However, upon DBA investigation, no observable lag was found in the cluster, with both the leader and the synchronous standby reporting no lag. This discrepancy raises concerns about the persistence of alerts despite the absence of actual lag. Solution: Monitoring on Postgres: […]

Problem: The organization is using Cassandra 4.0.x in its Production environment and requires tracking login and logout details from the Cassandra Audit logs. However, the Audit logs only record LOGIN_SUCCESS, LOGIN_ERROR, and UNAUTHORIZED_ATTEMPT events, with no mention of LOGOUT events. The organization seeks clarification on mechanisms for capturing logout-related entries. Solution: Cassandra does not inherently […]

Problem: Ceph Storage Almost Full but Should Have Space. The client reported that the Ceph storage is nearly full, even though there should be sufficient space available. The output of ceph osd status indicates that some OSDs have limited available space. The most common cause identified is not deleting the lost+found directory after a crash […]

Problem: The client reported sudden high CPU utilization on their RHEL 7.5 Nexus server. Despite disabling tasks and restarting the server, the issue persisted. Logs indicated that the Nexus process was the top contributor to CPU utilization. Process: Steps and measures undertaken to investigate the issue: Requesting Information: Furnished JVM logs, Nexus application logs, and […]

Problem: Need Explanation for the Pipeline: Our client has encountered a scenario in their Elasticsearch setup that requires clarification and understanding. The specific concern revolves around the configuration of the Logstash pipeline, more precisely, the interaction between the defined schedule and query parameters. Logstash Configuration LogstashConfig: pipelines.yml: | - pipeline.id: logstash-output-broker schedule: "*/5 * * […]

Problem: The client encountered an error while updating the Prometheus adapter in the Azure environment, specifically related to the prometheus-adapter-configmap. The error stemmed from the generated configmap exceeding the Kubernetes limit, preventing successful deployment. The client sought guidance on efficiently splitting the configuration for deployment across multiple namespaces. Solution: Step 1: Environment Preparation: Experts established […]

Problem: The Production (Prod) and Disaster Recovery (DR) environments for Cassandra databases exhibit different sizes, as observed from the nodetool status output. The client needs to identify the reason behind this discrepancy and ensure the readiness and reliability of the Disaster Recovery environment for use in the event of a failure. Process: The client provided […]

Problem: The issue revolves around an unexpected change in the TTL for our Cassandra cluster. There are approximately 28 tables, six of which contain around 2 million records each. The requirement is to update the TTL for these tables to three years. However, TTL is configured at the row level. The client seeks alternative methods […]