Case Studies

Problem: The client experienced frequent warnings in their Cassandra nodes, indicated by the log entry: INFO [CompactionExecutor:37] 2023-02-23 12:00:12,268 NoSpamLogger.java:91 – Maximum memory usage reached (536870912), cannot allocate chunk of 1048576 The client requested an investigation into the potential impact of increasing the file_cache_size_in_mb in the cassandra.yaml file, whether a restart (bounce) would be necessary, […]

Problem: The client has requested assistance with the following issue regarding the `connect-eoc-data-summary-to-grid-sink-httpfile-connector`. The connector is experiencing a lag where it is not reading any records, and the offset is not being committed, causing the lag to keep increasing. The client indicated that the grid connector appears to be stuck and has failed. The following […]

Problem: Certain Elasticsearch queries timed out after 30 seconds. Details: The customer used Elasticsearch (version 7.17.0 or slightly newer) to query documents created by the Actimize application. The Elastic index contained approximately 80 million documents, amounting to several terabytes. Typically, queries were executed within a few seconds, but some queries consistently took 30 seconds or […]

Problem: The customer is experiencing the “exec format error” issue when using Prometheus node exporter versions 1.5.0 and 1.6.0 on ARM64 machines, particularly Graviton-type instances in an AWS environment. This error is observed in the node exporter pods running as a DaemonSet in a Kubernetes cluster with nodes having ARM64 architecture. Process: The experts requested […]

Problem: The client encountered a Cassandra exception: “WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM.” This issue, occurring since March 9, 2023, revolves around an INSERT INTO query. They seek troubleshooting assistance as this exception had not occurred before that date. Process: Step 1 – Initial Investigation and Troubleshooting: The expert team initiated […]

Problem: The client reported issues with Keepalived crashes leading to high availability disruptions and application errors, particularly connection timeouts with the PostgreSQL server. Initial investigations revealed suspicions of network instability and outdated software versions contributing to the problem. Process: Requesting initial information for further investigation of the problem The number of servers in the HAProxy […]

Problem: In the Cassandra cluster, the database generated a large number of commit logs and didn’t delete them. Due to this, the commit logs filesystem is getting full and the database is crashing. This is relevant for all nodes. Process: Step 1: Initial Investigation and gather information from the client Initial troubleshooting and information gathering […]

Problem: The user was unable to reach the application page and received the error ‘Unauthenticated Page’. Process: Step 1: Initial Investigation The security issue pertains to an unauthenticated page within the Airflow version 2.5.2 instance. This unauthenticated page poses a potential security risk, as it can be accessed without proper authentication, potentially exposing sensitive information […]

Problem: The client has a two-datacenter (DC1 and DR1) Cassandra cluster. They encountered a failure while running nodetool repair on a node in DC1, which was traced to data corruption on a node in DR1. The logs indicated a corruption error in a specific SSTable file. Solution: Step 1. Initial Diagnosis: Ran nodetool repair in […]