Problem: Ceph Storage Almost Full but Should Have Space. The client reported that the Ceph storage is nearly full, even though there should be sufficient space available. The output of ceph osd status indicates that some OSDs have limited available space. The most common cause identified is not deleting the lost+found directory after a crash […]
Storage 22 Apr 2024 Resolving High CPU Utilization Issue on RHEL 7.5 Nexus ServerProblem: The client reported sudden high CPU utilization on their RHEL 7.5 Nexus server. Despite disabling tasks and restarting the server, the issue persisted. Logs indicated that the Nexus process was the top contributor to CPU utilization. Process: Steps and measures undertaken to investigate the issue: Requesting Information: Furnished JVM logs, Nexus application logs, and […]
Developer Tools 21 Apr 2024 Understanding Logstash Pipeline Configuration: Query and Schedule ParametersProblem: Need Explanation for the Pipeline: Our client has encountered a scenario in their Elasticsearch setup that requires clarification and understanding. The specific concern revolves around the configuration of the Logstash pipeline, more precisely, the interaction between the defined schedule and query parameters. Logstash Configuration LogstashConfig: pipelines.yml: | - pipeline.id: logstash-output-broker schedule: "*/5 * * […]
Data Analytics 20 Apr 2024 Handling DDL Changes and Replication Issues in Multi-DC Cassandra SetupProblem: The client operates a 6-node multi-DC replication setup for Cassandra, consisting of 3 nodes in the PROD datacenter (East US2) and 3 nodes in the DR datacenter (West US2). They are planning to perform DDL changes, including altering tables to adjust the default_time_to_live parameter and dropping and recreating a table with a new definition. […]
Database 19 Apr 2024 Resolving Kubernetes Deployment Error: Configmap Limit Exceeded for Prometheus Adapter in Azure EnvironmentProblem: The client encountered an error while updating the Prometheus adapter in the Azure environment, specifically related to the prometheus-adapter-configmap. The error stemmed from the generated configmap exceeding the Kubernetes limit, preventing successful deployment. The client sought guidance on efficiently splitting the configuration for deployment across multiple namespaces. Solution: Step 1: Environment Preparation: Experts established […]
Data Analytics 18 Apr 2024 Resolving Data Discrepancies in Cassandra Cluster EnvironmentsProblem: The Production (Prod) and Disaster Recovery (DR) environments for Cassandra databases exhibit different sizes, as observed from the nodetool status output. The client needs to identify the reason behind this discrepancy and ensure the readiness and reliability of the Disaster Recovery environment for use in the event of a failure. Process: The client provided […]
Database 17 Apr 2024 Updating TTL on Сassandra tablesProblem: The issue revolves around an unexpected change in the TTL for our Cassandra cluster. There are approximately 28 tables, six of which contain around 2 million records each. The requirement is to update the TTL for these tables to three years. However, TTL is configured at the row level. The client seeks alternative methods […]
Database 16 Apr 2024 Dag Factory Trigger Failure: Diagnosis and Resolution of Airflow Pod IssuesProblem: The client reported issues with all dags triggered from dag-factory failing, with associated problems such as pods not coming up when triggered and errors in collector pods. Furthermore, there was a lack of error messages in the airflow pods logs, making it challenging to identify the root cause. Process: In order to address the […]
Data Analytics 16 Apr 2024 Designing a Scalable Cassandra Topology for Production: Architectural Considerations and Best PracticesProblem: The client sought consultation on the best way establishing the standard Cassandra topology for a production-ready environment. Process: Cassandra Configurations: 2 Data Centers (PROD and DR). Each DC has 3 nodes of Cassandra. 2 DC are part of one cluster, thereby forming a ring architecture of 6 nodes (3 PROD + 3 DR). RF […]
Database