21.07.2026 Problem:
A production alert arrived with the subject indicating “client replied sms” and a Severity 3 priority: inbound reply messages were not being recorded by the downstream processing pipeline. The affected pipeline is an Apache Spark Structured Streaming job that consumes reply events from a message broker and uses a persistent checkpoint directory to track progress. Observable behavior included incoming messages visible in the broker but no corresponding processing records in the application datastore, and the Spark application showing long-running driver uptime with consumer offsets not advancing.
The original support contact contained only the alert notification and no attached logs; the case was later consolidated into an existing incident for the same symptom.
Process:
Step 1: Alert triage and incident consolidation
The initial step reviewed the incoming alert metadata and the active incident list, noticing no application logs attached to the new report; matching alert history and an open incident with identical symptomology were located, so the report was merged into that incident for a single-track investigation. This consolidation ensured subsequent investigations used a single set of application logs and metrics rather than duplicated, inconsistent data.
Step 2: Streaming health metrics and broker offset comparison
Observed Spark Structured Streaming metrics from the application UI and the message broker’s consumer offset view; compared committed offsets and broker-end offsets for the pipeline’s consumer group. Discovered that broker-end partitions contained recent reply messages while the Spark application’s committed offsets had not advanced for the same partitions. This indicated the pipeline was connected but not committing or processing new messages, establishing the scope of the failure (ingestion gap rather than upstream delivery failure).
Step 3: Driver and executor log inspection
Fetched and scanned driver and executor logs for the active Spark application focusing on checkpoint, commit, and filesystem errors. Found repeated filesystem access errors when the driver attempted to write checkpoint metadata, and occasional warnings about state reconstruction from a stale checkpoint. These log patterns explained why offsets weren’t advancing: checkpoint failures prevented the streaming query from committing progress, causing the application to remain in a non-advancing state.
Step 4: Configuration and storage access review
Reviewed Spark configuration values related to structured streaming (checkpointLocation), filesystem mount/authorization for the checkpoint path, and the consumer group id used for the broker subscription. Discovered a misaligned consumer group id between the deployed job configuration and the expected group recorded in operational documentation, plus restrictive permissions on the checkpoint storage location that had recently changed due to a storage-side policy update. Both findings mattered because the consumer group mismatch could route messages to a different consumer set while checkpoint write failures blocked the job from persisting processed offsets.
Step 5: Controlled remediation and verification
Validated remediation options by testing access to the checkpoint storage using a safe, read-only validation process and by deploying a config-only test of the consumer group change in a staging replica. After confirming that checkpoint writes were blocked by storage ACLs and that the corrected consumer group in staging consumed messages correctly, the decision was to apply a controlled fix: restore appropriate ACLs for the checkpoint directory, update the production Spark job configuration with the correct consumer group id, clear the minimal corrupted checkpoint state following the documented operational runbook, and restart the Structured Streaming query in a controlled window. This step introduced the implemented fix and led directly into the final solution verification: offsets advanced and reply messages were processed end-to-end.
Solution:
The implemented change corrected checkpoint storage permissions, aligned the Spark Structured Streaming job’s consumer group id with the documented deployment configuration, removed the corrupted checkpoint fragment (after taking backups), and restarted the Structured Streaming query. These changes explicitly targeted Apache Spark’s checkpoint-based progress tracking and the message broker consumer semantics. Architecturally, the fix works because Spark Structured Streaming relies on a writable checkpointLocation to persist query progress (offsets and state); restoring write access allowed the driver to commit progress again, and aligning the consumer group ensured the broker delivered replies to the intended consumer, enabling normal offset advancement and stateful processing.
Conclusion:
After the restart, broker offsets advanced and inbound reply messages flowed through to the datastore with no data loss observed for new messages. Operational stability improved: checkpoint persistence resumed, the processing lag returned to normal, and a short runbook update plus a checkpoint-access monitor were added to reduce recurrence risk.