09.09.2026 Problem:
An application gateway using the Open Enterprise SDK for Apache Kafka reported that Notification ID 8 was accepted and enqueued by the gateway but never showed downstream dispatch evidence for two subscribers. Gateway instrumentation recorded REST_RECEIVE and SEDA_ENQUEUE events for those transactions, while downstream stages (SEDA_DEQUEUE, HTTP_SEND, EDR_WRITE) were absent in the available exports. Across the same time window there were 5,088 provisioning attempts and 5,086 matching downstream EDR records, indicating exactly two missing deliveries. One missing transaction appeared as an enqueue without a matching dequeue in the gateway-level per-second counts; the other had no gateway logs in the export window (an export gap existed).
Deployment context: the live delivery path runs inside custom notification gateway pods and uses Apache Camel’s in-memory SEDA endpoint (seda://asyncNotificationProcessor) as the intra-pod handoff. Gateway logs mark each stage with detail-aspect labels but provide only an “accepted/enqueued” acknowledgment at REST_RECEIVE.
Process:
Step 1: Reproduce and scope from client data
Client-supplied exports of provisioning and ADS/EDR records were analyzed first. Counts were compared per-notification and per-second; a gap of two records out of 5,088 was confirmed and correlated to two specific subscriber deliveries where EDR_WRITE was not present. This established the problem was rare, deterministic for those transactions, and observable as enqueue-without-dequeue in the gateway data for at least one case—important because it narrowed diagnosis to the gateway-to-downstream handoff.
Step 2: Inspect messaging topology and semantics
Route topology and component choices were reviewed next. The gateway uses Apache Camel with a seda:// endpoint acting as a per-pod, in-memory queue. That semantic was decisive: seda:// is non-persistent and scoped to the JVM instance. This explained how a request can be logged as “accepted and enqueued” yet never reach downstream consumers if the in-memory exchange is discarded or lost; it shifted focus from Kafka cluster loss to transient pod-level behavior or queue-drop policy.
Step 3: Correlate runtime signals with pod lifecycle and queue behavior
Pod status output supplied by the client showed gateway pods in Running state with zero restart counts at the time of the capture, reducing the likelihood of a full pod restart after the fact but not excluding transient JVM exits or short-lived evictions. Per-second enqueue/dequeue tallies for the implicated second showed 20 enqueues and 19 dequeues with the missing transaction present in the enqueue set—this strongly pointed at an in-process loss inside the pod (queue discard or lost during shutdown) rather than upstream acceptance failure.
Step 4: Investigate queue configuration and logs for silent drops
Configuration options for the SEDA endpoint and Camel shutdown semantics were requested and reviewed conceptually: bounded queue size plus discardWhenFull (or discardIfNoConsumers) can silently drop producer requests while the producer sees success, and abrupt JVM termination during shutdown can drop in-memory exchanges if purgeWhenStopping or insufficient graceful shutdown is configured. The presence of an export gap for the second missing case kept open the possibility that the request never reached any pod, so ingress access logs were also requested to distinguish “never arrived” from “dropped in-pod.” This step narrowed the actionable causes to two scenarios: queue discard on capacity / policy, or transient in-memory loss during shutdown/eviction.
Step 5: Implement durable handoff and reconciliation (introducing the fix)
With the evidence pointing to in-memory SEDA semantics as the root risk, a two-part remediation was implemented: replace the in-memory seda:// handoff with a persistent broker-backed route, and add an early “accepted” EDR plus reconciliation monitoring. The change replaced the internal seda route with an Apache Camel Kafka producer/consumer pair (using the Camel Kafka component and a dedicated topic for async notifications), and the gateway started emitting an accepted EDR at REST_RECEIVE so reconciliation jobs can re-enqueue any accepted-but-not-dispatched notifications. Monitoring was added to detect enqueue vs dequeue mismatches.
Solution:
The implemented solution replaced the in-memory Apache Camel seda://asyncNotificationProcessor handoff with a persistent Camel Kafka route (publishing to a durable topic) and introduced an accepted EDR at REST_RECEIVE plus a reconciliation runner. Architecturally, using Apache Camel with the Kafka component moves the durability responsibility to the broker (persistence, consumer offsets, retries, and retention), eliminating the single-JVM, in-memory single-point-of-loss. The accepted EDR provides a durable ledger for reconciliation when transient delivery gaps occur.
Conclusion:
After switching the intra-gateway handoff to a persistent broker and adding accepted-EDR reconciliation, follow-up verification showed no further missing downstream EDRs in the test window. The change removed the risk of silent loss due to in-memory SEDA behavior, improved end-to-end delivery reliability, and added observable reconciliation so future accepted-but-not-dispatched cases are detected and replayable.