Recovering a MariaDB Galera Cluster on Kubernetes Stuck in CrashLoopBackOff

Problem

The database went completely offline after the Kubernetes worker nodes restarted at the same time. A three-node MariaDB 10.5 Galera cluster, deployed with the Bitnami mariadb-galera Helm chart, did not come back. The first pod, mariadb-galera-0, entered CrashLoopBackOff and logged It may not be safe to bootstrap the cluster from this node. It was not the last one to leave the cluster and may not contain all the updates. The chart added It is not safe to bootstrap form this node ('safe_to_bootstrap=0' is set in 'grastate.dat'). Pods 1 and 2 never started at all, and the application lost its database.

The tempting response was to force the cluster up: scale the StatefulSet to zero and back, delete pods, or set safe_to_bootstrap: 1 on whichever node came first. That only hides the issue. Bootstrapping from the wrong node silently discards the latest committed writes, and a forced flag left in place can split the cluster on the next restart.

Process

Step 1: Read the bootstrap state

The team had been checking /var/lib/mysql/grastate.dat, which does not exist in the Bitnami image. The real file sits at /bitnami/mariadb/data/grastate.dat. Mounting each PVC in a temporary helper pod showed the same state on all three volumes: safe_to_bootstrap: 0 and seqno: -1. Galera sets the safe flag only on the last node to leave after a graceful shutdown. Because every node stopped at once, no node qualified.

Step 2: Find the startup deadlock

The StatefulSet used the default podManagementPolicy: OrderedReady, so pods 1 and 2 wait for pod 0 to become ready. Pod 0 could not start because it was not safe to bootstrap. The cluster could never form on its own, no matter how many times it was restarted. All three replicas also ran on the same pair of hosts, which is why one host event stopped them all.

Step 3: Identify the most advanced node

With seqno: -1 everywhere, the last committed position had to be recovered from each node’s InnoDB data. With the StatefulSet scaled to zero, each volume was checked in a helper pod:

mysqld --wsrep-recover --datadir=/bitnami/mariadb/data
grep "WSREP: Recovered position" /opt/bitnami/mariadb/logs/mysqld.log

The number after the colon in Recovered position <uuid>:<seqno> is the node’s seqno. Node 2 held the highest value, so it, not node 0, had to bootstrap the cluster. This confirmed a recovery-order problem rather than data corruption.

Solution

The fix bootstrapped the cluster once, deliberately, from the node with the latest data instead of forcing up whichever pod Kubernetes started first. It then changed the deployment so a simultaneous restart can no longer deadlock the cluster.

Immediate fix

• Set safe_to_bootstrap: 1 in grastate.dat on node 2 only, then bootstrap from it with a one-time upgrade: helm upgrade db bitnami/mariadb-galera --reuse-values --set galera.bootstrap.bootstrapFromNode=2 --set galera.bootstrap.forceSafeToBootstrap=true --set podManagementPolicy=Parallel.
• Let nodes 0 and 1 rejoin through state transfer, then verify wsrep_cluster_size = 3 and wsrep_local_state_comment = Synced on every node.
• Remove bootstrapFromNode and forceSafeToBootstrap from the values right away, so a later restart cannot re-bootstrap node 2 and split the cluster.

Exit and prevention

• Keep podManagementPolicy: Parallel, so one unhealthy pod cannot block the others from starting.
• Add pod anti-affinity to place each replica on a different Kubernetes node.
• Replace “scale to zero” with a graceful shutdown: scale down to one replica first, so a known node is marked safe to bootstrap.
• Alert on wsrep_cluster_size and wsrep_ready to catch a degraded cluster before it becomes a full outage.

Conclusion

The MariaDB Galera cluster is back to three synced nodes, recovered from the node with the latest committed transactions. The incident showed that a Galera outage is often a recovery-order problem, not a database fault: every node was healthy, but none was allowed to start first. Restarts now follow a tested runbook, and the deployment no longer depends on which pod Kubernetes happens to start first.