Fixing Proxmox VE Kernel Regression Behind MariaDB I/O Failures

Problem

The database stopped working after a routine host upgrade. Once the Proxmox VE hosts moved to version 9.2.3 with the new default kernel 7.0.6-2-pve, a Debian virtual machine running MariaDB 10.11 began receiving hard I/O errors (EIO) on its virtual disk. The guest kernel logged write failures such as I/O error, dev sdb, sector 52428856 op 0x1:(WRITE) and SCSI “Aborted Command” / “I/O process terminated” errors. XFS then reported log I/O error -5 and shut the data filesystem down, and InnoDB failed during crash recovery and startup. A later attempt to restore the database failed the same way.

The tempting response was to treat this as database corruption: restart the VM, force InnoDB recovery, restore from backup after every failure. That only hides the issue. The failure sat below MariaDB, so every restore landed on the same broken I/O path.

Process

Step 1: Check the physical storage

The host stack was mdadm RAID 1 → LUKS → LVM-thin. It showed no disk, mdadm, LUKS or LVM errors, and SMART data was clean. The same VMs on the same storage had run without errors on the previous 6.17 kernel. This pointed away from hardware and toward the virtualization layer.

Step 2: Isolate the I/O path

The VM disk used virtio-scsi-single with cache=none and iothread=1. Switching the AIO backend from io_uring to threads did not stop the errors while the host stayed on kernel 7.0. The same failure then appeared on a second, unrelated host with different hardware, storage and workload. The only common factor was the 7.0 host kernel.

Step 3: Reproduce under load

A random-write stress test inside the guest triggered the errors on demand:

fio --name=stress-write --ioengine=libaio --iodepth=64 --rw=randwrite \
    --bs=4k --direct=1 --size=20G --numjobs=8 --runtime=600 \
    --time_based --ramp_time=30 --group_reporting --filename=/data/fio-test

On kernel 7.0 the test failed within seconds with fio: io_u error ... Input/output error, and the VM needed a hard stop and start. The same test on 6.17.13-13-pve ran for 30 minutes without errors. This confirmed a host kernel regression rather than a MariaDB or storage problem.

Solution

The fix removed the faulty kernel from the I/O path instead of repairing the database again and again. It addressed the layer that was actually failing, without changes to the physical storage or the database configuration.

Immediate fix

• Pin the hosts to the last known good kernel as a temporary measure: proxmox-boot-tool kernel pin 6.17.13-13-pve, then reboot and verify with uname -r.
• Restore MariaDB only after the host runs the pinned kernel, then verify table integrity with mariadb-check --all-databases.

Exit and prevention

• Upgrade to the fixed kernel 7.0.14-19, validate it with the same fio test on one node, then remove the pin with proxmox-boot-tool kernel unpin.
• Keep the fio test as a regression gate for every new host kernel.
• Roll out kernel updates to one node first before updating the rest of the cluster.
• Alert on guest block-device errors and on QEMU failed_wr_operations to catch I/O failures before filesystems shut down.

Conclusion

MariaDB on Proxmox is stable again, with no further disk errors after the kernel rollback. The incident showed that a database failure is not always a database problem: InnoDB was only reporting what the virtual disk returned. Kernel updates now go through one test node and the fio check before reaching production.