This article is mainly for CentOS 6, please visit here for CentOS 7.
After I upgraded the CentOS / RHEL system to the latest kernel, the ZFS failed to start. The system was unable to load the ZFS module, i.e., I could not access my data. Here are some error messages I found on the system:
#sudo zpool status The ZFS modules are not loaded. Try running '/sbin/modprobe zfs' as root to load them.
#sudo /sbin/modprobe zfs FATAL: Error inserting zfs (/lib/modules/2.6.32-573.7.1.el6.x86_64/weak-updates/zfs.ko): Unknown symbol in module, or unknown parameter (see dmesg)
#dmesg zfs: disagrees about version of symbol vn_openat zfs: Unknown symbol vn_openat zfs: disagrees about version of symbol taskq_dispatch_delay zfs: Unknown symbol taskq_dispatch_delay zfs: disagrees about version of symbol taskq_cancel_id zfs: Unknown symbol taskq_cancel_id zfs: disagrees about version of symbol vn_open zfs: Unknown symbol vn_open zfs: disagrees about version of symbol vn_remove zfs: Unknown symbol vn_remove zfs: disagrees about version of symbol taskq_dispatch_ent zfs: Unknown symbol taskq_dispatch_ent zfs: disagrees about version of symbol taskq_dispatch zfs: Unknown symbol taskq_dispatch zfs: disagrees about version of symbol system_taskq zfs: Unknown symbol system_taskq zfs: disagrees about version of symbol taskq_wait zfs: Unknown symbol taskq_wait zfs: Unknown symbol __cv_wait_interruptible zfs: disagrees about version of symbol taskq_wait_id zfs: Unknown symbol taskq_wait_id zfs: disagrees about version of symbol taskq_destroy zfs: Unknown symbol taskq_destroy zfs: disagrees about version of symbol vn_rdwr zfs: Unknown symbol vn_rdwr zfs: disagrees about version of symbol taskq_init_ent zfs: Unknown symbol taskq_init_ent zfs: disagrees about version of symbol taskq_create zfs: Unknown symbol taskq_create zfs: Unknown symbol __cv_timedwait_interruptible zfs: disagrees about version of symbol taskq_member zfs: Unknown symbol taskq_member
So what does these messages mean? Before I explain the details, let me explain how ZFS works on Linux. For legal reasons, unlike *BSD, Linux kernel does not support ZFS. In order to make Linux talks to ZFS, some people came up a very smart way: They inject the ZFS library at the kernel level, such that when Linux boots, it knows how to handle the ZFS. It sounds pretty ideal, isn’t it?
And now, we have a problem.
Many system administrators like to let the system upgrade automatically (such as running yum update -y in the cron job etc). Unlike *BSD, Linux bundles the kernel and application update together. In the other words, when you run the yum update, it will update both kernel and applications together, and there is no way for you to pick one and skip the other.
When the system upgrades the kernel, it refreshes everything, i.e., the new kernel will not know what is ZFS, because the process of injecting the ZFS happens when we install the ZFS on Linux. If there is no new version available, this process will not happen. So what happen after you reboot the computer, which by default, load the latest kernel? You got it, the ZFS won’t be loaded and your data is not accessible.
There are few ways to handle this. First, if you really want to keep your system up to dated (which I don’t recommend), exclude the kernel from the system update.
sudo nano /etc/yum.conf
[main] ..... exclude=kernel*
It doesn’t mean your system is 100% safe from now on. You may still get some chances to break your ZFS. Here is some funny messages after I turn on the exclusion and run the yum update:
Loading new zfs-0.6.5.4 DKMS files... Building for 2.6.32-504.23.4.el6.x86_64 Building initial module for 2.6.32-504.23.4.el6.x86_64 Done. Adding any weak-modules ERROR: modinfo: could not open /lib/modules/2.6.32-358.el6.x86_64/weak-updates/: Is a directory ERROR: modinfo: could not open /lib/modules/2.6.32-504.23.4.el6.x86_64/zavl.ko: No such file or directory FATAL: /lib/modules/2.6.32-504.23.4.el6.x86_64/zavl.ko: No such file or directory Warning: Module zavl.ko from kernel has no modversions, so it cannot be reused for kernel 2.6.32-358.el6.x86_64 ERROR: modinfo: could not open /lib/modules/2.6.32-358.el6.x86_64/weak-updates/: Is a directory ERROR: modinfo: could not open /lib/modules/2.6.32-504.23.4.el6.x86_64/znvpair.ko: No such file or directory FATAL: /lib/modules/2.6.32-504.23.4.el6.x86_64/znvpair.ko: No such file or directory Warning: Module znvpair.ko from kernel has no modversions, so it cannot be reused for kernel 2.6.32-358.el6.x86_64 ERROR: modinfo: could not open /lib/modules/2.6.32-358.el6.x86_64/weak-updates/: Is a directory ERROR: modinfo: could not open /lib/modules/2.6.32-504.23.4.el6.x86_64/zunicode.ko: No such file or directory FATAL: /lib/modules/2.6.32-504.23.4.el6.x86_64/zunicode.ko: No such file or directory Warning: Module zunicode.ko from kernel has no modversions, so it cannot be reused for kernel 2.6.32-358.el6.x86_64 ERROR: modinfo: could not open /lib/modules/2.6.32-358.el6.x86_64/weak-updates/: Is a directory ERROR: modinfo: could not open /lib/modules/2.6.32-504.23.4.el6.x86_64/zcommon.ko: No such file or directory FATAL: /lib/modules/2.6.32-504.23.4.el6.x86_64/zcommon.ko: No such file or directory Warning: Module zcommon.ko from kernel has no modversions, so it cannot be reused for kernel 2.6.32-358.el6.x86_64 ERROR: modinfo: could not open /lib/modules/2.6.32-358.el6.x86_64/weak-updates/: Is a directory ERROR: modinfo: could not open /lib/modules/2.6.32-504.23.4.el6.x86_64/zpios.ko: No such file or directory FATAL: /lib/modules/2.6.32-504.23.4.el6.x86_64/zpios.ko: No such file or directory Warning: Module zpios.ko from kernel has no modversions, so it cannot be reused for kernel 2.6.32-358.el6.x86_64 depmod... DKMS: install completed.
The second thing you will need to do is to increase the /boot partition from the default 200MB to at least 2GB. By default, RHEL will create a 200MB /boot for storing the kernel files. Kernels are small and they rarely go beyond 40MB. However, RHEL will only keep up to 5 recent kernels (40MB x 5 = 200MB), and it will remove the rest. So what happen if it removes the one that works with ZFS? The only thing you can do is to reinstall the system and import your ZFS again.
sudo zpool import
Here is how to modify the number:
sudo nano /etc/yum.conf
#Tell the system to keep the most 20 recent kernels installonly_limit=20
Another thing you may want to do is to select the working kernel (instead of the latest) one when boot. Here is how to change it:
sudo nano /boot/grub/grub.conf
Notice that I comment out the most recent kernels:
# grub.conf generated by anaconda # # Note that you do not have to rerun grub after making changes to this file # NOTICE: You have a /boot partition. This means that # all kernel and initrd paths are relative to /boot/, eg. # root (hd0,0) # kernel /vmlinuz-version ro root=/dev/sda3 # initrd /initrd-[generic-]version.img #boot=/dev/sda default=0 timeout=5 splashimage=(hd0,0)/grub/splash.xpm.gz hiddenmenu #title CentOS (2.6.32-573.7.1.el6.x86_64) # root (hd0,0) # kernel /vmlinuz-2.6.32-573.7.1.el6.x86_64 ro root=UUID=325cc438-33a6-46ae-8f1a-443ebd77c70a rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M KEYBOARDTYPE=pc$ # initrd /initramfs-2.6.32-573.7.1.el6.x86_64.img #title CentOS (2.6.32-573.8.1.el6.x86_64) # root (hd0,0) # kernel /vmlinuz-2.6.32-573.8.1.el6.x86_64 ro root=UUID=325cc438-33a6-46ae-8f1a-443ebd77c70a rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M KEYBOARDTYPE=pc$ # initrd /initramfs-2.6.32-573.8.1.el6.x86_64.img #title CentOS (2.6.32-573.12.1.el6.x86_64) # root (hd0,0) # kernel /vmlinuz-2.6.32-573.12.1.el6.x86_64 ro root=UUID=325cc438-33a6-46ae-8f1a-443ebd77c70a rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M KEYBOARDTYPE=p$ # initrd /initramfs-2.6.32-573.12.1.el6.x86_64.img #title CentOS (2.6.32-573.18.1.el6.x86_64) # root (hd0,0) # kernel /vmlinuz-2.6.32-573.18.1.el6.x86_64 ro root=UUID=325cc438-33a6-46ae-8f1a-443ebd77c70a rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M KEYBOARDTYPE=p$ # initrd /initramfs-2.6.32-573.18.1.el6.x86_64.img title CentOS (2.6.32-573.3.1.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-573.3.1.el6.x86_64 ro root=UUID=325cc438-33a6-46ae-8f1a-443ebd77c70a rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M KEYBOARDTYPE=pc$ initrd /initramfs-2.6.32-573.3.1.el6.x86_64.img title CentOS (2.6.32-573.1.1.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-573.1.1.el6.x86_64 ro root=UUID=325cc438-33a6-46ae-8f1a-443ebd77c70a rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M KEYBOARDTYPE=pc$ initrd /initramfs-2.6.32-573.1.1.el6.x86_64.img title CentOS (2.6.32-504.30.3.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-504.30.3.el6.x86_64 ro root=UUID=325cc438-33a6-46ae-8f1a-443ebd77c70a rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M KEYBOARDTYPE=p$ initrd /initramfs-2.6.32-504.30.3.el6.x86_64.img
Do not bother to remove the ZFS libraries and reinstall them again. It won’t work and it will make you system only more messy.
That’s it! Hope this tutorial saves you from heart attack.
–Derrick
Our sponsors: