{"id":4005,"date":"2016-03-02T11:32:09","date_gmt":"2016-03-02T17:32:09","guid":{"rendered":"http:\/\/icesquare.com\/wordpress\/?p=4005"},"modified":"2018-12-12T17:31:01","modified_gmt":"2018-12-12T23:31:01","slug":"centos-6-no-zfs-after-upgrading-the-kernel","status":"publish","type":"post","link":"https:\/\/icesquare.com\/wordpress\/centos-6-no-zfs-after-upgrading-the-kernel\/","title":{"rendered":"CentOS\/RHEL 6: No ZFS after upgrading the kernel"},"content":{"rendered":"

This article is mainly for CentOS 6, please visit here<\/a> for CentOS 7.<\/h3>\n

After I upgraded the CentOS \/ RHEL system to the latest kernel, the ZFS failed to start. The system was unable to load the ZFS module, i.e., I could not access my data. Here are some error messages I found on the system:<\/p>\n

\r\n#sudo zpool status\r\nThe ZFS modules are not loaded.\r\nTry running '\/sbin\/modprobe zfs' as root to load them.\r\n<\/pre>\n
\r\n#sudo \/sbin\/modprobe zfs\r\nFATAL: Error inserting zfs (\/lib\/modules\/2.6.32-573.7.1.el6.x86_64\/weak-updates\/zfs.ko): Unknown symbol in module, or unknown parameter (see dmesg)\r\n<\/pre>\n
\r\n#dmesg\r\nzfs: disagrees about version of symbol vn_openat\r\nzfs: Unknown symbol vn_openat\r\nzfs: disagrees about version of symbol taskq_dispatch_delay\r\nzfs: Unknown symbol taskq_dispatch_delay\r\nzfs: disagrees about version of symbol taskq_cancel_id\r\nzfs: Unknown symbol taskq_cancel_id\r\nzfs: disagrees about version of symbol vn_open\r\nzfs: Unknown symbol vn_open\r\nzfs: disagrees about version of symbol vn_remove\r\nzfs: Unknown symbol vn_remove\r\nzfs: disagrees about version of symbol taskq_dispatch_ent\r\nzfs: Unknown symbol taskq_dispatch_ent\r\nzfs: disagrees about version of symbol taskq_dispatch\r\nzfs: Unknown symbol taskq_dispatch\r\nzfs: disagrees about version of symbol system_taskq\r\nzfs: Unknown symbol system_taskq\r\nzfs: disagrees about version of symbol taskq_wait\r\nzfs: Unknown symbol taskq_wait\r\nzfs: Unknown symbol __cv_wait_interruptible\r\nzfs: disagrees about version of symbol taskq_wait_id\r\nzfs: Unknown symbol taskq_wait_id\r\nzfs: disagrees about version of symbol taskq_destroy\r\nzfs: Unknown symbol taskq_destroy\r\nzfs: disagrees about version of symbol vn_rdwr\r\nzfs: Unknown symbol vn_rdwr\r\nzfs: disagrees about version of symbol taskq_init_ent\r\nzfs: Unknown symbol taskq_init_ent\r\nzfs: disagrees about version of symbol taskq_create\r\nzfs: Unknown symbol taskq_create\r\nzfs: Unknown symbol __cv_timedwait_interruptible\r\nzfs: disagrees about version of symbol taskq_member\r\nzfs: Unknown symbol taskq_member\r\n<\/pre>\n
So what does these messages mean? Before I explain the details, let me explain how ZFS works on Linux. For legal reasons, unlike *BSD, Linux kernel does not support ZFS. In order to make Linux talks to ZFS, some people came up a very smart way: They inject the ZFS library at the kernel level, such that when Linux boots, it knows how to handle the ZFS. It sounds pretty ideal, isn’t it?<\/p>\n
And now, we have a problem.<\/p>\n
Many system administrators like to let the system upgrade automatically (such as running yum update -y in the cron job etc). Unlike *BSD, Linux bundles the kernel and application update together. In the other words, when you run the yum update, it will update both kernel and applications together, and there is no way for you to pick one and skip the other.<\/p>\n
When the system upgrades the kernel, it refreshes everything, i.e., the new kernel will not know what is ZFS, because the process of injecting the ZFS happens when we install the ZFS on Linux. If there is no new version available, this process will not happen. So what happen after you reboot the computer, which by default, load the latest kernel? You got it, the ZFS won’t be loaded and your data is not accessible.<\/p>\n
There are few ways to handle this. First, if you really want to keep your system up to dated (which I don’t recommend), exclude the kernel from the system update.<\/p>\n
\r\nsudo nano \/etc\/yum.conf\r\n<\/pre>\n
\r\n[main]\r\n.....\r\nexclude=kernel*\r\n<\/pre>\n
It doesn’t mean your system is 100% safe from now on. You may still get some chances to break your ZFS. Here is some funny messages after I turn on the exclusion and run the yum update:<\/p>\n
\r\nLoading new zfs-0.6.5.4 DKMS files...\r\nBuilding for 2.6.32-504.23.4.el6.x86_64\r\nBuilding initial module for 2.6.32-504.23.4.el6.x86_64\r\nDone.\r\n\r\nAdding any weak-modules\r\nERROR: modinfo: could not open \/lib\/modules\/2.6.32-358.el6.x86_64\/weak-updates\/: Is a directory\r\nERROR: modinfo: could not open \/lib\/modules\/2.6.32-504.23.4.el6.x86_64\/zavl.ko: No such file or directory\r\nFATAL: \/lib\/modules\/2.6.32-504.23.4.el6.x86_64\/zavl.ko: No such file or directory\r\nWarning: Module zavl.ko from kernel  has no modversions, so it cannot be reused for kernel 2.6.32-358.el6.x86_64\r\nERROR: modinfo: could not open \/lib\/modules\/2.6.32-358.el6.x86_64\/weak-updates\/: Is a directory\r\nERROR: modinfo: could not open \/lib\/modules\/2.6.32-504.23.4.el6.x86_64\/znvpair.ko: No such file or directory\r\nFATAL: \/lib\/modules\/2.6.32-504.23.4.el6.x86_64\/znvpair.ko: No such file or directory\r\nWarning: Module znvpair.ko from kernel  has no modversions, so it cannot be reused for kernel 2.6.32-358.el6.x86_64\r\nERROR: modinfo: could not open \/lib\/modules\/2.6.32-358.el6.x86_64\/weak-updates\/: Is a directory\r\nERROR: modinfo: could not open \/lib\/modules\/2.6.32-504.23.4.el6.x86_64\/zunicode.ko: No such file or directory\r\nFATAL: \/lib\/modules\/2.6.32-504.23.4.el6.x86_64\/zunicode.ko: No such file or directory\r\nWarning: Module zunicode.ko from kernel  has no modversions, so it cannot be reused for kernel 2.6.32-358.el6.x86_64\r\nERROR: modinfo: could not open \/lib\/modules\/2.6.32-358.el6.x86_64\/weak-updates\/: Is a directory\r\nERROR: modinfo: could not open \/lib\/modules\/2.6.32-504.23.4.el6.x86_64\/zcommon.ko: No such file or directory\r\nFATAL: \/lib\/modules\/2.6.32-504.23.4.el6.x86_64\/zcommon.ko: No such file or directory\r\nWarning: Module zcommon.ko from kernel  has no modversions, so it cannot be reused for kernel 2.6.32-358.el6.x86_64\r\nERROR: modinfo: could not open \/lib\/modules\/2.6.32-358.el6.x86_64\/weak-updates\/: Is a directory\r\nERROR: modinfo: could not open \/lib\/modules\/2.6.32-504.23.4.el6.x86_64\/zpios.ko: No such file or directory\r\nFATAL: \/lib\/modules\/2.6.32-504.23.4.el6.x86_64\/zpios.ko: No such file or directory\r\nWarning: Module zpios.ko from kernel  has no modversions, so it cannot be reused for kernel 2.6.32-358.el6.x86_64\r\n\r\ndepmod...\r\n\r\nDKMS: install completed.\r\n<\/pre>\n
The second thing you will need to do is to increase the \/boot partition from the default 200MB to at least 2GB. By default, RHEL will create a 200MB \/boot for storing the kernel files. Kernels are small and they rarely go beyond 40MB. However, RHEL will only keep up to 5 recent kernels (40MB x 5 = 200MB), and it will remove the rest. So what happen if it removes the one that works with ZFS? The only thing you can do is to reinstall the system and import your ZFS again. <\/p>\n
\r\nsudo zpool import\r\n<\/pre>\n
Here is how to modify the number:<\/p>\n
\r\nsudo nano \/etc\/yum.conf \r\n<\/pre>\n
\r\n#Tell the system to keep the most 20 recent kernels\r\ninstallonly_limit=20\r\n\r\n<\/pre>\n
Another thing you may want to do is to select the working kernel (instead of the latest) one when boot. Here is how to change it:<\/p>\n
\r\nsudo nano \/boot\/grub\/grub.conf\r\n<\/pre>\n
Notice that I comment out the most recent kernels:<\/p>\n
\r\n# grub.conf generated by anaconda\r\n#\r\n# Note that you do not have to rerun grub after making changes to this file\r\n# NOTICE:  You have a \/boot partition.  This means that\r\n#          all kernel and initrd paths are relative to \/boot\/, eg.\r\n#          root (hd0,0)\r\n#          kernel \/vmlinuz-version ro root=\/dev\/sda3\r\n#          initrd \/initrd-[generic-]version.img\r\n#boot=\/dev\/sda\r\ndefault=0\r\ntimeout=5\r\nsplashimage=(hd0,0)\/grub\/splash.xpm.gz\r\nhiddenmenu\r\n#title CentOS (2.6.32-573.7.1.el6.x86_64)\r\n#       root (hd0,0)\r\n#       kernel \/vmlinuz-2.6.32-573.7.1.el6.x86_64 ro root=UUID=325cc438-33a6-46ae-8f1a-443ebd77c70a rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M  KEYBOARDTYPE=pc$\r\n#       initrd \/initramfs-2.6.32-573.7.1.el6.x86_64.img\r\n#title CentOS (2.6.32-573.8.1.el6.x86_64)\r\n#       root (hd0,0)\r\n#       kernel \/vmlinuz-2.6.32-573.8.1.el6.x86_64 ro root=UUID=325cc438-33a6-46ae-8f1a-443ebd77c70a rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M  KEYBOARDTYPE=pc$\r\n#       initrd \/initramfs-2.6.32-573.8.1.el6.x86_64.img\r\n#title CentOS (2.6.32-573.12.1.el6.x86_64)\r\n#       root (hd0,0)\r\n#       kernel \/vmlinuz-2.6.32-573.12.1.el6.x86_64 ro root=UUID=325cc438-33a6-46ae-8f1a-443ebd77c70a rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M  KEYBOARDTYPE=p$\r\n#       initrd \/initramfs-2.6.32-573.12.1.el6.x86_64.img\r\n#title CentOS (2.6.32-573.18.1.el6.x86_64)\r\n#       root (hd0,0)\r\n#       kernel \/vmlinuz-2.6.32-573.18.1.el6.x86_64 ro root=UUID=325cc438-33a6-46ae-8f1a-443ebd77c70a rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M  KEYBOARDTYPE=p$\r\n#       initrd \/initramfs-2.6.32-573.18.1.el6.x86_64.img\r\ntitle CentOS (2.6.32-573.3.1.el6.x86_64)\r\n        root (hd0,0)\r\n        kernel \/vmlinuz-2.6.32-573.3.1.el6.x86_64 ro root=UUID=325cc438-33a6-46ae-8f1a-443ebd77c70a rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M  KEYBOARDTYPE=pc$\r\n        initrd \/initramfs-2.6.32-573.3.1.el6.x86_64.img\r\ntitle CentOS (2.6.32-573.1.1.el6.x86_64)\r\n        root (hd0,0)\r\n        kernel \/vmlinuz-2.6.32-573.1.1.el6.x86_64 ro root=UUID=325cc438-33a6-46ae-8f1a-443ebd77c70a rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M  KEYBOARDTYPE=pc$\r\n        initrd \/initramfs-2.6.32-573.1.1.el6.x86_64.img\r\ntitle CentOS (2.6.32-504.30.3.el6.x86_64)\r\n        root (hd0,0)\r\n        kernel \/vmlinuz-2.6.32-504.30.3.el6.x86_64 ro root=UUID=325cc438-33a6-46ae-8f1a-443ebd77c70a rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M  KEYBOARDTYPE=p$\r\n        initrd \/initramfs-2.6.32-504.30.3.el6.x86_64.img\r\n<\/pre>\n
Do not bother to remove the ZFS libraries and reinstall them again. It won’t work and it will make you system only more messy.<\/p>\n
That’s it! Hope this tutorial saves you from heart attack.<\/p>\n
–Derrick<\/p>\n
 <\/div>\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
This article is mainly for CentOS 6, please visit here for CentOS 7. After I upgraded the CentOS \/ RHEL system to the latest kernel, the ZFS failed to start. Continue Reading →<\/a><\/p>\n","protected":false},"author":1,"featured_media":1260,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":true,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_hide_featured_image":false,"_jetpack_featured_image_duplicate_dismissed":0,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[94,144,6,69],"tags":[48,177],"class_list":["post-4005","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-computer-problems-computer-help-pc-support","category-sever-problem","category-technology","category-troubleshooting-technology","tag-linux","tag-zfs"],"jetpack_publicize_connections":[],"jetpack_shortlink":"https:\/\/wp.me\/p1sqzr-12B","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":4436,"url":"https:\/\/icesquare.com\/wordpress\/how-to-fix-the-mess-created-by-zfs-on-linux-after-update\/","url_meta":{"origin":4005,"position":0},"title":"“ZFS on Linux”: The ZFS modules are not loaded. Try running ‘\/sbin\/modprobe zfs’ as root to load them.","author":"Derrick","date":"April 10, 2018","format":false,"excerpt":"Last Updated: Oct 10, 2021 This article is based on my experience with CentOS 7. If you are running other Linux distributions, please adjust the commands and package names accordingly (e.g., yum --> apt-get). As of Oct 3, 2019, I cannot get ZFS on Linux running on CentOS 8. ZFS\u2026","rel":"","context":"In "Computer Problems"","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4202,"url":"https:\/\/icesquare.com\/wordpress\/centosrhel-7-no-zfs-after-updating-the-kernel\/","url_meta":{"origin":4005,"position":1},"title":"CentOS\/RHEL 7 – No ZFS After Updating the Kernel","author":"Derrick","date":"December 12, 2016","format":false,"excerpt":"Let's all agree with this fact: ZFS is foreign to Linux. It is not native. You can't expect that ZFS on Linux will run smoothly as FreeBSD or Solaris. Having using ZFS on Linux since 2013 (and ZFS on FreeBSD since 2009), I've noticed that ZFS does not like Linux\u2026","rel":"","context":"In "Computer Problems"","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4286,"url":"https:\/\/icesquare.com\/wordpress\/how-to-install-zfs-on-rhel-centos-7\/","url_meta":{"origin":4005,"position":2},"title":"How to install ZFS on RHEL \/ CentOS 7","author":"Derrick","date":"January 4, 2017","format":false,"excerpt":"A friend of mine likes to try ZFS on CentOS 7, therefore I decide to make a guide for him. The following instructions have been well tested on CentOS 7. Before you decide to put ZFS in a production use, you should be aware of the following: ZFS is originally\u2026","rel":"","context":"In "Everyday Computer Problem"","block_context":{"text":"Everyday Computer Problem","link":"https:\/\/icesquare.com\/wordpress\/category\/everyday-computer-problem\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2011\/01\/zfs_logo.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":4137,"url":"https:\/\/icesquare.com\/wordpress\/modprobe-error-could-not-insert-zfs-required-key-not-available\/","url_meta":{"origin":4005,"position":3},"title":"modprobe: ERROR: could not insert ‘zfs’: Required key not available","author":"Derrick","date":"June 10, 2016","format":false,"excerpt":"Today I was trying to install ZFS on a CentOS 7 box. Typically rebooting the computer, the ZFS mododule will be turned on. However, it didn't turn on in my case. Failed to load ZFS module stack. Load the module manually by running 'insmod \/zfs.ko' as root. So I tried\u2026","rel":"","context":"In "Computer Problems"","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2011\/01\/zfs_logo.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":3914,"url":"https:\/\/icesquare.com\/wordpress\/zfs-linux-vs-freebsd\/","url_meta":{"origin":4005,"position":4},"title":"Running ZFS on Linux: Things you should know and be aware of","author":"Derrick","date":"February 29, 2016","format":false,"excerpt":"First Published: Feb 29, 2016 Last Revised: January 10, 2026 ZFS is a next-generation file system renowned for its reliability and advanced features. However, due to legal and licensing issues, it is not included by default in most Linux distributions. Fortunately, you can still install (ZFS on Linux) with just\u2026","rel":"","context":"In "Computer Problems"","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4679,"url":"https:\/\/icesquare.com\/wordpress\/zfs-on-linux-trouble-this-pool-uses-the-following-features-not-supported-by-this-system-all-unsupported-features-are-only-required-for-writing-to-the-pool\/","url_meta":{"origin":4005,"position":5},"title":"[ZFS On Linux Trouble] This pool uses the following feature(s) not supported by this system…All unsupported features are only required for writing to the pool, zpool create: invalid argument for this pool operation","author":"Derrick","date":"February 11, 2020","format":false,"excerpt":"When I rebooted my computer and loaded my ZFS pool today, I got this error message: #sudo zpool import -a This pool uses the following feature(s) not supported by this system: org.zfsonlinux:project_quota (space\/object accounting based on project ID.) com.delphix:spacemap_v2 (Space maps representing large segments are more efficient.) All unsupported features\u2026","rel":"","context":"In "Technology"","block_context":{"text":"Technology","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_featured_media_url":"https:\/\/icesquare.com\/wordpress\/wp-content\/uploads\/2011\/01\/zfs_logo.jpg","_links":{"self":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/4005","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/comments?post=4005"}],"version-history":[{"count":21,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/4005\/revisions"}],"predecessor-version":[{"id":4533,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/4005\/revisions\/4533"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/media\/1260"}],"wp:attachment":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/media?parent=4005"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/categories?post=4005"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/tags?post=4005"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}