{"id":4202,"date":"2016-12-12T17:13:04","date_gmt":"2016-12-12T23:13:04","guid":{"rendered":"http:\/\/icesquare.com\/wordpress\/?p=4202"},"modified":"2017-05-24T18:07:49","modified_gmt":"2017-05-24T23:07:49","slug":"centosrhel-7-no-zfs-after-updating-the-kernel","status":"publish","type":"post","link":"https:\/\/icesquare.com\/wordpress\/centosrhel-7-no-zfs-after-updating-the-kernel\/","title":{"rendered":"CentOS\/RHEL 7 &#8211; No ZFS After Updating the Kernel"},"content":{"rendered":"<p>Let&#8217;s all agree with this fact: ZFS is foreign to Linux. It is not native. You can&#8217;t expect that ZFS on Linux will run smoothly as FreeBSD or Solaris. Having using ZFS on Linux since 2013 (and ZFS on FreeBSD since 2009), I&#8217;ve noticed that ZFS does not like Linux (well, at least RHEL 7). Here are some few examples:<\/p>\n<ul>\n<li>ZFS is not loaded at the boot time. You will need to manually start it or load it via cron. Good luck if you have other services (like Apache, MySQL, NFS, or even users&#8217; home directories) that depend on the ZFS.<\/li>\n<li>Every single time you update the kernel, ZFS will not work after the reboot without some manual work. What if the system runs the update automatically, and one day there is a power failure which makes your server to reboot to a new kernel? Your system will not be able to mount your ZFS volume. If you integrate ZFS with other service applications such as web, database or network drive, oh well, good luck and I hope you will catch this problem fast enough before receiving thousands of emails and calls from your end-users.\n<\/li>\n<li>If you exclude the kernel from the updates (\/etc\/yum.conf), you will eventually run into trouble, because there are tons of other packages that require the latest kernel. In the other words, running the command: <strong>yum update -y<\/strong> will fail. You will need to run <strong>yum update &#8211;skip-broken<\/strong>, which means you will miss many latest packages. Here is an example:\n<pre>\r\n--> Finished Dependency Resolution\r\nError: Package: hypervvssd-0-0.29.20160216git.el7.x86_64 (base)\r\n           Requires: kernel >= 3.10.0-384.el7\r\n           Installed: kernel-3.10.0-327.el7.x86_64 (@anaconda)\r\n               kernel = 3.10.0-327.el7\r\n           Installed: kernel-3.10.0-327.22.2.el7.x86_64 (@updates)\r\n               kernel = 3.10.0-327.22.2.el7\r\nError: Package: hypervfcopyd-0-0.29.20160216git.el7.x86_64 (base)\r\n           Requires: kernel >= 3.10.0-384.el7\r\n           Installed: kernel-3.10.0-327.el7.x86_64 (@anaconda)\r\n               kernel = 3.10.0-327.el7\r\n           Installed: kernel-3.10.0-327.22.2.el7.x86_64 (@updates)\r\n               kernel = 3.10.0-327.22.2.el7\r\nError: Package: hypervkvpd-0-0.29.20160216git.el7.x86_64 (base)\r\n           Requires: kernel >= 3.10.0-384.el7\r\n           Installed: kernel-3.10.0-327.el7.x86_64 (@anaconda)\r\n               kernel = 3.10.0-327.el7\r\n           Installed: kernel-3.10.0-327.22.2.el7.x86_64 (@updates)\r\n               kernel = 3.10.0-327.22.2.el7\r\n You could try using --skip-broken to work around the problem\r\n You could try running: rpm -Va --nofiles --nodigest\r\n<\/pre>\n<\/li>\n<li>If you are running the stable Linux distributions like RHEL 7, you can load a more recent kernel like 4.x by installing the package: kernel-ml. However, don&#8217;t expect that ZFS will work with version 4:\n<pre>\r\nLoading new spl-0.6.5.9 DKMS files...\r\nBuilding for 4.11.2-1.el7.elrepo.x86_64\r\nBuilding initial module for 4.11.2-1.el7.elrepo.x86_64\r\nconfigure: error: unknown\r\nError! Bad return status for module build on kernel: 4.11.2-1.el7.elrepo.x86_64 (x86_64)\r\nConsult \/var\/lib\/dkms\/spl\/0.6.5.9\/build\/make.log for more information.\r\n\r\n<\/pre>\n<\/li>\n<\/ul>\n<p>Running ZFS on Linux is like putting a giraffe in the wild in Alaska. It is just not the right thing to do. Unfortunately, there are so many things that only available on Linux so we have to live with it. Just like FUSE (Filesystem in Userspace), many people feel hesitated to run their file systems on the userspace instead of kernel level, but hey, see how many people are happy with GlusterFS, a distributed file system that live on FUSE! Personally I just think it is not a right thing to do, especially in an enterprise environment. Running a production file system at the userspace level, seriously?<\/p>\n<p>Anyway, if you are running into trouble after upgrading your Linux kernel (and you almost had a heart attack when you think your data may be lost), you have two choices:<\/p>\n<ol>\n<li>Simply boot to the previous working kernel if you need to get your data back in quick. However, keep in mind that this will create two problems:\n<ul>\n<li>Since you already update the system with the new kernel and the new packages, your new packages probably will not work with the old kernel, and that may give you extra headache.<\/li>\n<li>Unless you manually overwrite the kernel boot order (boot loader config), otherwise you may get into the same trouble in the next boot.<\/li>\n<\/ul>\n<\/li>\n<li>If you want a more &#8220;permanent&#8221; fix, you will need to rebuild the dkms ZFS and SPL modules. See below for the instructions. Keep in mind that you will have the same problem again when the kernel receives a new update.<\/li>\n<\/ol>\n<p>You&#8217;ve tried to load the ZFS and realize that it is no longer available:<\/p>\n<pre>\r\n#sudo zpool import\r\nThe ZFS modules are not loaded.\r\nTry running '\/sbin\/modprobe zfs' as root to load them.\r\n\r\n#sudo \/sbin\/modprobe zfs\r\nmodprobe: FATAL: Module zfs not found.\r\n<\/pre>\n<p>You may want to check the dkms status. Write down the version number. In my case, it is <strong>0.6.5.9<\/strong><\/p>\n<pre>\r\n#sudo dkms status\r\nspl, 0.6.5.9, 3.10.0-327.28.3.el7.x86_64, x86_64: installed (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!)\r\nspl, 0.6.5.9, 3.10.0-514.2.2.el7.x86_64, x86_64: installed (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!)\r\nzfs, 0.6.5.9, 3.10.0-327.28.3.el7.x86_64, x86_64: installed (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!)\r\n<\/pre>\n<p>Before running the following commands, make sure that you know what you are doing.<\/p>\n<pre>\r\n\r\n#Make sure that you reboot to the kernel you want to fix.\r\n#Find out what is the current kernel\r\nuname -a\r\nLinux 3.10.0-514.2.2.el7.x86_64 #1 SMP Tue Dec 6 23:06:41 UTC 2016 x86_64 x86_64 x86_64 GNU\/Linux\r\n\r\n#In my example, it is:\r\n3.10.0-514.2.2.el7.x86_64\r\n\r\n#Now, let's get into the fun part. We will remove them and reinstall them.\r\n#Don't forget to match your version, in my base, my version is: 0.6.5.9\r\nsudo dkms remove zfs\/0.6.5.9 --all\r\nsudo dkms remove spl\/0.6.5.9 --all\r\nsudo dkms --force install spl\/0.6.5.9\r\nsudo dkms --force install zfs\/0.6.5.9\r\n\r\n#or you can run these commands in one line, so that you don't need to wait:\r\nsudo dkms remove zfs\/0.6.5.9 --all; sudo dkms remove spl\/0.6.5.9 --all; sudo dkms --force install spl\/0.6.5.9; sudo dkms --force install zfs\/0.6.5.9;\r\n<\/pre>\n<p>And we will verify the result.<\/p>\n<pre>\r\n#sudo dkms status\r\nspl, 0.6.5.9, 3.10.0-514.2.2.el7.x86_64, x86_64: installed\r\nzfs, 0.6.5.9, 3.10.0-514.2.2.el7.x86_64, x86_64: installed\r\n<\/pre>\n<p>Finally we can start the ZFS again.<\/p>\n<pre>\r\nsudo \/sbin\/modprobe zfs\r\n<\/pre>\n<p>Your ZFS pool should back. You can verify it by rebooting your machine. Notice that Linux may not automatically mount the ZFS volumes. You may want to mount it manually or via cron job.<\/p>\n<p>Here is how to mount the ZFS volumes manually.<\/p>\n<pre>\r\nsudo zpool import -a\r\n<\/pre>\n<p>You may want to remove all of the old kernels too.<\/p>\n<pre>\r\nsudo package-cleanup --oldkernels --count=1 -y\r\n<\/pre>\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Let&#8217;s all agree with this fact: ZFS is foreign to Linux. It is not native. You can&#8217;t expect that ZFS on Linux will run smoothly as FreeBSD or Solaris. Having <a class=\"more-link\" href=\"https:\/\/icesquare.com\/wordpress\/centosrhel-7-no-zfs-after-updating-the-kernel\/\">Continue Reading &rarr;<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":true,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_hide_featured_image":false,"_jetpack_featured_image_duplicate_dismissed":0,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[94,146,144,6,69,109],"tags":[48,177],"class_list":["post-4202","post","type-post","status-publish","format-standard","hentry","category-computer-problems-computer-help-pc-support","category-everyday-computer-problem","category-sever-problem","category-technology","category-troubleshooting-technology","category-zfs","tag-linux","tag-zfs"],"jetpack_publicize_connections":[],"jetpack_shortlink":"https:\/\/wp.me\/p1sqzr-15M","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":3914,"url":"https:\/\/icesquare.com\/wordpress\/zfs-linux-vs-freebsd\/","url_meta":{"origin":4202,"position":0},"title":"Running ZFS on Linux: Things you should know and be aware of","author":"Derrick","date":"February 29, 2016","format":false,"excerpt":"First Published: Feb 29, 2016 Last Revised: January 10, 2026 ZFS is a next-generation file system renowned for its reliability and advanced features. However, due to legal and licensing issues, it is not included by default in most Linux distributions. Fortunately, you can still install (ZFS on Linux) with just\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4286,"url":"https:\/\/icesquare.com\/wordpress\/how-to-install-zfs-on-rhel-centos-7\/","url_meta":{"origin":4202,"position":1},"title":"How to install ZFS on RHEL \/ CentOS 7","author":"Derrick","date":"January 4, 2017","format":false,"excerpt":"A friend of mine likes to try ZFS on CentOS 7, therefore I decide to make a guide for him. The following instructions have been well tested on CentOS 7. Before you decide to put ZFS in a production use, you should be aware of the following: ZFS is originally\u2026","rel":"","context":"In &quot;Everyday Computer Problem&quot;","block_context":{"text":"Everyday Computer Problem","link":"https:\/\/icesquare.com\/wordpress\/category\/everyday-computer-problem\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2011\/01\/zfs_logo.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":4436,"url":"https:\/\/icesquare.com\/wordpress\/how-to-fix-the-mess-created-by-zfs-on-linux-after-update\/","url_meta":{"origin":4202,"position":2},"title":"&#8220;ZFS on Linux&#8221;: The ZFS modules are not loaded. Try running &#8216;\/sbin\/modprobe zfs&#8217; as root to load them.","author":"Derrick","date":"April 10, 2018","format":false,"excerpt":"Last Updated: Oct 10, 2021 This article is based on my experience with CentOS 7. If you are running other Linux distributions, please adjust the commands and package names accordingly (e.g., yum --> apt-get). As of Oct 3, 2019, I cannot get ZFS on Linux running on CentOS 8. ZFS\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4679,"url":"https:\/\/icesquare.com\/wordpress\/zfs-on-linux-trouble-this-pool-uses-the-following-features-not-supported-by-this-system-all-unsupported-features-are-only-required-for-writing-to-the-pool\/","url_meta":{"origin":4202,"position":3},"title":"[ZFS On Linux Trouble] This pool uses the following feature(s) not supported by this system&#8230;All unsupported features are only required for writing to the pool, zpool create: invalid argument for this pool operation","author":"Derrick","date":"February 11, 2020","format":false,"excerpt":"When I rebooted my computer and loaded my ZFS pool today, I got this error message: #sudo zpool import -a This pool uses the following feature(s) not supported by this system: org.zfsonlinux:project_quota (space\/object accounting based on project ID.) com.delphix:spacemap_v2 (Space maps representing large segments are more efficient.) All unsupported features\u2026","rel":"","context":"In &quot;Technology&quot;","block_context":{"text":"Technology","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4005,"url":"https:\/\/icesquare.com\/wordpress\/centos-6-no-zfs-after-upgrading-the-kernel\/","url_meta":{"origin":4202,"position":4},"title":"CentOS\/RHEL 6: No ZFS after upgrading the kernel","author":"Derrick","date":"March 2, 2016","format":false,"excerpt":"This article is mainly for CentOS 6, please visit here for CentOS 7. After I upgraded the CentOS \/ RHEL system to the latest kernel, the ZFS failed to start. The system was unable to load the ZFS module, i.e., I could not access my data. Here are some error\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2011\/01\/zfs_logo.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":1214,"url":"https:\/\/icesquare.com\/wordpress\/building-a-super-large-and-reliable-file-server-with-mixed-size-of-harddisks\/","url_meta":{"origin":4202,"position":5},"title":"Building a Super Large and Reliable File Server with Mixed Size of Harddisks","author":"Derrick","date":"January 17, 2011","format":false,"excerpt":"In this article, I am going to show you how to build a super large, ultra reliable, expandable file server with mixed size of hard drives. Here is my criteria: Large Storage Capacity. At this moment, the largest hard drive is 2TB. I am looking for at least 10TB. Great\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"\/wordpress\/wp-content\/uploads\/2011\/01\/zfs_logo-150x150.jpg","width":350,"height":200},"classes":[]}],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/4202","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/comments?post=4202"}],"version-history":[{"count":45,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/4202\/revisions"}],"predecessor-version":[{"id":4305,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/4202\/revisions\/4305"}],"wp:attachment":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/media?parent=4202"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/categories?post=4202"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/tags?post=4202"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}