“ZFS on Linux”: The ZFS modules are not loaded. Try running ‘/sbin/modprobe zfs’ as root to load them.

Last Updated: Oct 3, 2019

This article is based on my experience with CentOS 7. If you are running other Linux distributions, please adjust the commands and package names accordingly (e.g., yum –> apt-get).

As of Oct 3, 2019, I cannot get ZFS on Linux running on CentOS 8.

ZFS on Linux is a poorly designed software solution to get ZFS up and running in Linux environments. Unlike FreeBSD, ZFS does not work with the Linux kernel natively. The developers of ZFS on Linux come up a crappy hack. By injecting the ZFS into the kernel via DKMS, Linux kernel will understand what is ZFS. It works very well, and it really works with a single assumption: The system will never be updated or rebooted after installing ZFS on Linux. So what will happen after you update the system (e.g., kernel, ZFS on Linux packages) and the system got rebooted? There is a good chance that your ZFS module will not be loaded. Otherwise you are not here, right?

Event What will happen after reboot? What do you need to do?
You update kernel first, then ZFS on Linux afterward
If you update before Dec 12, 2018: Your system will load the ZFS modules.
If you update after Dec 12, 2018: Probably not
Remove the old kernels from DKMS database. Rebuild the SPL and ZFS modules with the new kernel in the DKMS database.
You update ZFS on Linux first, then kernel afterward If your system boot into the new kernel (which is default), your system WILL NOT load the ZFS modules. Remove and install the ZFS and DKMS packages. Remove the old kernels from DKMS database. Rebuild the SPL and ZFS modules with the new kernel in the DKMS database.
You update ZFS on Linux only. Kernel has not been updated. Your system will load the ZFS modules. Remove the old kernels from DKMS database. Rebuild the SPL and ZFS modules with the new kernel in the DKMS database.
You update kernel only. ZFS on Linux has not been updated Your system will load the ZFS modules. Remove the old kernels from DKMS database. Rebuild the SPL and ZFS modules with the new kernel in the DKMS database.

There are two steps to rescue your data back. We will start with removing your DKMS module first. If it does not work, we will reinstall the ZFS packages. Also, I am assuming that your system is booted to the new kernel. Please keep in mind that ZFS on Linux does not work with Linux kernel v4 (either via kernel-ml or CentOS 8). It only works with v3.

If you need to access your data, the easiest way is to boot to the old working kernel. Once you are ready to clean up the mess created by the ZFS on Linux team, boot to the new kernel and follow my instructions below.

Step 1: Clean up and Reinstall DKMS Modules

Most of the time, the ZFS on Linux messes up the DKMS modules after the update. I suggest to clean up and reinstall DKMS modules once again. As of December 12, 2018, the ZFS on Linux will remove all of the DKMS modules for no reason.

First, check your DKMS status. You will need to clean up the DKMS if it is empty (nothing is installed), orphan (no kernel is installed) or multiple (multiple kernels installed). If it is clean (single kernel only), you may skip this step. If you are using ZFS on Linux ver 0.7.x, your DKMS will contain two modules (zfs and spl). If you are using ver. 0.8.x, your DKMS will contain one module only (zfs).

#dkms status
#An example of dirty DKMS status (This is bad):
spl, 0.7.12, 3.10.0-862.14.4.el7: installed (original_module exists) (WARNING! Diff between built and installed module!)
spl, 0.7.12, 3.10.0-957.1.3.el7: installed (original_module exists)
zfs, 0.7.12, 3.10.0-862.14.4.el7: installed (original_module exists) (WARNING! Diff between built and installed module!)
zfs, 0.7.12, 3.10.0-957.1.3.el7: installed (original_module exists)

#An example of empty DKMS status (This is bad):
(empty)

#An example of DKMS status without kernal (This is bad):
zfs, 0.7.12: added
spl, 0.7.12: added

#An example of clean DKMS status (This is good):
spl, 0.7.12, 3.10.0-957.1.3.el7.x86_64, x86_64: installed
zfs, 0.7.12, 3.10.0-957.1.3.el7.x86_64, x86_64: installed 

or 

spl, 0.7.12, 3.10.0-957.1.3.el7.x86_64, x86_64: installed (original_module exists)
zfs, 0.7.12, 3.10.0-957.1.3.el7.x86_64, x86_64: installed (original_module exists)

In my example above, my ZFS on Linux is 0.7.12, my old kernel is 3.10.0-862.14.4.el7, my new kernel is 3.10.0-957.1.3.el7. Your version may be different.

You may want to remove both ZFS and SPL DKMS modules first, then reinstall them:

#If your version is 0.7.x:
sudo dkms remove zfs/0.7.12 --all; 
sudo dkms remove spl/0.7.12 --all; 

#If your version is 0.8.x:
sudo dkms remove zfs/0.8.2 --all; 

Sometimes, you will need to remove the old kernel manually:

sudo dkms remove zfs/0.7.12 -k 3.10.0-862.14.4.el7.x86_64; 
sudo dkms remove spl/0.7.12 -k 3.10.0-862.14.4.el7.x86_64;

Time to reinstall them:

#If your version is 0.7.x:
sudo dkms --force install spl/0.7.12; 
sudo dkms --force install zfs/0.7.12;

#If your version is 0.8.x:
sudo dkms --force install zfs/0.8.2;

Run the DKMS status again. You should see both ZPL and SPL are attached to the new kernel:

#If your version is 0.7.x:
spl, 0.7.12, 3.10.0-957.1.3.el7.x86_64, x86_64: installed
zfs, 0.7.12, 3.10.0-957.1.3.el7.x86_64, x86_64: installed

#If your version is 0.8.x:
zfs, 0.8.2, 3.10.0-1062.1.2.el7.x86_64, x86_64: installed

Try to load the ZFS module and import your ZFS data:

sudo /sbin/modprobe zfs
sudo zpool import -a

If everything looks good, you can reboot your system and test to see if the ZFS is loaded automatically or not. Once everything is okay, remove the old kernel from the system.

sudo package-cleanup --oldkernels --count=1 -y

That’s it, you are good to go.


Step 2: Reinstall ZFS packages

If you have tried the first step and it didn’t work. You may want to reinstall the ZFS packages. Here is a typical error message:

You try to import the ZFS data and the system complains:

#zpool import -a
The ZFS modules are not loaded.
Try running '/sbin/modprobe zfs' as root to load them.

So you try to load the ZFS module and the system complains again:

#/sbin/modprobe zfs
modprobe: FATAL: Module zfs not found.
or
modprobe: ERROR: could not insert 'zfs': Invalid argument

What you need to do is to erase all the ZFS and related packages:

yum erase zfs zfs-dkms libzfs2 spl spl-dkms libzpool2 -y

Please reboot the system. This step is very important.

reboot

After that, try to install ZFS again.

yum install zfs -y

If the system complaints about mismatch dependent packages, try to remove the affected packages first and run the installation again.

After the installation, try to start the ZFS module:

/sbin/modprobe zfs
zpool import -a

If the ZFS is up and running, please clean up your DKMS from step 1. If it complains again, please follow the steps below:

  1. Reboot
  2. Clear the cache of the yum repository and try to update the system again. (sudo yum clean all)
  3. Reboot to the latest kernel
  4. Erase the ZFS and related packages, try it again.

Keep in mind that ZFS on Linux is based on DKMS, a very buggy and unreliable platform. Sometimes when you uninstall and install the packages, don’t expect that it will do the same thing as fresh install. Before you send your server to the landfill, try this:

Check the dkms status:

#dkms status
#version 0.7.x
zfs, 0.7.2: added
spl, 0.7.2: added

#version 0.8.x
zfs, 0.8.2: added

If you see this message, that means the ZFS packages have been installed, but the DKMS doesn’t know how to use it. You will need to tell DKMS about it:

#version 0.7.x
dkms --force install zfs/0.7.2
dkms --force install spl/0.7.2

#version 0.8.x
dkms --force install zfs/0.8.2
#Try to start ZFS again.
/sbin/modprobe zfs
zpool import -a

If you already tried it for more than 3 times without any luck, don’t waste your time. You may want to bring the ZFS disks to a different server. The new server should be able to recognize the ZFS disks. For the original server, you can connect to the ZFS disks on the new server via NFS using the original path. That will minimize the impact of changes.

Keep in mind that the ZFS version is very important. The server with newer ZFS version can read the ZFS disks created in older ZFS versions. You can always check the ZFS versions by running the following:

#Get the version of the host:
sudo zfs upgrade -v
sudo zpool upgrade -v


#Get the version of the ZFS disks:
sudo zfs get version
sudo zpool get version

This is pretty much what I need to do on my 60 servers every month. If you are in a similar situation like mine, I guarantee that you will become an expert of fixing this kind of mess after few months. Good luck!

Our sponsors:

9 thoughts on ““ZFS on Linux”: The ZFS modules are not loaded. Try running ‘/sbin/modprobe zfs’ as root to load them.”

  1. Thank you very much for sharing this! I had to make some minor changes to the above guidelines, but it really helped a lot!

    I had just performed an upgrade running Debian Stretch Backports and that broke everything on one particular box. After having spend several hours I stubled upon your post.

    Thanks!

  2. I think I’m in a worse position than this 🙁 I have upgraded `3.10.0-862.3.3.el7.x86_64` to `3.10.0-957.1.3.el7.x86_64` and have similar issue. This is my first issue with ZFS. I did have grub & kernel update issues, but those were easily resolved by reinstalling.

    “`
    $ zpool status
    The ZFS modules are not loaded.
    Try running ‘/sbin/modprobe zfs’ as root to load them.
    “`

    “`
    $ dkms status
    $
    “`

    “`
    $ /sbin/modprobe zfs
    modprobe: FATAL: Module zfs not found.
    “`

    “`
    $ dkms –force install spl/0.7.12;
    Error! Could not find module source directory.
    Directory: /usr/src/spl-0.7.12 does not exist.
    “`

    “`
    $ dkms –force install zfs/0.7.12;
    Error! Could not find module source directory.
    Directory: /usr/src/zfs-0.7.12 does not exist.
    “`

    Will keep searching, but if you’ve come across this I would really appreciate a point in the right direction.

    1. I hear you. If you need your data immediately, please reboot to the earlier working kernel. If you want to make it work with your latest kernel (v3, not v4), I will uninstall the ZFS and reinstall all of them. See Step 2 for details.

  3. This blog is helpful. I did patch my system(including ZFS) but kernel wasn’t updated.
    I ran into same issue, step 1 didn’t help, I add to do additions to step 2.
    erased new packages(0.7.12) and its dependencies too. Then reinstalled earlier version(0.7.11). Everything worked great.

    Below are the list of packages.

    libuutil1-0.7.11-1.el7_5.x86_64
    libnvpair1-0.7.11-1.el7_5.x86_64
    libzpool2-0.7.11-1.el7_5.x86_64
    libzfs2-0.7.11-1.el7_5.x86_64
    zlib-devel-1.2.7-17.el7.x86_64
    elfutils-libelf-devel-0.170-4.el7.x86_64
    dkms-2.6.1-1.el7.noarch
    spl-dkms-0.7.11-1.el7_5.noarch
    zfs-dkms-0.7.11-1.el7_5.noarch
    spl-0.7.11-1.el7_5.x86_64
    zfs-0.7.11-1.el7_5.x86_64

    1. Glad that it works. FYI, since you downgrade to an older version, you may want to disable the auto update (or exclude ZFS from the auto update). Also, I suggest to check which version of ZFS on Linux repository you are using, and they may give different version of packages (/etc/yum.repos.d/zfs.repo). Mine is 7.6 which is the latest one.

      You can install a different version here:
      sudo rpm -Uvh http://download.zfsonlinux.org/epel/zfs-release.el7_6.noarch.rpm

  4. I recently ran into this issue when I upgraded my ZFS test server from CentOS 7.3 to CentOS 7.6 (which also upgraded the kernel from 3.10.0-514.26.2.el7.x86_64 to 3.10.0-957.12.2.el7.x86_64). The ZFS on Linux github wiki page has pretty good instructions on this scenario (I think the page is fairly new and didn’t exist when this OP was created):

    https://github.com/zfsonlinux/zfs/wiki/RHEL-and-CentOS

    In short, I had to update the repo to point to the centos 7.6 branch and then follow the reinstallation instructions here:

    https://github.com/zfsonlinux/zfs/wiki/RHEL-and-CentOS#rhelcentos-7x-kmod-package-upgrade

    Ended up upgrading from zfs 0.6.5.9 to 0.7.13. Worked like a charm and no data was lost as far as I could tell. Hope this helps someone else out.

  5. I upgraded Debian 9 (stretch) to Debian 10 (buster). When booting, I now get the message ”

    Failed to load ZFS modules.
    Manually load the modules and exit.

    BusyBox v1.30.1 (Debian 1:1.30.1-4) built-in shell (ash)

    ‘modprobe zfs’ runs without reporting any error, but does not load the ZFS module. In fact ‘modprobe ‘ runs without an error and does nothing.

    There is no apt or apt-get anywhere (‘find / -iname “*apt*) does not find them.

    There are 6 kernels to boot from available:
    – 4.19.0-5-amd64
    – 4.9.0-9-amd64
    – 4.9.0-8-amd64
    (and each also in “recovery mode”) but none of them works, i.e. I get the same result when booting any of them.

    Any ideas?

    1. Sorry I don’t have much experience with Debian. But I will try my best to answer your questions.

      First, I will check if your ZFS on Linux package is the latest version or not (zfs-0.8.1). My experience told me that older version ZFS won’t work with Linux kernel v4. According to the ZFS release notes, the ZFS-0.8.1 supports kernel up to 5.1. Second, I will make sure that the ZFS meta data of your disks are not modified. You can verify this by booting an older version of Debian via USB/LiveCD etc. If the ZFS files are mountable, then we know that the problem is the OS/application. Otherwise you will need to go with a different route.

      Next, I will try the following:

      1.) Make sure that the ZFS / dkms are built correctly for your current kernel. Since you have three installed kernels in your system, you will need to make sure that each of them are build correctly.
      2.) Assuming that the dkms gives no error, then I will try to load the disk:

      #To load the disk
      sudo zpool import -a

      #To show the status of the disk
      sudo zdb

      Hopefully you can import the ZFS.

      3.) If you still can’t import your stuffs, you only have few options left:
      a.) Remove the ZFS and related packages and reinstall them again, or
      b.) Reinstall your Debian, or
      c.) Downgrade your Debian to an earlier version, or
      d.) Keep your current Debian, and load your ZFS disks on a different server with a different version of Debian. Mount those disks via NFS.

      Good luck!

      –Derrick

Leave a Reply

Your email address will not be published. Required fields are marked *