If you check your Zpool health status and you notice an error like the following:<\/p>\n
\r\nsudo zpool status\r\n pool: myzpool\r\n state: DEGRADED\r\nstatus: One or more devices are faulted in response to persistent errors.\r\n Sufficient replicas exist for the pool to continue functioning in a\r\n degraded state.\r\naction: Replace the faulted device, or use 'zpool clear' to mark the device\r\n repaired.\r\n scan: resilvered 49.6G in 0 days 00:11:25 with 0 errors on Fri Jan 10 15:52:05 2020\r\nconfig:\r\n\r\n NAME STATE READ WRITE CKSUM\r\n myzpool DEGRADED 0 0 0\r\n raidz1-0 DEGRADED 0 0 0\r\n ata-TOSHIBA_0001 ONLINE 0 0 0\r\n ata-TOSHIBA_0002 ONLINE 0 0 0\r\n ata-TOSHIBA_0003 FAULTED 36 0 0 too many errors\r\n cache\r\n nvme0n1 ONLINE 0 0 0\r\n\r\n<\/pre>\nThere are two possibilities: hardware error or software error. I will perform the following to identify whether it is hardware or software error.<\/p>\n
\n
- Check whether the disk is missing from the system or not. You can do it by running
fdisk -l<\/code>. If the disk is available, try to clear the ZFS status. If the disk is missing, try to reboot the system.<\/li>\n- If the disk is still missing after reboot the system, try to replace the hard drive cable.<\/li>\n
- Once the ZFS sees all disks, try to run
zpool clear myzpool<\/code>. This will force the ZFS to resilver the pool. If the pool is running at 100MB\/s or above, it sounds like a false alarm. You may stop here.<\/li>\n<\/ol>\n
\nAssuming that it is hardware related error. Typically you can do the following:<\/p>\n
\n
- Replace the SATA \/ SAS cable<\/li>\n
- Replace the hard drive<\/li>\n
- In the BIOS settings, change the write mode from SATA\/IDE to AHCI<\/li>\n<\/ul>\n
If you replace the hard drive, you will need to resilver the pool. If it is hardware error, the pool will read\/write the data at least 100MB\/s. Depending on the size of data on your faulty disk, it should take no more than 3 days to finish the entire process. Wait until the process is finished. If it gives no error, you may stop here.<\/p>\n
\r\nsudo zpool status\r\n pool: myzpool\r\n state: ONLINE\r\n scan: resilvered 215M in 0 days 00:00:04 with 0 errors on Mon Jan 13 18:24:48 2020\r\nconfig:\r\n\r\n NAME STATE READ WRITE CKSUM\r\n myzpool ONLINE 0 0 0\r\n raidz1-0 ONLINE 0 0 0\r\n ata-TOSHIBA_0001 ONLINE 0 0 0\r\n ata-TOSHIBA_0002 ONLINE 0 0 0\r\n ata-TOSHIBA_0003 ONLINE 0 0 0\r\n cache\r\n nvme0n1 ONLINE 0 0 0\r\n\r\nerrors: No known data errors\r\n<\/pre>\n
\nSo you may be in one or more of the following situations:<\/p>\n
\n
- Your hardware is consumer level, e.g., the motherboard is not server grade, or the hard drive is designed for general purposes rather than 24\/7.<\/li>\n
- You have replaced the hard drive, and the resilver process is very slow (e.g., 5-15MB\/s). The ZFS cannot even give you an estimated finish time. <\/li>\n
- The resilver estimated end time keeps being delayed, and it seems taking forever. For example, suppose ZFS estimates that the entire process may take 10 hours to finish. After 5 hours, it says 9 more hours to go, or once it reaches 99.9%, it starts the entire process again.<\/li>\n
- When you run
dmesg<\/code>, you see a lot of hardware related error, e.g.,\n\r\nata2.00: status: { DRDY }\r\nata2.00: failed command: READ FPDMA QUEUED\r\nata2.00: cmd 60\/78:e0:f0:70:77\/00:00:39:00:00\/40 tag 28 ncq 61440 in\r\nsd 1:0:0:0: [sdb] tag#25 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE\r\nsd 1:0:0:0: [sdb] tag#25 Sense Key : Illegal Request [current] [descriptor]\r\nsd 1:0:0:0: [sdb] tag#25 Add. Sense: Unaligned write command\r\n<\/pre>\n<\/li>\n- ZFS shows multiple disks are faulty. However their states become online after rebooting the system.<\/li>\n<\/ul>\n
If you are in any of these situations, there are multiple things you can do:<\/p>\n
If this is an important server and you don’t have any backup data, just wait. There is nothing you can do.<\/p>\n
If you have backup data, try to destroy the pool and rebuild again. The problem is that your Linux or ZFS does not like the current configurations. It is software rather than hardware issue. By rebuilding the entire zpool, everything will start over again. This will save you many days, weeks or even months of waiting time. In my situation, I had spent 2 months to resilver the data on my secondary backup server. After letting it to resilver for 2 months, I decided to rebuild the entire ZFS pool (using exact the same hardware) and loaded the data from my production server. It took less than a week to fill 50TB of data and the dmesg is clear of error message.<\/p>\n
Of course, sometimes the hard drive is faulty. We can perform a simple test with the following commands. First, try to link it to another server (or USB enclosure) and run the following (replace sdX with the actual hard drive identifier):<\/p>\n
\r\nsudo smartctl -a \/dev\/sdX | grep result\r\n\r\n#Bad hard drive\r\nSMART overall-health self-assessment test result: FAILED!\r\n\r\n#Good hard drive\r\nSMART overall-health self-assessment test result: PASSED\r\n<\/pre>\nNext, we perform a more intensive test. This will involve wiping your entire hard drive (writing data to every sector):<\/p>\n
\r\nnohup sudo dd if=\/dev\/zero of=\/dev\/sdX bs=1M status=progress > dd.log &\r\n\r\n#This may take few days. You can check the progress this way:\r\nsudo tail dd.log\r\n<\/pre>\nOnce the process is done, try to run
dmesg | grep sdX<\/code>. If the hard drive is faulty, you will definitely see lots of error messages. In my case, pretty much all of the hard drives give no error. What does it mean? It means the ZFS system doesn’t like those hard drives. All I can say is that my ZFS is up and running (and error free) after rebuilding the entire pool, using exact the same hard drives and cables.<\/p>\nIf you have tried this for multiple times but no luck, there is another thing you can try before dumping your hard drives: Switch to FreeBSD.<\/p>\n
I had a CentOS 7 server and I was having exact the same situations. I’ve wiped the disks and rebuilt the pool, and I couldn’t make the error go away. So I decided to switch to FreeBSD 12 (as of April, 2020), and I rebuilt the pool using exact the same specifications, and filled the pool with data. There was no error and the operation was extremely smooth. <\/p>\n
<\/p>\n","protected":false},"excerpt":{"rendered":"