Today I am going to share my story with you on how to expand my ZFS storage. I have a giant ZFS pool, which consists of 8x2TB (RAIDZ2) and 4×1.5TB (RAIDZ1), with a total of 15TB usable spaces. Since I am running out of space, I decide to upgrade the ZFS by replacing the disks one at a time. So here is the ZFS structure: <\/p>\n
\r\n#zpool status\r\n NAME STATE READ WRITE CKSUM\r\n storage ONLINE 0 0 0\r\n raidz2-0 ONLINE 0 0 0\r\n ada0 ONLINE 0 0 0\r\n ada1 ONLINE 0 0 0\r\n ada3 ONLINE 0 0 0\r\n ada4 ONLINE 0 0 0\r\n ada5 ONLINE 0 0 0\r\n ada6 ONLINE 0 0 0\r\n ada7 ONLINE 0 0 0\r\n ada8 ONLINE 0 0 0\r\n raidz1-1 ONLINE 0 0 0\r\n da0 ONLINE 0 0 0\r\n da1 ONLINE 0 0 0\r\n da2 ONLINE 0 0 0\r\n da3 ONLINE 0 0 0\r\n\r\n\r\n#df\r\nstorage\/data 14T 13T 1.3T 91% \/storage\/\r\n<\/pre>\nWhat I decide to do is to replace the 1.5TB disks by 3TB disks (i.e., da[0-3]) one at a time. Basically, here are the steps you typically found on the web<\/a>:<\/p>\n
\n
- Power down the server<\/li>\n
- Replace the 1.5TB disk by 3TB disk one at a time<\/li>\n
- Power on the server<\/li>\n
- Replace the disk<\/li>\n
- Resilver the entire pool.<\/li>\n
- Hope the resilver process does not return any error.<\/li>\n
- Repeat the above steps until all disks are replaced.<\/li>\n
- Turn on the auto expand option and enjoy the extra space<\/li>\n<\/ol>\n
Here are the corresponding commands. After replacing the first disk, you should see the following:<\/p>\n
\r\n#zpool status\r\n NAME STATE READ WRITE CKSUM\r\n storage DEGRADED 0 0 0\r\n raidz2-0 ONLINE 0 0 0\r\n ada0 ONLINE 0 0 0\r\n ada1 ONLINE 0 0 0\r\n ada3 ONLINE 0 0 0\r\n ada4 ONLINE 0 0 0\r\n ada5 ONLINE 0 0 0\r\n ada6 ONLINE 0 0 0\r\n ada7 ONLINE 0 0 0\r\n ada8 ONLINE 0 0 0\r\n raidz1-1 DEGRADED 0 0 0\r\n da0 UNAVAI 0 0 0 cannot open\r\n da1 ONLINE 0 0 0\r\n da2 ONLINE 0 0 0\r\n da3 ONLINE 0 0 0\r\n<\/pre>\nHowever, the pool will continue to work because it is a RAIDZ pool. So I decide to swap the 1.5TB with the 3TB disk:<\/p>\n
\r\n#sudo zpool replace mypool da0\r\n<\/pre>\n\r\n#zpool status\r\n NAME STATE READ WRITE CKSUM\r\n storage DEGRADED 0 0 0\r\n raidz2-0 ONLINE 0 0 0\r\n ada0 ONLINE 0 0 0\r\n ada1 ONLINE 0 0 0\r\n ada3 ONLINE 0 0 0\r\n ada4 ONLINE 0 0 0\r\n ada5 ONLINE 0 0 0\r\n ada6 ONLINE 0 0 0\r\n ada7 ONLINE 0 0 0\r\n ada8 ONLINE 0 0 0\r\n raidz1-1 DEGRADED 0 0 0\r\n replacing-2 DEGRADED 0 0 0\r\n da0\/old FAULTED 0 0 0 corrupted data\r\n da0 ONLINE 0 0 0 (resilvering)\r\n da1 ONLINE 0 0 0\r\n da2 ONLINE 0 0 0\r\n da3 ONLINE 0 0 0\r\n<\/pre>\nDepending on how much data you have on the old disk and the hardware (such as motherboard, SATA configurations etc), resilvering a 1.5TB drive (with 1.3TB of data) took me about 15 hours. Personally, I recommend to start this process in the morning, then you can check the progress and start the next one in the evening. That way you can speed up the work.<\/p>\n
So after the resilvering process is done. Make sure that you check the error status. If some files were missing, you need to delete those files first and restore them from your backup. Here is an example of the error:<\/p>\n
\r\n#sudo zpool status -v\r\n\r\nerrors: Permanent errors have been detected in the following files: \r\n\r\n\/storage\/data\/aaa\r\n\/storage\/data\/bbb\r\n\/storage\/data\/ccc\r\n<\/pre>\nRemember, you need to delete the file first, then put the file back from your backup. If you simply replace\/overwrite the file, it will not clear the error. Try to check the status again. If the error is not gone yet, you may want to scrub the zpool to trigger the resilver process.<\/p>\n
\r\nsudo zpool scrub mypool\r\n<\/pre>\nOr you can try to clear the error message. It will trigger the resilver process automatically.<\/p>\n
\r\nsudo zpool clear -f mypool\r\n<\/pre>\nI know what you are trying to say now. How come the ZFS will lose the data even I have RAIDZ and checksum enabled? I have no idea. That’s why we need back up on a different machine. Anyway, the resilver process will take another 15 hours.<\/p>\n
After the error is cleared, repeat the steps to replace the disks one by one. Make sure that the error is cleared after every replacement. For me, replacing four hard drives took me exact 5 days, or 120 hours in total. Yes, it is not a fun job.<\/p>\n
So after everything is completed, no error or anything bad. You try to check the pool status and you expect a magic will happen. Unfortunately, you will see the same amount of space available. Here are what you will need to do:<\/p>\n
\r\n#I still have 1.3TB space left.\r\nstorage\/data 14T 13T 1.3T 91% \/storage\/\r\n<\/pre>\n\r\nsudo zpool set autoexpand=on mypool \r\n<\/pre>\nLocate one of the disks you have replaced, in my case, they are da0, da1, da2 and da3<\/p>\n
\r\n#zpool status\r\n NAME STATE READ WRITE CKSUM\r\n storage ONLINE 0 0 0\r\n raidz2-0 ONLINE 0 0 0\r\n ada0 ONLINE 0 0 0\r\n ada1 ONLINE 0 0 0\r\n ada3 ONLINE 0 0 0\r\n ada4 ONLINE 0 0 0\r\n ada5 ONLINE 0 0 0\r\n ada6 ONLINE 0 0 0\r\n ada7 ONLINE 0 0 0\r\n ada8 ONLINE 0 0 0\r\n raidz1-1 ONLINE 0 0 0\r\n da0 ONLINE 0 0 0 --This\r\n da1 ONLINE 0 0 0 --This\r\n da2 ONLINE 0 0 0 --This\r\n da3 ONLINE 0 0 0 --And this\r\n<\/pre>\n\r\nsudo zpool online -e mypool da0\r\n<\/pre>\nAnd check the space again…<\/p>\n
\r\n#Now I have 5.3TB space available.\r\nstorage 18T 13T 5.3T 72% \/storage\r\n<\/pre>\nFYI, here is the math behind the free space calculations. I had 4 x 1.5TB on a RAIDZ1 setup. After the upgrade, I have 4 x 3TB on a RAIDZ1 set up. The increase space will be (3TB – 1.5TB) x (4 – 1) * 0.9 = 4TB<\/p>\n
Enjoy the new space!<\/p>\n
If you have spent too many hours on the resilvering process, consider the old school way. My old school method is nothing new, but it is rock solid, reliable, takes shorter time and most importantly, no data will be lost. Yes, you are right, I back up the data to another server first, and then I rebuild the ZFS pool on my main server, and copy the data back. Typically, copying the 10TB of data via rsync daemon<\/a> over a gigabit network will take about 3 days. So it isn’t too bad. The only down side of this solution is the down time, which is about 3 days in my case. If you decide to go with the ZFS replacement, the downtime will be minimized, because the ZFS pool will continue to work during the resilver process.<\/p>\n