![\"\"](\"\/wordpress\/wp-content\/uploads\/2011\/01\/zfs_logo-150x150.jpg\" "\\"Building"){kind=logo}
<\/p>\n<\/p>\n
In this article, I am going to show you how to build a super large, ultra reliable, expandable<\/strong> file server with mixed size of hard drives. <\/p>\n Here is my criteria:<\/p>\n I am going to show you how to implement all of these using FreeBSD and ZFS. The idea of the implementation is the same on other operating systems that support ZFS, such as *Solaris, *BSD etc. However, using ZFS on Linux (via Linux FUSE) is not recommended given the performance \/ stability issues (See Wikipedia<\/a> for details). I personally do not recommend running ZFS on Linux in production environment.<\/p>\n Before I talk about how I do it, I like to go over what other technologies I tried before, and why they are not qualified.<\/p>\n This is one of the most common techniques used by many people on building file server. First, you need to build a RAID device using mdadm on the selected harddrives with identical size. If you have different size of harddrive, you will need to put them on a separated RAID device because mdadm does not support mixing different size of harddrive in a single RAID device without losing usable space. Later, you combine multiple devices (RAID, single harddrives etc) into a single partition using LVM.<\/p>\n The good thing about this technique is the expandability, i.e., you can add any hard drive to the partition at any time without losing any data at any time. However, there are few disadvantages of this combination:<\/p>\n Poor reliability<\/strong> In this example, if the device 2 (2TB) is failed, the data that is stored on device 2 will be lost, because the data redundancy is only available on device 1.<\/p><\/div>\n Performance is not optimized<\/strong> So, let’s see how ZFS<\/strong> can solve these issues.<\/p>\n ZFS is the next generation file system developed by Sun. It comes with all advantages of mdadm + LVM, plus a lot of features such as compression, encryption, power failure handling, checksum etc. So why this setup is better than the LVM one? Let’s take a look to the following example:<\/p>\n In this example, all devices in the ZFS pool are protected against data failure.<\/p><\/div>\n First, let’s assume that I have the following hardware:<\/p>\n Using this combination(10GB in total), you can build a big file server with 8GB<\/strong> or 6GB<\/strong>, depending on your preference of data security.<\/p>\n I am going to build a RAIDZ2 device using FreeBSD and ZFS. In general ZFS is only available on the harddrives with the same size (without losing harddrive space). Since I want to put my 1TB and 2TB harddrives in the same pool, I first create couple RAID0 drives first. Then I add them together to make a big ZFS device. Here is the big picture:<\/p>\n As usual, login as root first. <\/p>\n And load the stripe module:<\/p>\n Now we need to create a RAID0 device from \/dev\/ad1(A: 1TB) and \/dev\/ad2(B:1TB). If you are unsure about the device name, try running dmesg for details:<\/p>\n And label the new device: \/dev\/stripe\/st0<\/strong><\/p>\n Format the new device:<\/p>\n Mount the device for testing:<\/p>\n Verify the size:<\/p>\n Add the following into \/boot\/loader.conf<\/strong>:<\/p>\n Now, reboot your machine. If \/dev\/stripe\/st0<\/strong> is available, then your RAID0 device is ready.<\/p>\n If you need to build more RAID0 devices, repeat the above steps. Keep in mind that you need to change the device name from st0<\/strong> to st1<\/strong>.<\/p>\n First, let’s enable ZFS first:<\/p>\n and start ZFS:<\/p>\n Get your devices ready. In my cases, my devices name are: Create the ZFS pool, which will mount on \/storage\/data<\/strong> Verify the result:<\/p>\n We need to make some performance tweaking first, otherwise the system will be very unstable. Add the following to the \/boot\/loader.conf<\/strong><\/p>\n Now reboot the system:<\/p>\n To test the stability of the system, I recommend saving some files into the ZFS pool and run it for few days.<\/p>\n ZFS is not supported by Linux natively due to legal issue. In the other words, it is not supported by Linux kernel. Although ZFS has been ported to Linux via FUSE, which is running ZFS as an application, it will introduce the performance and efficiency penalties. (Source: ZFS Wiki<\/a>). So I don’t recommend running ZFS on Linux.<\/p>\n<\/div>\n\n
RAID + LVM on Linux<\/h2>\n
\nThe reliability is handled at the RAID level. If your setup is not 100% RAID, such as the following example, the reliability can be discounted.<\/p>\n![\"\"](\"\/wordpress\/wp-content\/uploads\/2011\/01\/Super_Large_File_Server_LVM-300x173.png\" "\\"Super_Large_File_Server_LVM\\"")
<\/a>
\nData redundancy helps to improve the performance especially on reading. Again, if the setup is not one single RAID device, the performance can be discounted too.<\/p>\nZFS<\/h2>\n
![](\"\/wordpress\/wp-content\/uploads\/2011\/01\/Super_Large_File_Server_ZFS_is_better_than_LVM-300x173.png\" "\\"In")
<\/a>Let’s Start Building A Super Large File Server Now!<\/h3>\n
\n
![\"\"](\"\/wordpress\/wp-content\/uploads\/2011\/01\/Super_Large_File_Server_ZFS-300x119.png\" "\\"Super_Large_File_Server_ZFS\\"")
<\/a><\/p>\nBuilding RAID0 Devices<\/h3>\n
sudo su<\/pre>\n
kldload geom_stripe<\/pre>\n
dmesg | grep ad<\/pre>\n
gstripe label -v st0 \/dev\/ad1 \/dev\/ad2<\/pre>\n
bsdlabel -wB \/dev\/stripe\/st0<\/pre>\n
newfs -U \/dev\/stripe\/st0a<\/pre>\n
mount \/dev\/stripe\/st0a \/mnt<\/pre>\n
df -h<\/pre>\n
geom_stripe_load=\"YES\"<\/pre>\n
Putting all devices together into ZFS Pool<\/h3>\n
echo 'zfs_enable=\"YES\"' >> \/etc\/rc.conf<\/pre>\n
\/etc\/rc.d\/zfs start<\/pre>\n
\n\/dev\/ad5: 2TB
\n\/dev\/ad6: 2TB
\n\/dev\/ad7: 2TB
\n\/dev\/stripe\/st0: 2TB (RAID0: 2x1TB)
\n\/dev\/stripe\/st1: 2TB (RAID0: 2x1TB)<\/p>\n
\nNote that I use raidz2 here for extra protection against data failure. Basically, raidz (single parity) allows up to 1 failed harddrives while raidz2 (double parity) allows up to 2 failed harddrives. <\/p>\n\n
zpool create storage raidz2 \/dev\/ad5 \/dev\/ad6 \/dev\/ad7 \/dev\/stripe\/st0 \/dev\/stripe\/st1<\/pre>\n
zfs create storage\/data<\/pre>\n
zpool status<\/pre>\n
df -h<\/pre>\n
\r\n#Your Physical Memory minus 1 GB\r\nvm.kmem_size=\"3g\"\r\nvm.kmem_size_max=\"3g\"\r\n\r\nvfs.zfs.arc_min=\"512m\"\r\n\r\n#Your Physical Memory minus 2GB\r\nvfs.zfs.arc_max=\"2048m\"\r\n\r\n\r\nvfs.zfs.vdev.min_pending=\"1\"\r\nvfs.zfs.vdev.max_pending=\"1\"\r\n<\/pre>\n
\r\nreboot\r\n<\/pre>\n
Why Running ZFS on Linux is not a Solution?<\/h2>\n
Enhancement Idea<\/h3>\n
\n
Further Readings<\/h3>\n