{"id":4377,"date":"2017-06-05T08:57:43","date_gmt":"2017-06-05T13:57:43","guid":{"rendered":"http:\/\/icesquare.com\/wordpress\/?p=4377"},"modified":"2021-06-03T16:47:12","modified_gmt":"2021-06-03T21:47:12","slug":"zfs-cluster-a-network-based-zfs-implementation","status":"publish","type":"post","link":"https:\/\/icesquare.com\/wordpress\/zfs-cluster-a-network-based-zfs-implementation\/","title":{"rendered":"ZFS Cluster: A Network-Based ZFS Implementation"},"content":{"rendered":"

I always like to experimenting the idea of building a ZFS cluster, i.e., it has the robust of the ZFS with the cluster capacity. So I came up a test environment with this prototype. Keep in mind that this is just a proof of concept rather than a production-ready solution. I mainly want to test the idea of ZFS cluster.<\/p>\n

The idea is pretty simple. Typically when we build the ZFS server, the members of the RAID are the hard drives. In my experiment, I use files instead of hard drives, where the corresponding files live in a network share (mounted via NFS). Since the bottle neck of the I\/O will be limited by the network, I include a network bonding to increase the overall bandwidth.<\/p>\n

The yellow servers are simply regular servers running ZFS with NFS service. I use the following command to generate a simple file \/ place holder for ZFS mounting:<\/p>\n

\r\n#This will create an empty 1TB file, you can think of it as a 1TB hard drive \/ place holder.\r\ntruncate -s 1000G file.img\r\n<\/pre>\n
Make sure that the corresponding NFS service is serving the file.img to the client (the blue server).<\/p>\n
\n
The blue server will be the NFS client of the yellow servers, where I will use it to serve the data to other computers. It has the following features:<\/p>\n
It has a network bonding based on three Ethernet adapter:<\/p>\n
\r\n#cat \/proc\/net\/bonding\/bond0\r\n\r\n\r\nEthernet Channel Bonding Driver: v3.7.1 (April 27, 2011)\r\n\r\nBonding Mode: adaptive load balancing\r\nPrimary Slave: None\r\nCurrently Active Slave: enp0s25\r\nMII Status: up\r\nMII Polling Interval (ms): 1\r\nUp Delay (ms): 0\r\nDown Delay (ms): 0\r\n\r\nSlave Interface: enp0s25\r\nMII Status: up\r\nSpeed: 1000 Mbps\r\nDuplex: full\r\nLink Failure Count: 0\r\nPermanent HW addr: 00:19:d1:b2:1e:0d\r\nSlave queue ID: 0\r\n\r\nSlave Interface: enp6s0\r\nMII Status: up\r\nSpeed: 1000 Mbps\r\nDuplex: full\r\nLink Failure Count: 0\r\nPermanent HW addr: 00:18:4d:f0:12:7b\r\nSlave queue ID: 0\r\n\r\nSlave Interface: enp6s1\r\nMII Status: up\r\nSpeed: 1000 Mbps\r\nDuplex: full\r\nLink Failure Count: 0\r\nPermanent HW addr: 00:22:3f:f6:98:03\r\nSlave queue ID: 0\r\n<\/pre>\n
It mounts the yellow servers via NFS<\/p>\n
\r\n#df\r\n192.168.1.101:\/storage\/share   25T  3.9T   21T  16%  \/nfs\/192-168-1-101\r\n192.168.1.102:\/storage\/share   8.1T  205G  7.9T   3% \/nfs\/192-168-1-102\r\n192.168.1.103:\/storage\/share   8.3T  4.4T  4.0T  52% \/nfs\/192-168-1-103\r\n<\/pre>\n
The ZFS has the following structure:<\/p>\n
\r\n#sudo zpool status\r\n        NAME                                  STATE     READ WRITE CKSUM\r\n        storage                               ONLINE       0     0     0\r\n          raidz1-0                            ONLINE       0     0     0\r\n            \/nfs\/192-168-1-101\/file.img       ONLINE       0     0     0\r\n            \/nfs\/192-168-1-102\/file.img       ONLINE       0     0     0\r\n            \/nfs\/192-168-1-103\/file.img       ONLINE       0     0     0\r\n\r\nor:\r\n\r\nzpool create -f storage raidz \/nfs\/192-168-1-101\/file.img \\\r\n                              \/nfs\/192-168-1-102\/file.img \\\r\n                              \/nfs\/192-168-1-103\/file.img\r\n<\/pre>\n
\n
The speed of the network will be the limitation of the system, I don’t expect the I\/O speed goes beyond 375MB\/s (125MB\/s x 3). Also since it is a file-based ZFS (the ZFS on the blue server is based on files, not disks), so the overall performance will be discounted.<\/p>\n
\r\n#Write speed\r\ntime dd if=\/dev\/zero of=\/storage\/data\/file.out bs=1M count=1000\r\n1000+0 records in\r\n1000+0 records out\r\n1048576000 bytes (1.0 GB) copied, 3.6717 s, 285 MB\/s\r\n<\/pre>\n
\r\n#Read speed\r\ntime dd if=\/storage\/data\/file.out of=\/dev\/null\r\n1000+0 records in\r\n1000+0 records out\r\n1048576000 bytes (1.0 GB) copied, 3.9618 s, 265 MB\/s\r\n<\/pre>\n
Both read and the write speed are roughly around 75% of the maximum bandwidth, which is not bad at all.<\/p>\n
So I decide to make one of the yellow servers offline, let’s see what’s going on:<\/p>\n
\r\n#sudo zpool status\r\n        NAME                                  STATE     READ WRITE CKSUM\r\n        storage                               ONLINE       0     0     0\r\n          raidz1-0                            ONLINE       0     0     0\r\n            \/nfs\/192-168-1-101\/file.img       ONLINE       0     0     0\r\n            \/nfs\/192-168-1-102\/file.img       ONLINE       0     0     0\r\n            \/nfs\/192-168-1-103\/file.img       UNAVAIL      0     0     0 cannot open\r\n<\/pre>\n
And the pool is still functioning, that’s pretty cool!<\/p>\n
\n
Here are some notes that will affect the overall performance:<\/p>\n