{"id":3743,"date":"2014-09-25T12:38:20","date_gmt":"2014-09-25T17:38:20","guid":{"rendered":"http:\/\/icesquare.com\/wordpress\/?p=3743"},"modified":"2021-01-06T00:27:18","modified_gmt":"2021-01-06T06:27:18","slug":"zfs-performance-mirror-vs-raidz-vs-raidz2-vs-raidz3-vs-striped","status":"publish","type":"post","link":"https:\/\/icesquare.com\/wordpress\/zfs-performance-mirror-vs-raidz-vs-raidz2-vs-raidz3-vs-striped\/","title":{"rendered":"ZFS Performance: Mirror VS RAIDZ VS RAIDZ2 vs RAIDZ3 vs Striped"},"content":{"rendered":"<style>\nh3{\n   margin:15px 0px 15px 0px;\n}\n.simpleFrame{\n   border:2px solid #666;\n   color:#333;\n   box-shadow: 0 3px 3px rgba(34, 25, 25, 0.4);\n}\n.tableContainer{\n    display:table;\n    width:90%;\n    border-collapse: collapse;\n}\n.tableHeading{\n    font-weight: bold;\n     display:table-row;\n     text-align: center;\n     line-height: 25px;\n     font-size: 14px;\n     font-family:georgia;\n}\n.tableRow{  \n     display:table-row;\n     text-align: center;\n}\n.tableCol{\n    display:table-cell;\n    border: 1px solid #CCC;\n    padding:3px;\n}\n.seperator{\n   border-bottom: 1px dotted #999; \n   margin:20px 0px 20px 0px;\n}\n<\/style>\n<p>I always wanted to find out the performance difference among different ZFS types, such as mirror, RAIDZ, RAIDZ2, RAIDZ3, Striped, two RAIDZ vdevs vs one RAIDZ2 vdev etc. So I decide to create an experiment to test these ZFS types. Before we talk about the test result, let&#8217;s go over some background information, such as the details of each design and the hardware information.<\/p>\n<h3>Background<\/h3>\n<p>Here is a machine I used for experiment. It is a consumer grade desktop computer manufactured back in 2014:<\/p>\n<pre>\r\nCPU: Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz \/ quard cores \/ 8 threads\r\nOS: CentOS Linux release 7.3.1611 (Core)\r\nKernel: Linux 3.10.0-514.6.1.el7.x86_64 #1 SMP Wed Jan 18 13:06:36 UTC 2017 x86_64 x86_64 x86_64 GNU\/Linux\r\nMemory: 20 GB (2GB x 4)\r\nHard drives: 5 TB x 8 \r\n(Every hard drive is 4k sectors, non-SSD, consumer grade, connected via a PCI-e x 16 raid card with SAS interface)\r\nSystem Settings: Everything is system default. Nothing has been done to the kernel configuration.\r\n<\/pre>\n<p>Also, I tried to keep each test simple. Therefore I didn&#8217;t do anything special:<\/p>\n<pre>\r\nzpool create -f myzpool (different settings go here...)\r\nzfs create myzpool\/data\r\n<\/pre>\n<p>To optimize the I\/O performance, the block size of the zpool is based on the physical sector of the hard drive. In my case, all of the hard drives have 4k (4096 bytes) sectors, which is translated to 2^12, therefore, the ashift value of the zpool is 12.<\/p>\n<pre>\r\nzdb | grep ashift\r\nashift: 12\r\n<\/pre>\n<p>To measure the write performance, I first generate a zero based file with the size of 41GB and output to the zpool directly. To measure the read performance, I read the file and output to \/dev\/null. Notice that the file size is very large (41GB) such that it does not fit in the arc cache memory (50% of the system memory, i.e., 10GB). Notice that the block size is the physical sector of the hard drive.<\/p>\n<p>One of the readers asked me why I use a large file instead of many small files. There are few reasons:<\/p>\n<ul>\n<li>It is very easy to stress test \/ saturate the bandwidth (connection in between the hard drives, network etc) when working with large file.<\/li>\n<li>The results of testing large files is more consistent.<\/li>\n<\/ul>\n<pre>\r\n#To test the write performance:\r\ndd if=\/dev\/zero of=\/myzpool\/data\/file.out bs=4096 count=10000000\r\n\r\n#To test the read performance:\r\ndd if=\/myzpool\/data\/file.out of=\/dev\/null bs=4096\r\n<\/pre>\n<p>FYI, if the block size is not specified, the result can be very different:<\/p>\n<pre>\r\n#Using default block size:\r\ndd if=\/myzpool\/data\/file.out of=\/dev\/null\r\n40960000000 bytes (41 GB) copied, 163.046 s, 251 MB\/s\r\n\r\n#Using native block size:\r\ndd if=\/myzpool\/data\/file.out of=\/dev\/null bs=4096\r\n40960000000 bytes (41 GB) copied, 58.111 s, 705 MB\/s\r\n<\/pre>\n<p>After each test, I destroyed the zpool and created a different one. This ensures that the environment factors (such as hardware and OS) stay the same. Here is the test result. If you want to learn more about each design, such as the exact command I used for each test, the corresponding material will be available in the later section.<\/p>\n<p>Notice that I used eight 5TiB hard drives (Total: 40TiB) in this test. Typically hard drive of 5TiB of can hold about 4.5 TB of data, that&#8217;s around 86%-90% of the advertised number, depending on which OS you are using. For example, if we use the striped design, which is the maximum possible storage capacity in ZFS, the usable space will be 8 x 5TiB x 90% = 36TB. Therefore, the following percentages will be based on 36TB rather than 40TiB.<\/p>\n<p>You may notice that I use 10 disks in each diagram, while I use only 8 disks in the article here. That&#8217;s because the diagram was from my first edit. At that time I used a relative old machine, which may not reflect the modern ZFS design. The hardware and the test methods I used in the second edit is better, although both edits draw the same conclusion.<\/p>\n<div class=\"seperator\"><\/div>\n<h3>Test Result<\/h3>\n<h4>(Sorted by speed)<\/h4>\n<div class=\"tableContainer\">\n<div class=\"tableHeading\">\n<div class=\"tableCol\">No.<\/div>\n<div class=\"tableCol\">ZFS Type <br \/> (Click to see details)<\/div>\n<div class=\"tableCol\">Write Speed (MB\/s)<\/div>\n<div class=\"tableCol\">Time Spent on Writing a 41GB File<\/div>\n<div class=\"tableCol\">Read Speed (MB\/s)<\/div>\n<div class=\"tableCol\">Time Spent on Reading a 41GB File<\/div>\n<div class=\"tableCol\">Storage Capacity (Max: 36TB)<\/div>\n<div class=\"tableCol\"># of Disks Used On Data Parity<\/div>\n<div class=\"tableCol\">Disk Arrangement<\/div>\n<\/p><\/div>\n<div class=\"tableRow\">\n<div class=\"tableCol\"><a href=\"#striped\">1.<\/a><\/div>\n<div class=\"tableCol\"><a href=\"#striped\">Striped<\/a><\/div>\n<div class=\"tableCol\">705<\/div>\n<div class=\"tableCol\">58.111s<\/div>\n<div class=\"tableCol\">687<\/div>\n<div class=\"tableCol\">59.6386s<\/div>\n<div class=\"tableCol\">36TB (100%)<\/div>\n<div class=\"tableCol\">0<\/div>\n<div class=\"tableCol\">Striped (8)<\/div>\n<\/p><\/div>\n<div class=\"tableRow\">\n<div class=\"tableCol\"><a href=\"#raidz1x2\">2.<\/a><\/div>\n<div class=\"tableCol\"><a href=\"#raidz1x2\">RAIDZ1 x 2<\/a><\/div>\n<div class=\"tableCol\">670<\/div>\n<div class=\"tableCol\">61.1404s<\/div>\n<div class=\"tableCol\">680<\/div>\n<div class=\"tableCol\">60.2457s<\/div>\n<div class=\"tableCol\">26TB (72%)<\/div>\n<div class=\"tableCol\">2<\/div>\n<div class=\"tableCol\">RAIDZ (4) x 2<\/div>\n<\/p><\/div>\n<div class=\"tableRow\">\n<div class=\"tableCol\"><a href=\"#raidz2\">3.<\/a><\/div>\n<div class=\"tableCol\"><a href=\"#raidz2\">RAIDZ2<\/a><\/div>\n<div class=\"tableCol\">608<\/div>\n<div class=\"tableCol\">67.3897s<\/div>\n<div class=\"tableCol\">673<\/div>\n<div class=\"tableCol\">60.8205s<\/div>\n<div class=\"tableCol\">25TB (69%)<\/div>\n<div class=\"tableCol\">2<\/div>\n<div class=\"tableCol\">RAIDZ2 (8)<\/div>\n<\/p><\/div>\n<div class=\"tableRow\">\n<div class=\"tableCol\"><a href=\"#raidz1\">4.<\/a><\/div>\n<div class=\"tableCol\"><a href=\"#raidz1\">RAIDZ1<\/a><\/div>\n<div class=\"tableCol\">604<\/div>\n<div class=\"tableCol\">67.8107s<\/div>\n<div class=\"tableCol\">631<\/div>\n<div class=\"tableCol\">64.8782s<\/div>\n<div class=\"tableCol\">30TB (83%)<\/div>\n<div class=\"tableCol\">1<\/div>\n<div class=\"tableCol\">RAIDZ (8)<\/div>\n<\/p><\/div>\n<div class=\"tableRow\">\n<div class=\"tableCol\"><a href=\"#raidz3\">5.<\/a><\/div>\n<div class=\"tableCol\"><a href=\"#raidz3\">RAIDZ3<\/a><\/div>\n<div class=\"tableCol\">528<\/div>\n<div class=\"tableCol\">77.549s<\/div>\n<div class=\"tableCol\">577<\/div>\n<div class=\"tableCol\">70.9604s<\/div>\n<div class=\"tableCol\">21TB (58%)<\/div>\n<div class=\"tableCol\">3<\/div>\n<div class=\"tableCol\">RAIDZ3 (8)<\/div>\n<\/p><\/div>\n<div class=\"tableRow\">\n<div class=\"tableCol\"><a href=\"#mirror\">6.<\/a><\/div>\n<div class=\"tableCol\"><a href=\"#mirror\">Mirror<\/a><\/div>\n<div class=\"tableCol\">473<\/div>\n<div class=\"tableCol\">86.6451s<\/div>\n<div class=\"tableCol\">598<\/div>\n<div class=\"tableCol\">68.4477s<\/div>\n<div class=\"tableCol\">18TB (50%)<\/div>\n<div class=\"tableCol\">4<\/div>\n<div class=\"tableCol\">Mirror (2) x 4<\/div>\n<\/p><\/div>\n<div class=\"tableRow\">\n<div class=\"tableCol\"><a href=\"#raidz2x2\">7.<\/a><\/div>\n<div class=\"tableCol\"><a href=\"#raidz2x2\">RAIDZ2 x 2<\/a><\/div>\n<div class=\"tableCol\">414<\/div>\n<div class=\"tableCol\">98.9698s<\/div>\n<div class=\"tableCol\">441<\/div>\n<div class=\"tableCol\">92.963s<\/div>\n<div class=\"tableCol\">18TB (50%)<\/div>\n<div class=\"tableCol\">4<\/div>\n<div class=\"tableCol\">RAIDZ(2) x 2<\/div>\n<\/p><\/div>\n<\/div>\n<div class=\"seperator\"><\/div>\n<p><a name=\"striped\"><\/a><\/p>\n<h3>Striped<\/h3>\n<p>In this design, we use all disks to store data (i.e., zero data protection), which max out our total usable spaces to 36 TB.<\/p>\n<pre style=\"height:350px;\">\r\n#Command\r\nzpool create -f myzpool hd1 hd2 \\\r\n                        hd3 hd4 \\\r\n                        hd5 hd6 \\\r\n                        hd7 hd8\r\n\r\n#df -h\r\nFilesystem     Size    Used   Avail Capacity  Mounted on\r\nmyzpool         36T      0K     36T       0%  \/myzpool \r\n\r\n#zpool status -v\r\n        NAME        STATE     READ WRITE CKSUM\r\n        myzpool     ONLINE       0     0     0\r\n          hd1       ONLINE       0     0     0\r\n          hd2       ONLINE       0     0     0\r\n          hd3       ONLINE       0     0     0\r\n          hd4       ONLINE       0     0     0\r\n          hd5       ONLINE       0     0     0\r\n          hd6       ONLINE       0     0     0\r\n          hd7       ONLINE       0     0     0\r\n          hd8       ONLINE       0     0     0\r\n\r\n<\/pre>\n<div align=\"center\">\n<a href=\"\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-Stripped.jpg\"><br \/>\n<img decoding=\"async\" data-attachment-id=\"3757\" data-permalink=\"https:\/\/icesquare.com\/wordpress\/zfs-performance-mirror-vs-raidz-vs-raidz2-vs-raidz3-vs-striped\/zfs-mirror-2\/#main\" data-orig-file=\"https:\/\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-Mirror1.jpg\" data-orig-size=\"850,525\" data-comments-opened=\"1\" data-image-title=\"ZFS-Mirror\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-Mirror1.jpg\" src=\"\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-Stripped.jpg\" alt=\"ZFS-Striped\" width=\"550\" class=\"aligncenter size-full wp-image-3757 simpleFrame\"  \/><br \/>\n(Click to enlarge)<br \/>\n<\/a>\n<\/div>\n<p>And here is the test result:<\/p>\n<pre>\r\n#Write Test\r\ndd if=\/myzpool\/data\/file.out of=\/dev\/null bs=4096\r\n40960000000 bytes (41 GB) copied, 58.111 s, 705 MB\/s\r\n\r\n#Read Test\r\ndd if=\/myzpool\/data\/file.out of=\/dev\/null bs=4096\r\n40960000000 bytes (41 GB) copied, 59.6386 s, 687 MB\/s\r\n<\/pre>\n<div class=\"seperator\"><\/div>\n<p><a name=\"raidz1x2\"><\/a><\/p>\n<h3>RAIDZ x 2<\/h3>\n<p>In this design, we split the data into two groups. In each group, we store the data in a RAIDZ1 structure. This is similar to RAIDZ2 in terms of data protection, except that this design supports up to one failure disk in each group (local scale), while RAIDZ2 allows ANY two failure disks overall (global scale). Since we use two disks for parity purpose, the usable space drops from 36TB to 26TB.<\/p>\n<pre style=\"height:350px;\">\r\n#Command\r\nzpool create -f myzpool raidz hd1 hd2 hd3 hd4 \\\r\n                        raidz hd5 hd6 hd7 hd8\r\n\r\n#df -h\r\nFilesystem     Size    Used   Avail Capacity  Mounted on\r\nmyzpool         26T      0K     26T       0%  \/myzpool \r\n\r\n#zpool status -v\r\n        NAME        STATE     READ WRITE CKSUM\r\n        myzpool     ONLINE       0     0     0\r\n          raidz1-0  ONLINE       0     0     0\r\n            hd1     ONLINE       0     0     0\r\n            hd2     ONLINE       0     0     0\r\n            hd3     ONLINE       0     0     0\r\n            hd4     ONLINE       0     0     0\r\n          raidz1-1  ONLINE       0     0     0\r\n            hd5     ONLINE       0     0     0\r\n            hd6     ONLINE       0     0     0\r\n            hd7     ONLINE       0     0     0\r\n            hd8     ONLINE       0     0     0\r\n\r\n\r\n<\/pre>\n<div align=\"center\">\n<a href=\"\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-RAIDZ1x2.jpg\"><br \/>\n<img decoding=\"async\" data-attachment-id=\"3757\" data-permalink=\"https:\/\/icesquare.com\/wordpress\/zfs-performance-mirror-vs-raidz-vs-raidz2-vs-raidz3-vs-striped\/zfs-mirror-2\/#main\" data-orig-file=\"https:\/\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-Mirror1.jpg\" data-orig-size=\"850,525\" data-comments-opened=\"1\" data-image-title=\"ZFS-Mirror\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-Mirror1.jpg\" src=\"\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-RAIDZ1x2.jpg\" alt=\"ZFS-RAIDZ x 2\" width=\"550\" class=\"aligncenter size-full wp-image-3757 simpleFrame\"  \/><br \/>\n(Click to enlarge)<br \/>\n<\/a>\n<\/div>\n<p>And here is the test result:<\/p>\n<pre>\r\n#Write Test\r\ndd if=\/dev\/zero of=\/storage\/data\/file.out bs=4096 count=10000000\r\n40960000000 bytes (41 GB) copied, 61.1401 s, 670 MB\/s\r\n\r\n#Read Test\r\ndd if=\/storage\/data\/file.out of=\/dev\/null bs=4096\r\n40960000000 bytes (41 GB) copied, 60.2457 s, 680 MB\/s\r\n\r\n\r\n<\/pre>\n<div class=\"seperator\"><\/div>\n<p><a name=\"raidz2\"><\/a><\/p>\n<h3>RAIDZ2<\/h3>\n<p>In this design, we use two disks for data protection. This allow up to two disks fail without losing any data. The usable space will drop from 36TB to 25TB.<\/p>\n<pre style=\"height:350px;\">\r\n#Command\r\nzpool create -f myzpool raidz2 hd1 hd2 hd3 hd4 \\\r\n                               hd5 hd6 hd7 hd8\r\n\r\n#df -h\r\nFilesystem     Size    Used   Avail Capacity  Mounted on\r\nmyzpool         25T     31K     25T       0%  \/myzpool \r\n\r\n#zpool status -v\r\n        NAME        STATE     READ WRITE CKSUM\r\n        myzpool     ONLINE       0     0     0\r\n          raidz2-0  ONLINE       0     0     0\r\n            hd1     ONLINE       0     0     0\r\n            hd2     ONLINE       0     0     0\r\n            hd3     ONLINE       0     0     0\r\n            hd4     ONLINE       0     0     0\r\n            hd5     ONLINE       0     0     0\r\n            hd6     ONLINE       0     0     0\r\n            hd7     ONLINE       0     0     0\r\n            hd8     ONLINE       0     0     0\r\n\r\n<\/pre>\n<div align=\"center\">\n<a href=\"\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-RAIDZ2.jpg\"><br \/>\n<img decoding=\"async\" data-attachment-id=\"3757\" data-permalink=\"https:\/\/icesquare.com\/wordpress\/zfs-performance-mirror-vs-raidz-vs-raidz2-vs-raidz3-vs-striped\/zfs-mirror-2\/#main\" data-orig-file=\"https:\/\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-Mirror1.jpg\" data-orig-size=\"850,525\" data-comments-opened=\"1\" data-image-title=\"ZFS-Mirror\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-Mirror1.jpg\" src=\"\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-RAIDZ2.jpg\" alt=\"ZFS-RAIDZ2\" width=\"550\" class=\"aligncenter size-full wp-image-3757 simpleFrame\"  \/><br \/>\n(Click to enlarge)<br \/>\n<\/a>\n<\/div>\n<p>And here is the test result:<\/p>\n<pre>\r\n#Write Test\r\ndd if=\/dev\/zero of=\/storage\/data\/file.out bs=4096 count=10000000\r\n40960000000 bytes (41 GB) copied, 67.3897 s, 608 MB\/s\r\n\r\n#Read Test\r\ndd if=\/storage\/data\/file.out of=\/dev\/null bs=4096\r\n40960000000 bytes (41 GB) copied, 60.8205 s, 673 MB\/s\r\n<\/pre>\n<div class=\"seperator\"><\/div>\n<p><a name=\"raidz1\"><\/a><\/p>\n<h3>RAIDZ1<\/h3>\n<p>In this design, we use one disk for data protection. This allow up to one disk fails without losing any data. The usable space will drop from 36TB to 30TB.<\/p>\n<pre style=\"height:350px;\">\r\n#Command\r\nzpool create -f myzpool raidz hd1 hd2 hd3 hd4 \\\r\n                              hd5 hd6 hd7 hd8\r\n\r\n#df -h\r\nFilesystem     Size    Used   Avail Capacity  Mounted on\r\nmyzpool         30T      0K     30T       0%  \/myzpool \r\n\r\n#zpool status -v\r\n        NAME        STATE     READ WRITE CKSUM\r\n        myzpool     ONLINE       0     0     0\r\n          raidz1-0  ONLINE       0     0     0\r\n            hd1     ONLINE       0     0     0\r\n            hd2     ONLINE       0     0     0\r\n            hd3     ONLINE       0     0     0\r\n            hd4     ONLINE       0     0     0\r\n            hd5     ONLINE       0     0     0\r\n            hd6     ONLINE       0     0     0\r\n            hd7     ONLINE       0     0     0\r\n            hd8     ONLINE       0     0     0\r\n\r\n<\/pre>\n<div align=\"center\">\n<a href=\"\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-RAIDZ1.jpg\"><br \/>\n<img decoding=\"async\" data-attachment-id=\"3757\" data-permalink=\"https:\/\/icesquare.com\/wordpress\/zfs-performance-mirror-vs-raidz-vs-raidz2-vs-raidz3-vs-striped\/zfs-mirror-2\/#main\" data-orig-file=\"https:\/\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-Mirror1.jpg\" data-orig-size=\"850,525\" data-comments-opened=\"1\" data-image-title=\"ZFS-Mirror\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-Mirror1.jpg\" src=\"\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-RAIDZ1.jpg\" alt=\"ZFS-RAIDZ\" width=\"550\" class=\"aligncenter size-full wp-image-3757 simpleFrame\"  \/><br \/>\n(Click to enlarge)<br \/>\n<\/a>\n<\/div>\n<p>And here is the test result:<\/p>\n<pre>\r\n#Write Test\r\ndd if=\/dev\/zero of=\/storage\/data\/file.out bs=4096 count=10000000\r\n40960000000 bytes (41 GB) copied, 67.8107 s, 604 MB\/s\r\n\r\n#Read Test\r\ndd if=\/storage\/data\/file.out of=\/dev\/null bs=4096\r\n40960000000 bytes (41 GB) copied, 64.8782 s, 631 MB\/s\r\n\r\n\r\n<\/pre>\n<div class=\"seperator\"><\/div>\n<p><a name=\"raidz3\"><\/a><\/p>\n<h3>RAIDZ3<\/h3>\n<p>In this design, we use three disks for data protection. This allow up to three disks fail without losing any data. The usable space will drop from 36TB to 21TB.<\/p>\n<pre style=\"height:350px;\">\r\n#Command\r\nzpool create -f myzpool raidz3 hd1 hd2 hd3 hd4 \\\r\n                               hd5 hd6 hd7 hd8\r\n\r\n#df -h\r\nFilesystem     Size    Used   Avail Capacity  Mounted on\r\nmyzpool         21T     31K     21T     0%    \/myzpool \r\n\r\n#zpool status -v\r\n        NAME        STATE     READ WRITE CKSUM\r\n        myzpool     ONLINE       0     0     0\r\n          raidz3-0  ONLINE       0     0     0\r\n            hd1     ONLINE       0     0     0\r\n            hd2     ONLINE       0     0     0\r\n            hd3     ONLINE       0     0     0\r\n            hd4     ONLINE       0     0     0\r\n            hd5     ONLINE       0     0     0\r\n            hd6     ONLINE       0     0     0\r\n            hd7     ONLINE       0     0     0\r\n            hd8     ONLINE       0     0     0\r\n\r\n\r\n<\/pre>\n<div align=\"center\">\n<a href=\"\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-RAIDZ3.jpg\"><br \/>\n<img decoding=\"async\" data-attachment-id=\"3757\" data-permalink=\"https:\/\/icesquare.com\/wordpress\/zfs-performance-mirror-vs-raidz-vs-raidz2-vs-raidz3-vs-striped\/zfs-mirror-2\/#main\" data-orig-file=\"https:\/\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-Mirror1.jpg\" data-orig-size=\"850,525\" data-comments-opened=\"1\" data-image-title=\"ZFS-Mirror\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-Mirror1.jpg\" src=\"\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-RAIDZ3.jpg\" alt=\"ZFS-RAIDZ3\" width=\"550\" class=\"aligncenter size-full wp-image-3757 simpleFrame\"  \/><br \/>\n(Click to enlarge)<br \/>\n<\/a>\n<\/div>\n<p>And here is the test result:<\/p>\n<pre>\r\n#Write Test\r\ndd if=\/dev\/zero of=\/storage\/data\/file.out bs=4096 count=10000000\r\n40960000000 bytes (41 GB) copied, 77.549 s, 528 MB\/s\r\n\r\n#Read Test\r\ndd if=\/storage\/data\/file.out of=\/dev\/null bs=4096\r\n40960000000 bytes (41 GB) copied, 70.9604 s, 577 MB\/s\r\n\r\n\r\n<\/pre>\n<div class=\"seperator\"><\/div>\n<p><a name=\"mirror\"><\/a><\/p>\n<h3>Mirror<\/h3>\n<p>In this design, we use half of our disks for data protection, which makes our total usable spaces drop from 36 TB to 18 TB.<\/p>\n<pre style=\"height:350px;\">\r\n#Command\r\nzpool create -f myzpool mirror hd1 hd2 \\\r\n                        mirror hd3 hd4 \\\r\n                        mirror hd5 hd6 \\\r\n                        mirror hd7 hd8\r\n\r\n#df -h\r\nFilesystem     Size    Used   Avail Capacity  Mounted on\r\nmyzpool         18T     31K     18T       0%  \/myzpool \r\n\r\n#zpool status -v\r\n        NAME        STATE     READ WRITE CKSUM\r\n        myzpool     ONLINE       0     0     0\r\n          mirror-0  ONLINE       0     0     0\r\n            hd1     ONLINE       0     0     0\r\n            hd2     ONLINE       0     0     0\r\n          mirror-1  ONLINE       0     0     0\r\n            hd3     ONLINE       0     0     0\r\n            hd4     ONLINE       0     0     0\r\n          mirror-2  ONLINE       0     0     0\r\n            hd5     ONLINE       0     0     0\r\n            hd6     ONLINE       0     0     0\r\n          mirror-3  ONLINE       0     0     0\r\n            hd7     ONLINE       0     0     0\r\n            hd8     ONLINE       0     0     0\r\n          \r\n<\/pre>\n<div align=\"center\">\n<a href=\"\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-Mirror1.jpg\"><br \/>\n<img decoding=\"async\" data-attachment-id=\"3757\" data-permalink=\"https:\/\/icesquare.com\/wordpress\/zfs-performance-mirror-vs-raidz-vs-raidz2-vs-raidz3-vs-striped\/zfs-mirror-2\/#main\" data-orig-file=\"https:\/\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-Mirror1.jpg\" data-orig-size=\"850,525\" data-comments-opened=\"1\" data-image-title=\"ZFS-Mirror\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-Mirror1.jpg\" src=\"\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-Mirror1.jpg\" alt=\"ZFS-Mirror\" width=\"550\" class=\"aligncenter size-full wp-image-3757 simpleFrame\" srcset=\"https:\/\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-Mirror1.jpg 850w, https:\/\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/09\/ZFS-Mirror1-300x185.jpg 300w\" sizes=\"(max-width: 850px) 100vw, 850px\" \/><br \/>\n(Click to enlarge)<br \/>\n<\/a>\n<\/div>\n<p>And here is the test result:<\/p>\n<pre>\r\n#Write Test\r\ndd if=\/dev\/zero of=\/storage\/data\/file.out bs=4096 count=10000000\r\n40960000000 bytes (41 GB) copied, 86.6451 s, 473 MB\/s\r\n\r\n#Read Test\r\ndd if=\/storage\/data\/file.out of=\/dev\/null bs=4096\r\n40960000000 bytes (41 GB) copied, 68.4477 s, 598 MB\/s\r\n\r\n\r\n<\/pre>\n<div class=\"seperator\"><\/div>\n<p><a name=\"raidz2x2\"><\/a><\/p>\n<h3>RAIDZ2 x 2<\/h3>\n<p>In this design, we split the data into two groups. In each group, we store the data in a RAIDZ2 structure. Since we use two disks for parity purpose, the usable space drops from 36TB to 18TB.<\/p>\n<pre style=\"height:350px;\">\r\n#Command\r\nzpool create -f myzpool raidz2 hd1 hd2 hd3 hd4 \\\r\n                        raidz2 hd5 hd6 hd7 hd8\r\n\r\n#df -h\r\nFilesystem     Size    Used   Avail Capacity  Mounted on\r\nmyzpool         18T      0K     18T       0%  \/myzpool \r\n\r\n#zpool status -v\r\n        NAME        STATE     READ WRITE CKSUM\r\n        myzpool     ONLINE       0     0     0\r\n          raidz2-0  ONLINE       0     0     0\r\n            hd1     ONLINE       0     0     0\r\n            hd2     ONLINE       0     0     0\r\n            hd3     ONLINE       0     0     0\r\n            hd4     ONLINE       0     0     0\r\n          raidz2-1  ONLINE       0     0     0\r\n            hd5     ONLINE       0     0     0\r\n            hd6     ONLINE       0     0     0\r\n            hd7     ONLINE       0     0     0\r\n            hd8     ONLINE       0     0     0\r\n\r\n\r\n<\/pre>\n<p>And here is the test result:<\/p>\n<pre>\r\n#Write Test\r\ndd if=\/dev\/zero of=\/storage\/data\/file.out bs=4096 count=10000000\r\n40960000000 bytes (41 GB) copied, 98.9698 s, 414 MB\/s\r\n\r\n#Read Test\r\ndd if=\/storage\/data\/file.out of=\/dev\/null bs=4096\r\n40960000000 bytes (41 GB) copied, 92.963 s, 441 MB\/s\r\n\r\n\r\n<\/pre>\n<div class=\"seperator\"><\/div>\n<h3>Summary<\/h3>\n<p>I am not surprised that the striped layout offers the fastest writing speed and maximum storage space. The only drawback is zero data protection. Unless you mirror the data at the server level (e.g., Hadoop), or the data is not important, otherwise I won&#8217;t recommend you to use this design.<\/p>\n<p>Personally I recommend to go with <a href=\"#raidz1x2\">Striped RAIDZ<\/a>, i.e., we try to make multiple RAIDZ vdev, and each vdev has no more than 5 disks. In theory, ZFS recommends the number of disks in each vdev is no more than 8 to 9 disks. Based on my experience, ZFS will slow down when it has about 30% free space left if we have too many disks in one single vdev. <\/p>\n<p>So which design you should use? Here is my recommendation:<\/p>\n<pre>\r\n#Do you care your data?\r\nNo: Go with striped.\r\nYes: See below:\r\n\r\n#How many disks do you have?\r\n1:     ZFS is not for you.\r\n2:     Mirror\r\n3-5:   RAIDZ1\r\n6-10   RAIDZ1 x 2\r\n10-15: RAIDZ1 x 3\r\n16-20: RAIDZ1 x 4\r\n<\/pre>\n<p>And yes, you can pretty much forget about RAIDZ2, RAIDZ3 and mirror if you need speed and data protection together.<\/p>\n<p>So, you may ask a question, what should I do if there are more than one hard drive fail? The answer is: You need to keep an eye on the health of your ZFS pool every day. I have been managing over 60 servers since 2009, and I&#8217;ve used only RAIDZ1 with my consumer level harddrives (most of them actually was taken from the external harddrives). So far I don&#8217;t have any data lost.<\/p>\n<pre>\r\nsudo zpool status -v\r\n\r\nor\r\n\r\nsudo zpool status -v | grep 'state: ONLINE'\r\n\r\n<\/pre>\n<p>Simply write a program to get the result from this command, and send yourself an email if there is anything go wrong. You can include the program in your cron job and have it run daily or hourly. This is my version:<\/p>\n<pre>\r\n#!\/bin\/bash\r\n\r\nresult=`sudo zpool status -x`\r\n\r\nif [[ $result != 'all pools are healthy' ]]; then\r\n        echo \"Something is wrong.\"\r\n        #Do something here such as send an email, such as sending an email via HTTP...\r\n        \/usr\/bin\/wget \"http:\/\/example.com\/send_email.php?subject=Alert&body=File%20System%20Has%20Problem\" -O \/dev\/null > \/dev\/null\r\n        exit 1;\r\nfi\r\n\r\n<\/pre>\n<p>Enjoy ZFS.<\/p>\n<p>&#8211;Derrick<\/p>\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>I always wanted to find out the performance difference among different ZFS types, such as mirror, RAIDZ, RAIDZ2, RAIDZ3, Striped, two RAIDZ vdevs vs one RAIDZ2 vdev etc. So I <a class=\"more-link\" href=\"https:\/\/icesquare.com\/wordpress\/zfs-performance-mirror-vs-raidz-vs-raidz2-vs-raidz3-vs-striped\/\">Continue Reading &rarr;<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":true,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_hide_featured_image":false,"_jetpack_featured_image_duplicate_dismissed":0,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[94,7,144,6,109],"tags":[177],"class_list":["post-3743","post","type-post","status-publish","format-standard","hentry","category-computer-problems-computer-help-pc-support","category-freebsd","category-sever-problem","category-technology","category-zfs","tag-zfs"],"jetpack_publicize_connections":[],"jetpack_shortlink":"https:\/\/wp.me\/p1sqzr-Yn","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":3914,"url":"https:\/\/icesquare.com\/wordpress\/zfs-linux-vs-freebsd\/","url_meta":{"origin":3743,"position":0},"title":"Running ZFS on Linux: Things you should know and be aware of","author":"Derrick","date":"February 29, 2016","format":false,"excerpt":"First Published: Feb 29, 2016 Last Revised: January 10, 2026 ZFS is a next-generation file system renowned for its reliability and advanced features. However, due to legal and licensing issues, it is not included by default in most Linux distributions. Fortunately, you can still install (ZFS on Linux) with just\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4725,"url":"https:\/\/icesquare.com\/wordpress\/amazon-ec2-vs-google-cloud-platform-storage-speed-comparison\/","url_meta":{"origin":3743,"position":1},"title":"Amazon EC2 VS Google Cloud Platform: Storage Speed Comparison","author":"Derrick","date":"April 1, 2020","format":false,"excerpt":"We've owned multiple cloud instances on both Amazon ECS and Google Cloud Platform. I always wonder what is the difference between them. So I decide to perform a very simple speed comparisons. All storage\/disks are attached on RHEL Linux instance and formatted to XFS. Everything are using the default settings.\u2026","rel":"","context":"In &quot;Technology&quot;","block_context":{"text":"Technology","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1214,"url":"https:\/\/icesquare.com\/wordpress\/building-a-super-large-and-reliable-file-server-with-mixed-size-of-harddisks\/","url_meta":{"origin":3743,"position":2},"title":"Building a Super Large and Reliable File Server with Mixed Size of Harddisks","author":"Derrick","date":"January 17, 2011","format":false,"excerpt":"In this article, I am going to show you how to build a super large, ultra reliable, expandable file server with mixed size of hard drives. Here is my criteria: Large Storage Capacity. At this moment, the largest hard drive is 2TB. I am looking for at least 10TB. Great\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"\/wordpress\/wp-content\/uploads\/2011\/01\/zfs_logo-150x150.jpg","width":350,"height":200},"classes":[]},{"id":3438,"url":"https:\/\/icesquare.com\/wordpress\/zfs-performance-boostimprovement-how-i-push-the-io-speed-to-126mbps\/","url_meta":{"origin":3743,"position":3},"title":"ZFS Performance Boost\/Improvement: How I push the I\/O speed to 126MBps","author":"Derrick","date":"January 25, 2014","format":false,"excerpt":"This article is part of my main ZFS tutorial: How to improve ZFS performance. That article covers everything you need. If you already have the basic knowledge, or you just want to know how I push the I\/O speed to 120+ MBps, you can skip that article and read this\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/02\/speedometer.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/02\/speedometer.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/02\/speedometer.jpg?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/02\/speedometer.jpg?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/02\/speedometer.jpg?resize=1050%2C600&ssl=1 3x"},"classes":[]},{"id":4286,"url":"https:\/\/icesquare.com\/wordpress\/how-to-install-zfs-on-rhel-centos-7\/","url_meta":{"origin":3743,"position":4},"title":"How to install ZFS on RHEL \/ CentOS 7","author":"Derrick","date":"January 4, 2017","format":false,"excerpt":"A friend of mine likes to try ZFS on CentOS 7, therefore I decide to make a guide for him. The following instructions have been well tested on CentOS 7. Before you decide to put ZFS in a production use, you should be aware of the following: ZFS is originally\u2026","rel":"","context":"In &quot;Everyday Computer Problem&quot;","block_context":{"text":"Everyday Computer Problem","link":"https:\/\/icesquare.com\/wordpress\/category\/everyday-computer-problem\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2011\/01\/zfs_logo.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":3551,"url":"https:\/\/icesquare.com\/wordpress\/how-to-expand-zfs\/","url_meta":{"origin":3743,"position":5},"title":"How to Expand ZFS","author":"Derrick","date":"June 23, 2014","format":false,"excerpt":"Today I am going to share my story with you on how to expand my ZFS storage. I have a giant ZFS pool, which consists of 8x2TB (RAIDZ2) and 4x1.5TB (RAIDZ1), with a total of 15TB usable spaces. Since I am running out of space, I decide to upgrade the\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/3743","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/comments?post=3743"}],"version-history":[{"count":146,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/3743\/revisions"}],"predecessor-version":[{"id":4873,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/3743\/revisions\/4873"}],"wp:attachment":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/media?parent=3743"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/categories?post=3743"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/tags?post=3743"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}