{"id":3438,"date":"2014-01-25T11:22:59","date_gmt":"2014-01-25T17:22:59","guid":{"rendered":"http:\/\/icesquare.com\/wordpress\/?p=3438"},"modified":"2014-07-06T18:29:44","modified_gmt":"2014-07-06T23:29:44","slug":"zfs-performance-boostimprovement-how-i-push-the-io-speed-to-126mbps","status":"publish","type":"post","link":"https:\/\/icesquare.com\/wordpress\/zfs-performance-boostimprovement-how-i-push-the-io-speed-to-126mbps\/","title":{"rendered":"ZFS Performance Boost\/Improvement: How I push the I\/O speed to 126MBps"},"content":{"rendered":"

This article is part of my main ZFS tutorial: How to improve ZFS performance<\/a>. That article covers everything you need. If you already have the basic knowledge, or you just want to know how I push the I\/O speed to 120+ MBps, you can skip that article and read this one.<\/p>\n

Long story short, here is the result of my iostat:<\/p>\n

\r\nsudo zpool iostat -v\r\n\r\n               capacity     operations    bandwidth\r\npool        alloc   free   read  write   read  write\r\n----------  -----  -----  -----  -----  -----  -----\r\nstorage     7.85T  12.1T      1  1.08K  2.60K   126M < ---126 MBps!\r\n  raidz2    5.51T  8.99T      1    753  2.20K  86.9M\r\n    ada0        -      -      0    142    102  14.5M\r\n    ada1        -      -      0    143    102  14.5M\r\n    ada3        -      -      0    146    307  14.5M\r\n    ada4        -      -      0    145    409  14.5M\r\n    ada5        -      -      0    144    204  14.5M\r\n    ada6        -      -      0    143    204  14.5M\r\n    ada7        -      -      0    172    409  14.5M\r\n    ada8        -      -      0    171    511  14.5M\r\n  raidz1    2.34T  3.10T      0    349    409  39.5M\r\n    da0         -      -      0    169    204  13.2M\r\n    da1         -      -      0    176      0  13.2M\r\n    da2         -      -      0    168    102  13.2M\r\n    da3         -      -      0    173    102  13.2M\r\n----------  -----  -----  -----  -----  -----  -----\r\n<\/pre>\n
I measured the speed while I was transferring 10TB of data from one FreeBSD-based ZFS server to another FreeBSD-based ZFS server, over a consumer-level gigabit network. Basically, every components I used are consumer-level. The hard drives I used are standard PATA and SATA hard drives (i.e., not even the light speed SSD hard drive). The data I transferred are real data (rather than all zeros or ones generated through dd).<\/p>\n
First of all, here are the settings I used. Both server and client have similar configurations.<\/p>\n
\r\nsudo zpool history\r\nHistory for 'storage':\r\n2014-01-22.20:28:59 zpool create -f storage raidz2 \/dev\/ada0 \/dev\/ada1 \/dev\/ada3 \/dev\/ada4 \/dev\/ada5 \/dev\/ada6 \/dev\/ada7 \/dev\/ada8 raidz \/dev\/da0 \/dev\/da1 \/dev\/da2 \/dev\/da3\r\n2014-01-22.20:29:07 zfs create storage\/data\r\n2014-01-22.20:29:15 zfs set compression=lz4 storage\r\n2014-01-22.20:29:19 zfs set compression=lz4 storage\/data\r\n2014-01-22.20:30:19 zfs set atime=off storage\r\n<\/pre>\n
\r\n#where the ad* and da* are nothing more than standard SATA drives:\r\n#ZFS Drives\r\n#8 x 2TB standard SATA hard drives connected to the motherboard\r\n#4 x 1.5TB standard SATA hard drives connected to a PCI-e RAID card\r\n\r\nda0: 1430799MB (2930277168 512 byte sectors: 255H 63S\/T 182401C)\r\nda1: 1430799MB (2930277168 512 byte sectors: 255H 63S\/T 182401C)\r\nda2: 1430799MB (2930277168 512 byte sectors: 255H 63S\/T 182401C)\r\nda3: 1430799MB (2930277168 512 byte sectors: 255H 63S\/T 182401C)\r\nada0: 300.000MB\/s transfers (SATA 2.x, UDMA6, PIO 512bytes)\r\nada0: 1907729MB (3907029168 512 byte sectors: 16H 63S\/T 16383C)\r\nada1: 300.000MB\/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)\r\nada1: 1907729MB (3907029168 512 byte sectors: 16H 63S\/T 16383C)\r\nada3: 300.000MB\/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)\r\nada3: 1907729MB (3907029168 512 byte sectors: 16H 63S\/T 16383C)\r\nada4: 300.000MB\/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)\r\nada4: 1907729MB (3907029168 512 byte sectors: 16H 63S\/T 16383C)\r\nada5: 300.000MB\/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)\r\nada5: 1907729MB (3907029168 512 byte sectors: 16H 63S\/T 16383C)\r\nada6: 300.000MB\/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)\r\nada6: 1907729MB (3907029168 512 byte sectors: 16H 63S\/T 16383C)\r\nada7: 300.000MB\/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)\r\nada7: 1907729MB (3907029168 512 byte sectors: 16H 63S\/T 16383C)\r\nada8: 300.000MB\/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)\r\nada8: 1907729MB (3907029168 512 byte sectors: 16H 63S\/T 16383C)\r\n\r\n\r\n#System Drive\r\n#A PATA 200MB drive from 2005\r\nada2: maxtor STM3200820A 3.AAE ATA-7 device\r\nada2: 100.000MB\/s transfers (UDMA5, PIO 8192bytes)\r\nada2: 190782MB (390721968 512 byte sectors: 16H 63S\/T 16383C)\r\n<\/pre>\n
Here are the corresponding hardware components:<\/p>\n
\r\n#CPU\r\nhw.machine: amd64\r\nhw.model: Intel(R) Core(TM) i7 CPU         920  @ 2.67GHz\r\nhw.ncpu: 8\r\nhw.machine_arch: amd64\r\n\r\n\r\n#Memory - 3 x 2GB\r\nreal memory  = 6442450944 (6144 MB)\r\navail memory = 6144352256 (5859 MB)\r\n\r\n#Network - Gigabit network on the motherboard \/ PCI-e\r\n#If your gigabit network card is PCI, that will be your bottleneck.\r\nre0: realtek 8168\/8111 B\/C\/CP\/D\/DP\/E\/F PCIe Gigabit Ethernet\r\n<\/pre>\n
Other than that, I didn’t really specify any special settings in my OS. Everything else are default:<\/p>\n
\r\n#uname -a\r\nFreeBSD 9.2-RELEASE-p3 FreeBSD 9.2-RELEASE-p3 #0: Sat Jan 11 03:25:02 UTC 2014     root@amd64-builder.daemonology.net:\/usr\/obj\/usr\/src\/sys\/GENERIC  amd64\r\n<\/pre>\n
The way how to transfer 10TB of data is through rsync and rsyncd. First of all, I set up a rsync as daemon on the server. Keep in mind that rsync as a daemon (service) is not the same as rsync. If you want to know how I set up the rsyncd, please visit How to Improve rsync Performance<\/a>.<\/p>\n
After you have set up the rsyncd on the server, I run the following on the client.<\/p>\n
\r\nrsync -av server_ip::rsync_share_name \/my_directory\/\r\n\r\n#Example\r\nrsync -av 192.168.1.101::storage \/mydata\/\r\n<\/pre>\n
Notice that I didn’t enable the compression option. That’s because my files are pretty much compressed (jpeg, zip, tar, 7z etc). If your file types are different, you may want to enable the compression, i.e.,<\/p>\n
\r\nrsync -avz server_name::rsync_share_name \/my_directory\/\r\n<\/pre>\n
Give it a try and see whether it improves the I\/O speed.<\/p>\n
Here are few things I have learned to improve the ZFS performance.<\/p>\n
Keep the ZFS structure small and simple. For example, keep the number of disks in your vdev small. Previously, I set up a giant big pool with 14 disks RAIDZ in one single vdev. This is a bad idea. There is a maximum number of disks for RAIDZ. That’s why I split my disks into two groups:<\/p>\n
\r\n#Group 1 - RAIDZ2\r\n#8 x 2TB standard SATA hard drives connected to the motherboard\r\n\r\n#Group2 - RAIDZ\r\n#4 x 1.5TB standard SATA hard drives connected to a PCI-e RAID card\r\n\r\nzpool create -f storage myzpool group1 raidz group2\r\n<\/pre>\n
Keep the ZFS settings clean. Enable the only thing you need, and disable the junk settings. In my case, I enable the best compression (lz4) and disable the access time:<\/p>\n
\r\n2014-01-22.20:29:15 zfs set compression=lz4 storage\r\n2014-01-22.20:30:19 zfs set atime=off storage\r\n<\/pre>\n
Keep the ZFS clean. I see the performance gain on a brand new ZFS than a few years old ZFS, even both of them contain the same data. Unfortunately, there is no easy way to clean the ZFS. The only way is to destroy the current ZFS first, create a new one and bring the data back. Typically, it takes about 2 days to transfer 10TB of data over a gigabit LAN. So it is really not too bad to spend about 5 days to rebuild your ZFS.<\/p>\n
That’s it. Again, if you want to learn about tricks, please visit How to improve ZFS performance<\/a> for more information.<\/p>\n
–Derrick<\/p>\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
This article is part of my main ZFS tutorial: How to improve ZFS performance. That article covers everything you need. If you already have the basic knowledge, or you just Continue Reading →<\/a><\/p>\n","protected":false},"author":1,"featured_media":3476,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_hide_featured_image":false,"_jetpack_featured_image_duplicate_dismissed":0,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[94,146,7,144,6,69],"tags":[171,177],"class_list":["post-3438","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-computer-problems-computer-help-pc-support","category-everyday-computer-problem","category-freebsd","category-sever-problem","category-technology","category-troubleshooting-technology","tag-freebsd","tag-zfs"],"jetpack_publicize_connections":[],"jetpack_shortlink":"https:\/\/wp.me\/p1sqzr-Ts","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":2064,"url":"https:\/\/icesquare.com\/wordpress\/zfsusb-building-a-super-large-server-using-usb-memory-cf-card-and-sd-card\/","url_meta":{"origin":3438,"position":0},"title":"ZFS+USB: Building a Super Large Server Using USB Memory, CF Card and SD Card.","author":"Derrick","date":"December 17, 2011","format":false,"excerpt":"[ad#adbrite-square] I got a lot of unused USB thumb drives, CF flash card and SD cards sitting in my drawer. The sizes range from 8MB to 8GB. Unlike few years ago, it is getting a lot easier to access to the Internet, so I no longer need to carry my\u2026","rel":"","context":"In "Computer Problems"","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1380,"url":"https:\/\/icesquare.com\/wordpress\/how-to-improve-zfs-performance\/","url_meta":{"origin":3438,"position":1},"title":"How to Improve ZFS performance","author":"Derrick","date":"August 24, 2014","format":false,"excerpt":"First Edition: March 14, 2010 Last Edited: January 10, 2026 \u00a0 In this tutorial, I will show you how to improve the performance of your ZFS using the affordable consumer-grade hardware (e.g., Gigabit network card, standard SATA non-SSD hard drives, consumer-grade motherboard etc.). Many people found a problem on their\u2026","rel":"","context":"In "Computer Problems"","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2011\/01\/zfs_logo.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":4377,"url":"https:\/\/icesquare.com\/wordpress\/zfs-cluster-a-network-based-zfs-implementation\/","url_meta":{"origin":3438,"position":2},"title":"ZFS Cluster: A Network-Based ZFS Implementation","author":"Derrick","date":"June 5, 2017","format":false,"excerpt":"I always like to experimenting the idea of building a ZFS cluster, i.e., it has the robust of the ZFS with the cluster capacity. So I came up a test environment with this prototype. Keep in mind that this is just a proof of concept rather than a production-ready solution.\u2026","rel":"","context":"In "Everyday Computer Problem"","block_context":{"text":"Everyday Computer Problem","link":"https:\/\/icesquare.com\/wordpress\/category\/everyday-computer-problem\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2017\/06\/ZFS-Based-Network.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2017\/06\/ZFS-Based-Network.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2017\/06\/ZFS-Based-Network.jpg?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2017\/06\/ZFS-Based-Network.jpg?resize=700%2C400&ssl=1 2x"},"classes":[]},{"id":1214,"url":"https:\/\/icesquare.com\/wordpress\/building-a-super-large-and-reliable-file-server-with-mixed-size-of-harddisks\/","url_meta":{"origin":3438,"position":3},"title":"Building a Super Large and Reliable File Server with Mixed Size of Harddisks","author":"Derrick","date":"January 17, 2011","format":false,"excerpt":"In this article, I am going to show you how to build a super large, ultra reliable, expandable file server with mixed size of hard drives. Here is my criteria: Large Storage Capacity. At this moment, the largest hard drive is 2TB. I am looking for at least 10TB. Great\u2026","rel":"","context":"In "Computer Problems"","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"\/wordpress\/wp-content\/uploads\/2011\/01\/zfs_logo-150x150.jpg","width":350,"height":200},"classes":[]},{"id":3914,"url":"https:\/\/icesquare.com\/wordpress\/zfs-linux-vs-freebsd\/","url_meta":{"origin":3438,"position":4},"title":"Running ZFS on Linux: Things you should know and be aware of","author":"Derrick","date":"February 29, 2016","format":false,"excerpt":"First Published: Feb 29, 2016 Last Revised: January 10, 2026 ZFS is a next-generation file system renowned for its reliability and advanced features. However, due to legal and licensing issues, it is not included by default in most Linux distributions. Fortunately, you can still install (ZFS on Linux) with just\u2026","rel":"","context":"In "Computer Problems"","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3447,"url":"https:\/\/icesquare.com\/wordpress\/zfs-compression-lz4-vs-lzjb\/","url_meta":{"origin":3438,"position":5},"title":"ZFS Compression: lz4 VS lzjb","author":"Derrick","date":"January 26, 2014","format":false,"excerpt":"ZFS offers a new compression method in the latest version: lz4. It claims to be better than lzjb. Since lzjb is pretty good already, I am curious to find out how good will lz4 be comparing to lzjb. According to my tests, lz4 is performing better than lzjb in terms\u2026","rel":"","context":"In "Computer Problems"","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_featured_media_url":"https:\/\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/02\/speedometer.jpg","_links":{"self":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/3438","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/comments?post=3438"}],"version-history":[{"count":8,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/3438\/revisions"}],"predecessor-version":[{"id":3640,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/3438\/revisions\/3640"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/media\/3476"}],"wp:attachment":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/media?parent=3438"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/categories?post=3438"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/tags?post=3438"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}