{"id":2064,"date":"2011-12-17T16:13:54","date_gmt":"2011-12-17T22:13:54","guid":{"rendered":"http:\/\/icesquare.com\/wordpress\/?p=2064"},"modified":"2012-08-14T18:05:31","modified_gmt":"2012-08-14T23:05:31","slug":"zfsusb-building-a-super-large-server-using-usb-memory-cf-card-and-sd-card","status":"publish","type":"post","link":"https:\/\/icesquare.com\/wordpress\/zfsusb-building-a-super-large-server-using-usb-memory-cf-card-and-sd-card\/","title":{"rendered":"ZFS+USB: Building a Super Large Server Using USB Memory, CF Card and SD Card."},"content":{"rendered":"<div width=\"150\" class=\"alignleft\" style=\"padding:10px;\"><\/div>\n<p>I got a lot of unused USB thumb drives, CF flash card and SD cards sitting in my drawer. The sizes range from 8MB to 8GB. Unlike few years ago, it is getting a lot easier to access to the Internet, so I no longer need to carry my data via memory device any more. Instead, I simply connect to the Internet and the data is with me. That&#8217;s why my USB thumb drivers \/ CF flash cards \/ SD cards have been sitting in my drawer for few years.<\/p>\n<p>I got an idea one day. It would be a waste to let them sitting in my drawer (or waiting to be sent to landfill). Why not I use them to build a file server. At least I can test out whether the idea is doable or not. So here are the candidates:<\/p>\n<ul>\n<li>\n<strong>USB Thumb drives<\/strong><\/p>\n<ol>\n<li>8GB x 2<\/li>\n<li>1GB  x 1<\/li>\n<li>256MB x 2<\/li>\n<\/ol>\n<\/li>\n<li>\n<strong>CF Flash Card<\/strong><\/p>\n<ol>\n<li>2GB x 1<\/li>\n<li>1GB  x 1<\/li>\n<li>512MB x 2<\/li>\n<\/ol>\n<\/li>\n<li>\n<strong>SD Card<\/strong><\/p>\n<ol>\n<li>4GB x 2<\/li>\n<\/ol>\n<\/li>\n<\/ul>\n<p>As you can see from my list, the size of each candidates varies from 256MB to 8GB. So it will be interesting to put them together and build a super large file server.<\/p>\n<p><a href=\"\/wordpress\/wp-content\/uploads\/2011\/12\/CF-PCI-adapter1.png\"><img decoding=\"async\" data-attachment-id=\"2079\" data-permalink=\"https:\/\/icesquare.com\/wordpress\/zfsusb-building-a-super-large-server-using-usb-memory-cf-card-and-sd-card\/cf-pci-adapter-2\/#main\" data-orig-file=\"https:\/\/icesquare.com\/wordpress\/wp-content\/uploads\/2011\/12\/CF-PCI-adapter1.png\" data-orig-size=\"565,384\" data-comments-opened=\"1\" data-image-title=\"CF-PCI-adapter\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/icesquare.com\/wordpress\/wp-content\/uploads\/2011\/12\/CF-PCI-adapter1.png\" src=\"\/wordpress\/wp-content\/uploads\/2011\/12\/CF-PCI-adapter1-300x203.png\" alt=\"\" title=\"CF-PCI-adapter\" width=\"150\" class=\"alignleft size-thumbnail wp-image-2079\" srcset=\"https:\/\/icesquare.com\/wordpress\/wp-content\/uploads\/2011\/12\/CF-PCI-adapter1-300x203.png 300w, https:\/\/icesquare.com\/wordpress\/wp-content\/uploads\/2011\/12\/CF-PCI-adapter1.png 565w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>Most computer has multiple USB ports available. If you don&#8217;t have enough USB port, get a powered USB hub (i.e., the USB hub has it own power unit), it will be more efficient then getting the power from the computer. For the CF card, I use a <a href=\"http:\/\/www.newegg.com\/Product\/Product.aspx?Item=N82E16812186075\" target=\"_blank\">SYBA SY-PCI48001 PCI to Compact Flash Adapter<\/a> to connect my CF Flash cards to my computer. For the SD cards, I simply connect each of them to a Sandisk USB SD card reader.<\/p>\n<div align=\"center\"><\/div>\n<p>Okay, let&#8217;s talk about the software. I am going to use ZFS to implement it, because it is quick and simple. First, connect all devices to your computer, and make sure that your operating systems can recognize all of them. In this tutorial, I am using FreeBSD as a tutorial. However, the idea should be the same in other ZFS ready system, such as Solaris.<\/p>\n<p>Make sure that all USB devices are recognized by your operating system. In FreeBSD, the devices are registered as \/dev\/da* or \/dev\/ad*:<\/p>\n<pre>\r\ndmesg | egrep 'ad|da'\r\n<\/pre>\n<p>Now, you need to think about how to group your devices together. Do you simply want to build a pure USB ZFS pool, or a hybrid hard drive\/USB pool. To keep thing simple, I will go with pure USB ZFS pool.<\/p>\n<p>Suppose I am going to create a pure USB pool, which simply include every device in one single place:<\/p>\n<pre>\r\nzpool create myzpool \/dev\/ad0 \/dev\/ad1 \/dev\/ad2 \/dev\/da0 \/dev\/da1 \/dev\/da2\r\n<\/pre>\n<p>where the ad* and da* are the locations of my devices.<\/p>\n<p>This will create a big pool. When you write some data to this pool, e.g.,<\/p>\n<pre>\r\nsudo dd if=\/dev\/random of=\/myzpool\/test_file count=10g bs=1M\r\n<\/pre>\n<p><\/p>\n<p>The system will simply split the file into multiple chunks, and write all chunks to each USB devices at the same time.<\/p>\n<p>Now let&#8217;s verify the pool information:<\/p>\n<pre>\r\nzpool iostat -v\r\n<\/pre>\n<pre>\r\n               capacity     operations    bandwidth\r\npool         used  avail   read  write   read  write\r\n----------  -----  -----  -----  -----  -----  -----\r\nmyzpool\r\n  ad12       969M   112K      0      0  1.15K  66.4K\r\n  ad13      1.90G   112K      0      0  1.74K  66.4K\r\n  ad14       480M    11K      0      0  3.08K  66.9K\r\n  da0          1G  2.78G      0      0  5.55K  66.1K\r\n  da1          1G  2.78G      0      0  5.55K  66.1K\r\n  da2        240M    80K      0      0  2.41K  66.8K\r\n  da3        240M   112K      0      0  1.87K  66.8K\r\n  da4       7.50G    80K      0      0  4.35K  66.5K\r\n  da5       7.50G    96K      0      0  2.98K  66.3K\r\n  da6        972M   112K      0      0    278  39.9K\r\n----------  -----  -----  -----  -----  -----  -----\r\n<\/pre>\n<p>As you can see, the system is split the data and write it to each devices. ZFS is very smart to adjust the number of split to optimize the performance.<\/p>\n<p>Okay, what about the performance? Honestly you can&#8217;t expect too much from a pure-USB zpool, because the write speed is limited to 40MB\/s, which is way too slow compared to the disk. The only advantage is that there is no moving parts, which significant decrease the failure rate, and the overall cost is cheap. Now, let&#8217;s make talk about the hybrid pool, a combination of USB and hard drive pool.<\/p>\n<p>A hybrid ZFS pool is a combination of hard drives and USB drives. In my experiment, I put the USB devices as log and cache devices, while the hard drives are used as main storage. If you don&#8217;t know what is ZFS log or ZFS cache, you can think about a ZFS log devices is a buffer for writing the data, while a ZFS cache is for reading the data.<\/p>\n<p>Ideally, you should use two identical devices (same size) for ZFS log (writing the data). For ZFS cache, it doesn&#8217;t matter.<\/p>\n<p>First, let&#8217;s create our ZFS pool with the storage devices (i.e., hard drives) only.<\/p>\n<pre>\r\nzpool create myzpool raidz \/dev\/ad0 \/dev\/ad1 \/dev\/ad2\r\n<\/pre>\n<p>Next, we need to add the ZFS log. We are going to create a mirror, so that they have to be identical.<\/p>\n<pre>\r\nzpool add myzpool log mirror \/dev\/da0 \/dev\/da1\r\n<\/pre>\n<p>Finally, we add the ZFS cache.<\/p>\n<pre>\r\nzpool add myzpool cache \/dev\/da2 \/dev\/da3 \/dev\/da4\r\n<\/pre>\n<p>And let&#8217;s take a look to the whole picture:<\/p>\n<pre>\r\nzpool iostat -v\r\n<\/pre>\n<pre>\r\n               capacity     operations    bandwidth\r\npool         used  avail   read  write   read  write\r\n----------  -----  -----  -----  -----  -----  -----\r\nmyzpool     5.83T  6.79T      9      2  1.03M   311K\r\n  raidz1    5.83T  6.79T      9      1  1.03M   179K\r\n    ad0         -      -      2      0   171K  29.9K\r\n    ad1         -      -      2      0   171K  29.9K\r\n    ad2         -      -      2      0   171K  29.9K\r\n    ad3         -      -      2      0   171K  29.9K\r\n    ad4         -      -      2      0   171K  29.9K\r\n    ad5         -      -      2      0   171K  29.9K\r\n    ad6         -      -      2      0   171K  29.9K\r\n  da0        128K  3.78G      0      0      0  65.9K\r\n  da1           -      -      0      0      0  65.9K\r\ncache           -      -      -      -      -      -\r\n  da2        961M     8M      0      0  1.14K  66.6K\r\n  da3       1.89G     8M      0      0  1.73K  66.6K\r\n  da4        472M     8M      0      0  3.06K  67.0K\r\n  da5        232M     8M      0      0  2.40K  66.9K\r\n  da6        232M     8M      0      0  1.86K  66.9K\r\n  da7       7.50G     8M      0      0  4.33K  66.6K\r\n  da8       7.50G     8M      0      0  2.96K  66.5K\r\n  da9        964M     8M      0      0    276  39.6K\r\n----------  -----  -----  -----  -----  -----  -----\r\n<\/pre>\n<p>\nWith this combination, I get a pretty good performance (both read\/write). When I copy the data from Windows to this ZFS pool using Samba, I can get a pretty high transfer speed (Over 100MB\/s). Sometimes it get even close to 110MB\/s. This result is very amazing given that my hard drives are the standard SATA drives (non-SSD) only.<\/p>\n<p>\nThe reliability of the USB devices \/ CF card \/ SD card sometimes questionable. That&#8217;s one of the reason why I don&#8217;t use them as the permanent storage media (using as Cache \/ log is okay). In this design, I use two SD cards (4GB x 2 = 8GB) as the ZFS log devices. Since they are set up as mirror, if one dies, the other one will kick in, which will minimize the data lost. For the cache devices, if one device is failed, I can remove it from the ZFS pool at any time. There will be no data lost so it will be okay.<\/p>\n<p>I have run this super large server for few months already. There is about 200GB data I\/O everyday, so far I am very happy with the overall performance. The most important thing is, those unused memory devices are now very happy as they don&#8217;t need to be sent to landfill.<\/p>\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>I got a lot of unused USB thumb drives, CF flash card and SD cards sitting in my drawer. The sizes range from 8MB to 8GB. Unlike few years ago, <a class=\"more-link\" href=\"https:\/\/icesquare.com\/wordpress\/zfsusb-building-a-super-large-server-using-usb-memory-cf-card-and-sd-card\/\">Continue Reading &rarr;<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_hide_featured_image":false,"_jetpack_featured_image_duplicate_dismissed":0,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[94,7,95,144,6,69,109],"tags":[171,139,177],"class_list":["post-2064","post","type-post","status-publish","format-standard","hentry","category-computer-problems-computer-help-pc-support","category-freebsd","category-howto","category-sever-problem","category-technology","category-troubleshooting-technology","category-zfs","tag-freebsd","tag-usb","tag-zfs"],"jetpack_publicize_connections":[],"jetpack_shortlink":"https:\/\/wp.me\/p1sqzr-xi","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":1214,"url":"https:\/\/icesquare.com\/wordpress\/building-a-super-large-and-reliable-file-server-with-mixed-size-of-harddisks\/","url_meta":{"origin":2064,"position":0},"title":"Building a Super Large and Reliable File Server with Mixed Size of Harddisks","author":"Derrick","date":"January 17, 2011","format":false,"excerpt":"In this article, I am going to show you how to build a super large, ultra reliable, expandable file server with mixed size of hard drives. Here is my criteria: Large Storage Capacity. At this moment, the largest hard drive is 2TB. I am looking for at least 10TB. Great\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"\/wordpress\/wp-content\/uploads\/2011\/01\/zfs_logo-150x150.jpg","width":350,"height":200},"classes":[]},{"id":1380,"url":"https:\/\/icesquare.com\/wordpress\/how-to-improve-zfs-performance\/","url_meta":{"origin":2064,"position":1},"title":"How to Improve ZFS performance","author":"Derrick","date":"August 24, 2014","format":false,"excerpt":"First Edition: March 14, 2010 Last Edited: January 10, 2026 \u00a0 In this tutorial, I will show you how to improve the performance of your ZFS using the affordable consumer-grade hardware (e.g., Gigabit network card, standard SATA non-SSD hard drives, consumer-grade motherboard etc.). Many people found a problem on their\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2011\/01\/zfs_logo.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":4648,"url":"https:\/\/icesquare.com\/wordpress\/zfs-on-linux-what-to-do-when-resilver-takes-very-long\/","url_meta":{"origin":2064,"position":2},"title":"[ZFS On Linux] What to Do When Resilver Takes Very Long","author":"Derrick","date":"January 13, 2020","format":false,"excerpt":"If you check your Zpool health status and you notice an error like the following: sudo zpool status pool: myzpool state: DEGRADED status: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3438,"url":"https:\/\/icesquare.com\/wordpress\/zfs-performance-boostimprovement-how-i-push-the-io-speed-to-126mbps\/","url_meta":{"origin":2064,"position":3},"title":"ZFS Performance Boost\/Improvement: How I push the I\/O speed to 126MBps","author":"Derrick","date":"January 25, 2014","format":false,"excerpt":"This article is part of my main ZFS tutorial: How to improve ZFS performance. That article covers everything you need. If you already have the basic knowledge, or you just want to know how I push the I\/O speed to 120+ MBps, you can skip that article and read this\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/02\/speedometer.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/02\/speedometer.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/02\/speedometer.jpg?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/02\/speedometer.jpg?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/02\/speedometer.jpg?resize=1050%2C600&ssl=1 3x"},"classes":[]},{"id":3342,"url":"https:\/\/icesquare.com\/wordpress\/freebsd-zfs-faulted-kernel-ada0ahcich4000-lost-device\/","url_meta":{"origin":2064,"position":4},"title":"[FreeBSD] ZFS FAULTED \/ kernel: (ada0:ahcich4:0:0:0): lost device","author":"Derrick","date":"October 21, 2013","format":false,"excerpt":"I am so happy that I finally solved a problem today. This problem had existed in my server farms for few months already. In the past few months, I had absolutely no idea what was causing the issues. Here is my story: I built a file server using FreeBSD and\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3242,"url":"https:\/\/icesquare.com\/wordpress\/freebsdhow-to-remove-zfs-meta-data\/","url_meta":{"origin":2064,"position":5},"title":"[FreeBSD\/Linux]How To Remove ZFS Meta Data","author":"Derrick","date":"May 14, 2013","format":false,"excerpt":"I have many hard drives circulating among my servers for testing purpose. For example, I took a hard drive from one server and put it on a different server. After doing this for many times, I've noticed that the ZFS has put many header information \/ meta data left on\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2012\/01\/400px-broom_icon.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]}],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/2064","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/comments?post=2064"}],"version-history":[{"count":52,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/2064\/revisions"}],"predecessor-version":[{"id":2856,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/2064\/revisions\/2856"}],"wp:attachment":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/media?parent=2064"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/categories?post=2064"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/tags?post=2064"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}