{"id":2689,"date":"2012-02-14T10:36:45","date_gmt":"2012-02-14T16:36:45","guid":{"rendered":"http:\/\/icesquare.com\/wordpress\/?p=2689"},"modified":"2021-01-17T23:50:03","modified_gmt":"2021-01-18T05:50:03","slug":"zfs-compression-vs-deduplication-dedup-in-simple-english","status":"publish","type":"post","link":"https:\/\/icesquare.com\/wordpress\/zfs-compression-vs-deduplication-dedup-in-simple-english\/","title":{"rendered":"ZFS: Compression VS Deduplication (Dedup) in Simple English"},"content":{"rendered":"<style>\n.separator{\n   border:1px dashed #999;\n   margin:20px;\n}\n<\/style>\n<p>Last Edited: Jan 17, 2021<\/p>\n<p>Many people are confused between the ZFS compression and ZFS deduplication because they are so similar. Both of them are designed to reduce the size of the data being stored in the storage. Let me explain the difference between them in simple English.<\/p>\n<div style=\"margin-top:100px;\"><\/div>\n<h3>1. This is how your data looks like originally (Assuming only one unique file):<\/h3>\n<p><img decoding=\"async\" src=\"\/wordpress\/wp-content\/uploads\/2012\/02\/apple_big.png\"\/><\/p>\n<div class=\"separator\"><\/div>\n<h3>2. This is how your data look like after being stored in a ZFS pool with compression enabled.<\/h3>\n<p><img decoding=\"async\" src=\"\/wordpress\/wp-content\/uploads\/2012\/02\/apple_small.png\"\/><\/p>\n<div class=\"separator\"><\/div>\n<h3>3. This is how your data look like after being stored in a ZFS pool with deduplication enabled.<\/h3>\n<p><img decoding=\"async\" src=\"\/wordpress\/wp-content\/uploads\/2012\/02\/apple_big.png\"\/><\/p>\n<div class=\"separator\"><\/div>\n<h3>4. Let say we are storing three identical files, i.e., <\/h3>\n<p><img decoding=\"async\" src=\"\/wordpress\/wp-content\/uploads\/2012\/02\/apple_big.png\"\/><img decoding=\"async\" src=\"\/wordpress\/wp-content\/uploads\/2012\/02\/apple_big.png\"\/><img decoding=\"async\" src=\"\/wordpress\/wp-content\/uploads\/2012\/02\/apple_big.png\"\/><\/p>\n<div class=\"separator\"><\/div>\n<h3>5. ZFS: Compression Only<\/h3>\n<p><img decoding=\"async\" src=\"\/wordpress\/wp-content\/uploads\/2012\/02\/apple_small.png\"\/><img decoding=\"async\" src=\"\/wordpress\/wp-content\/uploads\/2012\/02\/apple_small.png\"\/><img decoding=\"async\" src=\"\/wordpress\/wp-content\/uploads\/2012\/02\/apple_small.png\"\/><\/p>\n<div class=\"separator\"><\/div>\n<h3>6. ZFS: Deduplication Only<\/h3>\n<p><img decoding=\"async\" src=\"\/wordpress\/wp-content\/uploads\/2012\/02\/apple_big.png\"\/><\/p>\n<div class=\"separator\"><\/div>\n<h3>7. ZFS: Compression + Deduplication<\/h3>\n<p><img decoding=\"async\" src=\"\/wordpress\/wp-content\/uploads\/2012\/02\/apple_small.png\"\/><\/p>\n<div style=\"margin-top:100px;\"><\/div>\n<p>The biggest difference between deduplication and compression is the scope. File compression works at the file level. For example, if you have three identical files, ZFS will store the compressed files three times. Deduplication works at the block level. A block is simply the basic unit of the ZFS storage (e.g., 512 bytes, 4k etc). Imagine ZFS needs to store a big file. What it will do is to divide the file into multiple chunks. Each chunk will be stored in a block. What deduplication does is to remember the content of each block (checksum), and avoid storing the same content again. In other words, deduplication works at a narrower level (think of it as a molecule).<\/p>\n<p>One of the reasons why the drug are usually tested on mice because some of mouse genes are 99% identical to human genes. Imagine we need to store the mouse genes into the database. All we need is to store the gene of mouse once. Later on if we need to store the human genes into the database, we can reference the mouse one rather than storing the same copy again.<\/p>\n<p>Of course, enabling both compression and deduplication will save lots of free space. However, it comes with a very high price tag. If you like to enable deduplication, you need to make sure that you have at least 2GB of memory per 1TB of storage. For example, if your ZFS pool is 10TB, you need to have 20GB of memory installed in your system. Otherwise, you will experience a huge performance hit.<\/p>\n<p>Hope this article helps you to understand the difference between compression and deduplication.<\/p>\n<p>&#8211;Derrick<\/p>\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Last Edited: Jan 17, 2021 Many people are confused between the ZFS compression and ZFS deduplication because they are so similar. Both of them are designed to reduce the size <a class=\"more-link\" href=\"https:\/\/icesquare.com\/wordpress\/zfs-compression-vs-deduplication-dedup-in-simple-english\/\">Continue Reading &rarr;<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_hide_featured_image":false,"_jetpack_featured_image_duplicate_dismissed":0,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[94,144,6],"tags":[149,150,151,177],"class_list":["post-2689","post","type-post","status-publish","format-standard","hentry","category-computer-problems-computer-help-pc-support","category-sever-problem","category-technology","tag-compression","tag-dedup","tag-deduplication","tag-zfs"],"jetpack_publicize_connections":[],"jetpack_shortlink":"https:\/\/wp.me\/p1sqzr-Hn","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":3447,"url":"https:\/\/icesquare.com\/wordpress\/zfs-compression-lz4-vs-lzjb\/","url_meta":{"origin":2689,"position":0},"title":"ZFS Compression: lz4 VS lzjb","author":"Derrick","date":"January 26, 2014","format":false,"excerpt":"ZFS offers a new compression method in the latest version: lz4. It claims to be better than lzjb. Since lzjb is pretty good already, I am curious to find out how good will lz4 be comparing to lzjb. According to my tests, lz4 is performing better than lzjb in terms\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1380,"url":"https:\/\/icesquare.com\/wordpress\/how-to-improve-zfs-performance\/","url_meta":{"origin":2689,"position":1},"title":"How to Improve ZFS performance","author":"Derrick","date":"August 24, 2014","format":false,"excerpt":"First Edition: March 14, 2010 Last Edited: January 10, 2026 \u00a0 In this tutorial, I will show you how to improve the performance of your ZFS using the affordable consumer-grade hardware (e.g., Gigabit network card, standard SATA non-SSD hard drives, consumer-grade motherboard etc.). Many people found a problem on their\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2011\/01\/zfs_logo.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":3438,"url":"https:\/\/icesquare.com\/wordpress\/zfs-performance-boostimprovement-how-i-push-the-io-speed-to-126mbps\/","url_meta":{"origin":2689,"position":2},"title":"ZFS Performance Boost\/Improvement: How I push the I\/O speed to 126MBps","author":"Derrick","date":"January 25, 2014","format":false,"excerpt":"This article is part of my main ZFS tutorial: How to improve ZFS performance. That article covers everything you need. If you already have the basic knowledge, or you just want to know how I push the I\/O speed to 120+ MBps, you can skip that article and read this\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/02\/speedometer.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/02\/speedometer.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/02\/speedometer.jpg?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/02\/speedometer.jpg?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2014\/02\/speedometer.jpg?resize=1050%2C600&ssl=1 3x"},"classes":[]},{"id":4286,"url":"https:\/\/icesquare.com\/wordpress\/how-to-install-zfs-on-rhel-centos-7\/","url_meta":{"origin":2689,"position":3},"title":"How to install ZFS on RHEL \/ CentOS 7","author":"Derrick","date":"January 4, 2017","format":false,"excerpt":"A friend of mine likes to try ZFS on CentOS 7, therefore I decide to make a guide for him. The following instructions have been well tested on CentOS 7. Before you decide to put ZFS in a production use, you should be aware of the following: ZFS is originally\u2026","rel":"","context":"In &quot;Everyday Computer Problem&quot;","block_context":{"text":"Everyday Computer Problem","link":"https:\/\/icesquare.com\/wordpress\/category\/everyday-computer-problem\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2011\/01\/zfs_logo.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":1214,"url":"https:\/\/icesquare.com\/wordpress\/building-a-super-large-and-reliable-file-server-with-mixed-size-of-harddisks\/","url_meta":{"origin":2689,"position":4},"title":"Building a Super Large and Reliable File Server with Mixed Size of Harddisks","author":"Derrick","date":"January 17, 2011","format":false,"excerpt":"In this article, I am going to show you how to build a super large, ultra reliable, expandable file server with mixed size of hard drives. Here is my criteria: Large Storage Capacity. At this moment, the largest hard drive is 2TB. I am looking for at least 10TB. Great\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"\/wordpress\/wp-content\/uploads\/2011\/01\/zfs_logo-150x150.jpg","width":350,"height":200},"classes":[]},{"id":4798,"url":"https:\/\/icesquare.com\/wordpress\/vmvirtual-machine-file-vs-shared-folder-performance\/","url_meta":{"origin":2689,"position":5},"title":"[VM]Virtual Machine &#8211; File vs Shared Folder &#8211; Performance","author":"Derrick","date":"August 4, 2020","format":false,"excerpt":"I decide to move my Windows 10 system from a physical environment to a Linux based virtual environment. I am curious about what is the I\/O performance difference between VM image and Shared Folder. The reason why I prefer putting the data at Linux level because I can rsync the\u2026","rel":"","context":"In &quot;Technology&quot;","block_context":{"text":"Technology","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/"},"img":{"alt_text":"","src":"\/wordpress\/wp-content\/uploads\/2020\/08\/Windows10-on-ZFS-Wrote-on-File-on-ZFS.png","width":350,"height":200,"srcset":"\/wordpress\/wp-content\/uploads\/2020\/08\/Windows10-on-ZFS-Wrote-on-File-on-ZFS.png 1x, \/wordpress\/wp-content\/uploads\/2020\/08\/Windows10-on-ZFS-Wrote-on-File-on-ZFS.png 1.5x"},"classes":[]}],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/2689","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/comments?post=2689"}],"version-history":[{"count":37,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/2689\/revisions"}],"predecessor-version":[{"id":4902,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/2689\/revisions\/4902"}],"wp:attachment":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/media?parent=2689"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/categories?post=2689"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/tags?post=2689"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}