ZFS Compression: lz4 VS lzjb

ZFS offers a new compression method in the latest version: lz4. It claims to be better than lzjb. Since lzjb is pretty good already, I am curious to find out how good will lz4 be comparing to lzjb. According to my tests, lz4 is performing better than lzjb in terms of spacing saving and I/O, but not too much.

lz4 VS lzjb: Space Saving

Long story short, here is what I did. I set up two servers with brand new ZFS settings. One server is set to lzjb, and the other server is set to lz4. Both servers have exact the same hardware, software, operating system etc. Both of them are loaded with the same set of data (10TB) via rsync and rsyncd. These data are real data that I use everyday, which includes office documents (PDF, Word, Excel), photos(Raw and JPEG), music(mp3), video(mpeg), zip, source codes, binary applications, database (MySQL, Redis), webserver files (PHP, HTML, CSS), etc. Notice that some of these files are already compressed (such as jpeg, zip etc), and the file types are not evenly distributed (e.g., 40% of the files are jpeg, 25% are docs, 10% are zip, 5% are something else). I want to make a test in a more realistic scenario.

Also, all ZFS settings are set BEFORE loading the data. This will ensure that all data on the same server share the same compression settings. Below is the summary of the used space. Keep in mind that the space saving test is independent to the hardware, such as CPU type, memory, speed of the disks, etc. Just like running the same command to compress the same set of files in two systems. We expect the result will have the same size. The only difference will be the time spent on compressing the data.

#ZFS with lzjb
df
Filesystem    512-blocks        Used     Avail Capacity  Mounted on
storage/data 23007987368 22996620466  11366902   100%    /storage/data

#Used space: 22996620466 blocks = 10965.6 GB
#ZFS with lz4
df
Filesystem    512-blocks        Used      Avail Capacity  Mounted on
storage/data 31527470077 22942284913 8585185164    73%    /storage/data

#Used space: 22942284913 blocks = 10939 GB

I found that for 10TB of data, the server with lzjb uses 26GB more space than lz4, which translate to 0.23% (26GB out of 10TB). The difference is quite small and not too significant. However, when your ZFS is nearly full or you are too busy to upgrade your system, the extra saving may matter a lot.

lz4 VS lzjb: Time Saving

I am also curious to know how much time I will save by switching from lzjb to lz4. Therefore I did another simple test. First, I generated a 1GB of random data and wrote to the lzjb system. After that, I destroyed the zpool and rebuilt the zpool with lz4. This will eliminate other factors such as hardware, network traffic etc, because both tests were done within the same system.

#ZFS with lzjb
time dd if=/dev/random of=/storage/data/file.out bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 23.725955 secs (44195313 bytes/sec)

real    0m24.144s
user    0m0.024s
sys     0m18.326s
#ZFS with lz4
time dd if=/dev/random of=/storage/data/file.out bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 22.589257 secs (46419234 bytes/sec)

real    0m22.802s
user    0m0.016s
sys     0m18.273s

By just upgrading from lzjb to lz4, the time is reduced from 24.144 to 22.802 seconds, which translate to 5.5% (1.342s out of 24.144s) improvement.

This result also agrees with my overall experience: The improvement is small and not noticeable. In fact, I barely notice any performance improvement after switching from lzjb to lz4, which primarily includes transferring the files between Windows and FreeBSD-based ZFS system via Samba on a gigabit network. The I/O speed are about the same. However, if you are talking about a busy server with lots of traffic, a 5% improvement will be something.

Anyway, this is what I recommend. If lz4 is available in your ZFS version, use it. It can’t be wrong.

sudo zfs set compression=lz4 myzpool

–Derrick

Our sponsors:

5 comments

  1. > These data are real data that I use everyday, which includes documents, photos, music, video, zip, source codes, binary etc.

    Most of the documents mentioned in this test list are already compressed, and they won’t be compressed anymore by either lzjb nor lz4, nor gzip. Since none of them will compress anything, the difference will be zero.

    Both lzjb & lz4 are also pretty fast at detecting this situation to skip over it. Although lz4 has a significant advantage at this task, it is likely to be completely dwarfed by bandwidth issues.

    Another metric to look at is CPU usage : when bandwidth is the limiting factor, a faster algorithm will not improve the speed of an operation, but will make it using less CPU, which can be important is this CPU is also useful for other tasks. It’s an important difference for web server and databases for example.

    Bottom line : to feel a storage difference, you need compressible data. The test set describes in this post seems to have too little of it.

    Other comparison tests have been performed and published over Internet. They mention something like a 10% storage gain for lz4 over lzjb, on compressible data. Typical scenario are log files.

    Obviously, servers handling primarily log files are meant for purpose than servers handling primarily video files. Since both cases exist, it’s important to state for what kind of data the test is done, in order to avoid over-generalization of one result.

    1. Hello Sinior,

      Thanks for your feedback. In reality, it is rare that we only put one kind of files in the server. Usually the file types will be mixed. For example, a production web server will contain source code (small, compressible), database (medium to large, compressible), image (already compressed) and user attachments (depends). Notice that they are not evenly distributed. Let’s take one of the production servers I run for work for example.

      The server contains about 10GB of data. The source codes and libraries (compressible) use roughly about 20MB of data, images takes about 200MB of space. The database takes up about 1GB of data. The application logs take about 1GB. The rest are user attachments, which is over 8GB. As you see from this scenario, it is difficult to generate an ideal environment (i.e., every files types are evenly distributed) in reality.

      What do you think?

      –Derrick

  2. You are perfectly right.
    It completely depends on the kind of application the server is providing.

    Whenever user data storage is part of the formula, there is a good probability that “high bandwidth data” get a pretty large share, and they are typically, video first, photo second, and music a distant third, with sometimes compressed packages getting 2nd or 3rd place. Combined, they may well represent over >90% of total data, which means there is very little left to compress.

    But application server can provide quite different services, without the capability to store user data. We can list a little few examples, such as packet analyzer, science probe experiment, purchase accounting, code repository, etc. but the possibilities are limitless (and boring). Let’s just say : in all these cases, data tend to be much more compressible, and scenario with compression ratio above 50% are commonplace.

    In the above example, source code is compressible (but very small), log file is very compressible, database may be compressible, but it depends too much on the type of data and the type of storage strategy, and the rest, user attachment, is likely to be already compressed (images, photos, compressed files, etc.).

    As soon as you have some “user data storage” in the formula, it tends to tilt heavily towards “already compressed data” side.
    For all other cases, the conclusion can be radically different.

  3. Sorry I couldn’t find a contact link. There’s a typo in the sentence just after the two yellow text boxes, regarding speed/time:

    “By just upgrading from lzjb to lz4, the time is reduced from 24.144 to 12.802 seconds, which translate to 5.5% (1.342s out of 24.144s) improvement.”

    12.802 should be 22.802.

Leave a Reply

Your email address will not be published. Required fields are marked *