After upgrading to PHP 5.4.3 (which is the first version of PHP 5.4 available in the FreeBSD port), I found that the pecl-tokyo_tyrant could not be compiled. Here is the error messages:

cd /usr/ports/databases/pecl-tokyo_tyrant
sudo make

/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:1827: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'php_tokyo_tyrant_class_methods'
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:1867: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'php_tokyo_tyrant_table_class_methods'
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:1911: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'php_tokyo_tyrant_query_class_methods'
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:1936: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'php_tokyo_tyrant_iterator_class_methods'
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c: In function 'php_tokyo_tyrant_query_object_new':
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2001: error: 'zend_class_entry' has no member named 'default_properties'
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c: In function 'php_tokyo_tyrant_iterator_object_new':
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2052: error: 'zend_class_entry' has no member named 'default_properties'
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c: In function 'php_tokyo_tyrant_object_new_ex':
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2090: error: 'zend_class_entry' has no member named 'default_properties'
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c: In function 'zm_startup_tokyo_tyrant':
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2166: error: 'php_tokyo_tyrant_class_methods' undeclared (first use in this function)
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2166: error: (Each undeclared identifier is reported only once
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2166: error: for each function it appears in.)
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2171: error: 'php_tokyo_tyrant_table_class_methods' undeclared (first use in this function)
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2176: error: 'php_tokyo_tyrant_query_class_methods' undeclared (first use in this function)
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2182: error: 'php_tokyo_tyrant_iterator_class_methods' undeclared (first use in this function)
*** Error code 1

Stop in /usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0.
*** Error code 1

Stop in /usr/ports/databases/pecl-tokyo_tyrant.

Basically, it means that the pecl-tokyo_tyrant is not happy with the new version of PHP 5.4. I’ve tried several methods, including using pkg_add, compiling from source, copying a working tokyo_tyrant.so etc, unfortunately none of these methods works.

Finally, I found that the maintainer of this package has made a new version available(0.6.2) at github. Here is how I did it:

First, go to here to grab the source code. Make sure that you download the tag.gz version because it has the file attribute available.

Second, let’s remove the pecl-tokyo_tyrant from your system. You can either do it in the port or pkg_delete:

From port:

cd /usr/ports/databases/pecl-tokyo_tyrant
sudo make deinstall

Or using pkg_delete:

sudo pkg_delete pecl-tokyo_tyrant-0.6.0

Now, extract the files and compile it from source:

tar -zxvf mkoppanen-php-tokyo_tyrant-0.6.0-2-gb0d7e0c.tar.gz
cd mkoppanen-php-tokyo_tyrant-b0d7e0c
sudo phpize
sudo ./configure
sudo make
sudo make install

and don’t forget to include the .so file in your configuration:

sudo nano /usr/local/etc/php/extensions.ini

Add the following (if not available) to the end of the file:

extension=tokyo_tyrant.so

Finally, let’s restart the Apache:

sudo apachectl stop
sudo apachectl start

That’s it! Have fun with Tokyo Tyrant with PHP 5.4.

–Derrick

When PHP 5.4 was available few months ago, I was hesitated to upgrade my FreeBSD box to upgrade to 5.4, because it comes with lots of big changes, including removing LOTS of old stuffs that were available in 5.3 or earlier versions.

Today PHP 5.4.3 becomes available in FreeBSD ports. I guess it is stable enough and I should give it a try, and I am glad that I tried it in my test machine first.

Warning: Before you upgrade to PHP 5.4, make sure that you are not using any of the following packages, because they are not compatible with the PHP 5.4. Unfortunately, I don’t have a solution yet. Basically, I could not get them compiled.

• eAccelerator: /usr/ports/www/eaccelerator/
• PECL-TokyoTyrant: /usr/ports/databases/pecl-tokyo_tyrant/ (Updated: I just came up a solution here)

When I performed the upgrade using the following command:

sudo portsnap fetch update
sudo portmaster -Da

I got the following error message:

===>>> The databases/php5-sqlite port has been deleted: Removed from core php
===>>> Aborting update

The reason why you see this problem because php5-sqlite is no longer available in PHP 5.4. In order to solve this problem, you will need to delete the php5-sqlite extension.

cd /usr/port/lang/php5-extensions
sudo make config

Uncheck the following: sqlite3 support

Now, deinstall this port. Yes, let’s remove it and install it again. (I’ve tried the reinstall option and it didn’t work, so let’s stick with deinstall and install.

sudo make deinstall clean install clean

Now let’s find out which php-SQLite we should delete:

pkg_info | grep php5-sqlite

In my case, I deleted this one:

sudo pkg_delete php5-sqlite-5.3.13

Now upgrade the port again and FreeBSD should be happy about it. Enjoy PHP 5.4.

–Derrick

A user complained to me that my PHP web applications stopped working today. After I digged into the error log, I found that following message:

PHP Fatal error:  Allowed memory size of 262144 bytes exhausted (tried to allocate 261900 bytes) in Unknown on line 0

Obviously, this is related to the memory error. Therefore I tried to increase the limit of memory_limit in php.ini. Since I have 8G of memory in my box, I think it is okay to set the memory limit of PHP to 4GB.

;memory_limit = 128M
memory_limit = 4096M

When I restart the Apache/PHP, I found another nightmare. My web application stopped working! When the user tries to access http://mywebsite/index.php, instead of rendering the code, the browser kept prompting the user to download the file.

After scratching my head for a while, I found that I missed something, which is very very important. I was running XAMPP (32-bit) on a Fedora 64-bit machine. See the problem? Since XAMPP is running in 32-bit, it does not understand what is 4096 MB. (If you install 4GB of memory in any Windows 32-bit machine, you will notice that Windows can only allocate 3GB of them, and the 1GB left memory is sitting inside your computer and doing nothing…) Therefore, I decrease the memory_limit to 2048MB, and that solved the problem.

Time to move away from XAMPP.

–Derrick

Recently, I found the following error message when updating my FreeBSD system regarding to the OpenLDAP:

===>  Installing for openldap-client-2.4.31
===>   Generating temporary packing list
===>  Checking if net/openldap24-client already installed
===> Creating users and/or groups.
Using existing group `ldap'.
Creating user `ldap' with uid `389'.
pw: user 'ldap' already exists
*** Error code 74

Stop in /usr/ports/net/openldap24-client.
*** Error code 1

Stop in /usr/ports/net/openldap24-client.

And I don’t use LDAP in particular, however it affects other applications that use LDAP, e.g., PHP

php -v
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20090626/ldap.so' - Shared object "libldap-2.4.so.8" not found, required by "ldap.so" in Unknown on line 0
PHP 5.3.10 with Suhosin-Patch (cli) (built: Feb 14 2012 16:59:15)
Copyright (c) 1997-2012 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2012 Zend Technologies
    with eAccelerator v0.9.6.1, Copyright (c) 2004-2010 eAccelerator, by eAccelerator

It is very simple to solve this problem. All you need is to remove the user ‘ldap’, and install the port again. First, let’s remove the user ‘ldap’ first:

sudo vipw

You will see this line:

ldap:*:389:389::0:0:OpenLDAP Server:/nonexistent:/usr/sbin/nologin

Remove the line by pressing ‘dd’, hit ‘ESC’ button, and then press ‘:wq’ to save the file and quit the editor.

After that, let’s clean the port and run the update again:

cd /usr/ports/net/openldap24-client
sudo make clean
sudo make install clean

That’s it. Your LDAP will be happy and your applications (such as PHP) will not complain.

–Derrick

Recently, I built a FreeBSD server with ZFS file system using six 2TB harddisks. If you like to learn what I have done or understand more about ZFS, you can read the story here: ZFS+USB: Building a Super Large Server Using USB Memory, CF Card and SD Card..

Many people found a problem on their ZFS system. The speed is slow! It is slow to read or write files to the system. In this article, I am going to show you some tips on improving the speed of your ZFS file system.

Improve ZFS Performance: Step 1

A 64-bit decent CPU + Lots of memory

Traditionally, we are told to use a less powerful computer for a file/data server. That’s not true on ZFS. ZFS is more than a file system. It uses a lot of resources to improve the performance of the input/output, such as compression on the fly. For example, suppose you need to write a 1GB file. Without enabling the compression, the system will write the entire 1GB file on the disk. With the compression being enabled, the CPU will compress the data first, and write the data on the disk after that. Since the compressed file is smaller, it takes shorter time to write to the disk, which results a higher writing speed. Same for the reading. ZFS can cache the file for you in the memory, which results a higher reading speed.

That’s why a 64-bit CPU and higher amount of memory is recommended. I recommended at least a Quad Core CPU with 4GB of memory (I personally use i7 + 8GB). Of course, the faster the CPU, the more the memory, the better performance. Of course, you need a 64-bit operating system.

Don’t believe? Let’s do an experiment. Suppose I am going to create a 10GB file with all zero. Let’s see how long does it take to write on the disk:

#CPU: i7 920 (8 cores) + 8GB Memory + FreeBSD 8.2 64-bit
#time dd if=/dev/zero of=./file.out bs=1M count=10k
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 6.138918 secs (1749073364 bytes/sec)

real    0m6.140s
user    0m0.023s
sys     0m3.658s

That’s 1.6GB/s! What is it so fast? That’s because it is a zero based file. After the compression, a compressed 10GB file may result in several bytes only. Since the performance of the compression is highly depended on the CPU, that’s why a fast CPU matters.

Now, let’s do the same thing on a not-so-fast CPU:

#CPU: AMD 4600 (2 cores) + 5GB Memory + FreeBSD 8.2 64-bit
#time dd if=/dev/zero of=./file.out bs=1M count=10k
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 23.672373 secs (453584362 bytes/sec)

real    0m23.675s
user    0m0.091s
sys     0m22.409s

That’s 434MB/s only. See the difference?

Improve ZFS Performance: Step 2

Tweaking the Boot Loader Parameters

Many people complain about ZFS for its stability issues, such as kernel panic, reboot randomly, crash when copying large files (> 2GB) etc. It has something to do with the boot loader settings. By default, ZFS will not work smoothly without tweaking the system parameters system. Even FreeBSD claims that no tweaking is necessary for 64-bit system, my FreeBSD server crashes very often when writing large files to the pool. After trial and error for many times, I figure out an equation. You can tweak your boot loader (/boot/loader.conf) using the following parameters:

#I have 8G of Ram
vfs.zfs.prefetch_disable=0

#If Ram = 4GB, set the value to 512M
#If Ram = 8GB, set the value to 1024M
vfs.zfs.arc_min="1024M"

#Ram x 0.5 - 512 MB
vfs.zfs.arc_max="3584M"

#Ram x 2
vm.kmem_size_max="16G"

#Ram x 1.5
vm.kmem_size="12G"

Don’t forget to reboot your system after making any changes. After changing to the new settings, the writing speed improves from 60MB/s to 80MB/s, sometimes it even goes above 110MB/s! That’s a 33% improvement!

Improve ZFS Performance: Step 3

Use disks with the same specifications

A lot of people may not realize the importance of using exact the same hardware. Mixing different disks of different models/manufacturers can bring performance penalty. For example, if you are mixing a slower disk (e.g., 5900 rpm) and a faster disk(7200 rpm) in the same virtual device (vdev), the overall speed will depend on the slowest disk. Also, different harddrives may have different sector size. For example, Western Digital releases a harddrive with 4k sector, while the general harddrives use 512 byte. Mixing harddrives with different sectors can bring performance penalty too. Here is a quick way to check the model of your harddrive:

sudo dmesg | grep ad

In my cases, I have the following:

ad10: 1907729MB Hitachi HDS722020ALA330 JKAOA20N at ata5-master UDMA100 SATA 3Gb/s
ad11: 1907729MB Seagate ST32000542AS CC34 at ata5-slave UDMA100 SATA 3Gb/s
ad12: 1907729MB WDC WD20EARS-00MVWB0 51.0AB51 at ata6-master UDMA100 SATA 3Gb/s
ad13: 1907729MB Hitachi HDS5C3020ALA632 ML6OA580 at ata6-slave UDMA100 SATA 3Gb/s
ad14: 1907729MB WDC WD20EARS-00MVWB0 50.0AB50 at ata7-master UDMA100 SATA 3Gb/s
ad16: 1907729MB WDC WD20EARS-00MVWB0 51.0AB51 at ata8-master UDMA100 SATA 3Gb/s

Notice that my Western Digital harddrives with 4k sectors all ends with EARS in the model number.

If you don’t have enough budget to replace all disks with the same specifications, try to group the disks with similar specifications in the same vdev.

Improve ZFS Performance: Step 4

Compression

ZFS supports compressing the data on the fly. This is a nice feature that improves the I/O speed – only if you have a high speed CPU (such as Quad core or higher). If your CPU is not fast enough, I don’t recommend you to turn on the compression feature, because the benefit from reducing the file size is smaller than the time spent on the CPU calculation. Also, the compression algorithm plays an important role here. ZFS supports two compression algorithms, LZJB and GZIP. I personally use LZJB because it gives a better balance between the compression radio and the performance. You can also use GZIP and specify your own compression ratio (i.e., GZIP-N). FYI, I tried GZIP-9 (The maximum compression ratio available) and I found that the overall performance gets worse on my i7 with 8GB of memory.

There is no solid answer here because it all depends on what files you store. Different files such as large file, small files, already compressed files (such as Xvid movie) need different compression settings.

If you cannot decide, just go with lzjb. It can’t be wrong:

sudo zfs set compression=lzjb mypool

Improve ZFS Performance: Step 5

Disable the unwanted features

By default, ZFS enables a lot of settings for data security, such as checksum etc. If you don’t care about the additional data security, just disable them. You can use the following command to view your ZFS settings.

sudo zfs get all

FYI, I usually disable the following:

#I don't care about the checksum.
sudo zfs set checksum=off myzpool

#I don't need ZFS to update the access time when reading the file
sudo zfs set atime=off myzpool

For more info, go to:

man zfs

Improve ZFS Performance: Step 6

Keep your ZFS up to date

By default, ZFS will not update the file system itself even if a newer version is available on the system. For example, I created a ZFS file system on FreeBSD 8.1 with ZFS version 14. After upgrading to FreeBSD 8.2 (which supports ZFS version 15), my ZFS file system was still on version 14. I needed to upgrade it manually using the following commands:

sudo zfs upgrade my_pool
sudo zpool upgrade my_pool

Improve ZFS Performance: Step 7

Use Cache / Log Devices

Suppose you have a very fast SSD harddrive. You can use it for logging/caching the data for your ZFS pool.

To improve the reading performance:

sudo zpool add 'zpool name' cache 'ssd device name'

To improve the writing performance:

sudo zpool add 'zpool name' log 'ssd device name'

It was impossible to remove the log devices without losing the data until ZFS v.19 (FreeBSD 8.3+/9.0+). I highly recommend to add the log drives as a mirror, i.e.,

sudo zpool add zpool_name log mirror /dev/log_drive1 /dev/log_drive2

Now you may ask a question. How about using a ram disk as log / cache devices? First, ZFS already uses your system memory for I/O, so you don’t need to set up a dedicated ram disk by yourself. Also, using a ram disk for log (writing) devices is not a good idea. When somethings go wrong, such as power failure, you will end up losing your data during the writing.

Improve ZFS Performance: Step 8

Two is better than one

Do you know ZFS works faster on multiple devices pool than single device pool, even they have the same storage size?

Improve ZFS Performance: Step 9

Use Mirror, not RAIDZ if speed is your first concern.

If you need performance, go with mirror, not RAIDZ. When ZFS stores the data in a mirror pool, it simply stores the whole file in each device. When it reads the file, it simply get the partial copy from each device first, and combine them at the end. Since this process can happen in parallel, it will speed up the reading process.

On the other hand, RAIDZ works a bit differently. Suppose there are N devices in your RAIDZ pool. When the data is written to a RAIDZ pool, ZFS needs to divided it into N-1 parts first, calculate the parity, and write all of them into the N devices. When reading the data, ZFS will read the N-1 devices first, make sure that the result is okay (otherwise it will read the data again from the remaining device), and combine them together. This additional work adds more work. That’s why RAIDZ is always slower than mirror.

So this is the ideal set up (given that you have enough budget):

Command to create a mirror zpool.

sudo zpool create zpool_name mirror /dev/hd1 /dev/hd2 mirror /dev/hd3 /dev/hd4 mirror /dev/hd5 /dev/hd6

Here is another model: Strip only, very fast with no data security.

Command to create a strip only zpool.

sudo zpool create zpool_name /dev/hd1 /dev/hd2 /dev/hd3 /dev/hd4 /dev/hd5 /dev/hd6

Here is the most popular model: RAIDZ, not so fast, but with okay data security.

Command to create a RAIDZ zpool.

sudo zpool create zpool_name raidz /dev/hd1 /dev/hd2 /dev/hd3 /dev/hd4 /dev/hd5 /dev/hd6

Improve ZFS Performance: Step 10

Distribute your free space evenly

One of the important tricks to improve ZFS performance is to keep the free space evenly distributed across all devices.

You can check it using the following command:

zpool iostat -v

The free space is show on the second column (available capacity)

               capacity     operations    bandwidth
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
storage     3.23T  1.41T      0      3  49.1K   439K
  ad4        647G   281G      0      0  5.79K  49.2K
  ad8        647G   281G      0      0  5.79K  49.6K
  ad10       647G   281G      0      0  5.82K  49.6K
  ad16       647G   281G      0      0  5.82K  49.6K
  ad18       647G   281G      0      0  5.77K  49.5K

When ZFS writes a new file to replace the old file in the system, it will first write the file in the free space first, then move the file pointer from the old one to the new one. In this case, even there is a power failure during writing the data, no data will be lost because the file pointer is still pointing to the old file. That’s why ZFS does not need fsck (file system check).

In order to keep the performance at a good level, we need to make sure that the free space is available in every device in the pool. Otherwise ZFS can only write the data to some of the devices only (instead of all). In the other words, the higher number of devices ZFS write, the better the performance.

Technically, if the structure of a zpool has not been modified or alternated, you should not need to worry about the free space distribution because ZFS will take care of that for you automatically. However, when you add a new device to an existing pool, that will be a different story, e.g.,

               capacity     operations    bandwidth
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
storage     3.88T  2.33T      0      3  49.1K   439K
  ad4        647G   281G      0      0  5.79K  49.2K
  ad8        647G   281G      0      0  5.79K  49.6K
  ad10       647G   281G      0      0  5.82K  49.6K
  ad16       647G   281G      0      0  5.82K  49.6K
  ad18       647G   281G      0      0  5.77K  49.5K
  ad20          0   928G      0      0  5.77K  49.5K

In this example, I add a 1TB hard drive (ad20) to my existing pool, which gives about 928GB of free space. Let say I add a 6GB file, the free space will look something like this:

               capacity     operations    bandwidth
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
storage     4.48T  1.73T      0      3  49.1K   439K
  ad4        648G   280G      0      0  5.79K  49.2K
  ad8        648G   280G      0      0  5.79K  49.6K
  ad10       648G   280G      0      0  5.82K  49.6K
  ad16       648G   280G      0      0  5.82K  49.6K
  ad18       648G   280G      0      0  5.77K  49.5K
  ad20         1G   927G      0      0  5.77K  49.5K

In the other words, ZFS will still divide my 6GB file into six equal pieces and write each piece to each device. Eventually, ZFS will use up the free space in the older devices, and it can write the data to the new devices only (ad20), which will decrease the performance. Unfortunately, there is no way to redistribute the data / free space evenly without destroying the pool, i.e.,

1. Back up your data
2. Destroy the pool
3. Rebuild the pool
4. Put the data back

Depending on how much data do you have, it can take 2 to 3 days to copy 5TB of data from one server to another server over a gigabit network. You don’t want to use scp to do it because you will need to re-do everything again if the process is dropped. In my case, I use rsync:

(One single line)

#Run this command on the production server:
rsync -avzr --delete-before backup_server:/path_to_zpool_in_backup_server /path_to_zpool_in_production_server

Improve ZFS Performance: Step 11

Backup the data on different machine, not on the same pool

ZFS comes with a very cool feature. It allows you to save multiple copies of the same data in the same pool. This adds an additional layer on data security. However, I don’t recommend using this feature for backup purpose because it adds more work when writing the data to the disks. Also, I don’t think this is a good way to secure the data. I prefer to set up a mirror on a different server (Master-Slave). Since the chance of two machines fail at the same time is much smaller than one machine fails. Therefore the data is safer in this settings.

Here is how I synchronize two machines together:

Create a script in the slave machine: getContentFromMaster.sh
(One single line)

rsync -avzr -e ssh --delete-before master:/path/to/zpool/in/master/machine /path/to/zpool/in/slave/machine

And put this file in a cronjob, i.e.,
/etc/crontab

@daily root /path/to/getContentFromMaster.sh

Now, you may ask a question. Should I go with strip-only ZFS (i.e., stripping only. No mirror, RAIDZ, RAIDZ2) when I set up my pool? Yes or no. ZFS allows you to mix any size of har ddrive in one single pool. Unlike RAIZ{0,1,5,10} and concatenation, it can be any size and there is no lost in the disk space, i.e., you can connect 1TB, 2TB, 3TB into one single pool while enjoying the data-stripping (Total usable space = 6TB). It is fast (because there is no overhead such as parity etc) and simple. The only down side is that the entire pool will stop working if there is at least one device fails.

Let’s come back to the question, should we employ simple stripping in production environment? I prefer not. Strip-only ZFS divides all data into all vdev. If each vdev is simply a hard drive, and if one fails, there is NO WAY to get the original data back. If something screws up in the master machine, the only way is to destroy and rebuild the pool, and restore the data from the backup. (This process can takes hours to days if you have large amount of data, say 6TB.) Therefore, I strongly recommend to use at least RAIDZ in the production environment. If one device fails, the pool will keep working and no data is lost. Simply replace the bad hard drive with a good one and everything is good to go.

To minimize the downtime when something goes wrong, go with at least RAIDZ in a production environment (ideally, RAIDZ or strip-mirror).

For the backup machine, I think using simple stripping is completely fine.

Here is how to build a pool with simple stripping, i.e., no parity, mirror or anything

zpool create mypool /dev/dev1 /dev/dev2 /dev/dev3

And here is how to monitor the health

zpool status

Some websites suggest to use the following command instead:

zpool status -x

Don’t believe it! This command will return “all pools are healthy” even if one device is failed in a RAIDZ pool. In the other words, your data is healthy doesn’t mean all devices in your pool are healthy. So go with “zpool status” at any time.

FYI, it can easily takes few days to copy 10TB of data from one machine to another through a gigabit network. In case you need to restore large amount of data through the network, use rsync, not scp. I found that scp sometimes fail in the middle of transfer. Using rsync allows me to resume it at any time.

Improve ZFS Performance: Step 12

rsync or ZFS send?

So what’s the main difference between rsync and ZFS send? What’s the advantage of one over the other?

rsync is a file level synchronization tool. It simply goes through the source, find out which files have been changed, and copy the corresponding files to the destination.

ZFS send is doing something similar. First, it takes a snapshot on the ZFS pool first:

zfs snapshot mypool/vdev@20120417

After that, you can generate a file that contains the pool and data information, copy to the new server to restore it:

#Method 1: Generate a file first
zfs send mypool/vdev@20120417 > myZFSfile
scp myZFSfile backupServer:~/
zfs receive mypool/vdev@20120417 < ~/myZFSfile

Or you can do everything in one single command line:

#Method 2: Do everything over the pipe (One command)
zfs send pool/vdev@20120417 | ssh backupServer zfs receive pool/vdev@20120417

In general, the preparation time of ZFS send is much shorter than rsync, because ZFS already knows which files have been modified. Unlike rsync, a file-level tool, ZFS send does not need to go though the entire pool and find out such information. In terms of the transfer speed, both of them are similar.

So why do I prefer rsync over ZFS send (both methods)? It’s because the latter one is not practical! In method #1, the obvious issue is the storage space. Since it requires generating a file that contains your entire pool information. For example, suppose your pool is 10TB, and you have 8TB of data (i.e., 2TB of free space), if you go with method #1, you will need another 8TB of free space to store the file. In the other words, you will need to make sure that at least 50% of free space is available all the time. This is a quite expensive way to run ZFS.

What about method #2? Yes, it does not have the storage problem because it copies everything over the pipe line. However, what if the process is interrupted? It is a common thing due to high traffic in the network, high I/O to the disk etc. Worst worst case, you will need to re-do everything again, say, copying 8TB over the network, again.

rsync does not have these two problems. In rsync, it uses relatively small space for temporary storage, and in case the rsync process is interrupted, you can easily resume the process without copying everything again.

Improve ZFS Performance: Step 13

Disable dedup if you don’t have enough memory (2GB memory per 1TB storage)

Deduplication (dedup) is a space-saving technology. It simply stores one copy of your file instead of storing multiple copies. For example, suppose you have ten identical folders with the same files. If the dedup is enabled, ZFS only stores one copy instead of multiple copies. Notice that dedup is not the same as compression. ZFS: Compression VS Deduplication(Dedup) in Simple English

The idea of dedup is very simple. ZFS maintains an index of your files. Before writing any incoming files to the pool, it checks whether the storage has a copy of this file or not. If the file already exists, it will skip the file. With dedup enabled, instead of store 10 identical files, it stores one only copy. Unfortunately, the drawback is that it needs to check every incoming file before making any decision.

After upgrading my ZFS pool to version 28, I enabled dedup for testing. I found that it really caused huge performance hit. The writing speed over the network dropped from 80MB/s to 5MB/s!!! After disabling this feature, the speed goes up again.

sudo zfs set dedup=off your-zpool

In general, dedup is an expensive feature that requires a lot of hardware resources. You will need 2GB memory per 1TB. For example, if zpool is 10TB, I will need 20GB of memory! (Which I only have 12GB). Therefore, think twice before enabling dedup!

Notice that it won’t solve all the performance problem by disabling the dedup. For example, if you enable dedup before and disable it afterward, all files stored during this period are dedup dependent, even dedup is disabled. When you need to update these files (e.g., delete), the system still needs to check again the dedup index before any processing your file. Therefore, the performance issue still exists when working with these affected files. For the new files, it should be okay. Unfortunately, there is no way to find out the affected dedup files. The only way is to re-build/empty the ZFS pool, which will clear the list of dedup files.

Enjoy ZFS.

–Derrick

After I updated my Samba to 3.5.14 on my FreeBSD box today, it stopped working. I couldn’t connect to the Samba server from my Windows clients. Initially, I thought it was the memory cache issues, and I thought the problem would be solved by rebooting both machines. Unfortunately, it didn’t work. Therefore, I accessed to the server and check the cause of the problem.

First, I found that Samba was not evening running.

ps -ax | grep smb

So I started the Samba service manually:

sudo /usr/local/etc/rc.d/samba start

And it returned me the following message:

Cannot 'start' samba. Set winbindd_enable to YES in /etc/rc.conf or use 'onestart' instead of 'start'.

Oh that was easy. All I need to do was to modify my rc.conf:

#Samba service
samba_enable="YES"
winbindd_enable="YES"

And then I started the Samba again:

sudo /usr/local/etc/rc.d/samba start

Yay, Samba is up and running again!

–Derrick

Recently I got a new hard drive (3TB) for my Linux box at work. I didn’t realize that it is problematic / nightmare to do such a simple work.

First, if you try to use fdisk to add the drive, it won’t work. Or I should say… it works, but it only give you a 2TB usable space. This is a limitation of fdisk, because it can handle up to 2^32 -1 sectors (or 2TB). That’s why fdisk is not a solution here.

It seems that the only solution is using parted.

sudo parted /dev/sdd

It will return something like this:

GNU Parted 2.3
Using /dev/sdd
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted)

First, we want to create a label:

(parted) mklabel gpt
Warning: The existing disk label on /dev/sdd will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? Yes

Then, we want to specify the unit:

(parted) unit TB

And tell parted to use all available space:

(parted) mkpart primary 0 -0

Next, we need to review the summary:

(parted) print
Model: ATA WDC WD30EZRX-00M (scsi)
Disk /dev/sdd: 3.00TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
 1      0.00TB  3.00TB  3.00TB  ext2         primary

As you can see, the size is 3TB. Don’t worry about the file system for now (ext2). If it looks good to you, now we need to leave parted.

(parted) quit
Information: You may need to update /etc/fstab.

Next, we need to format the partition:

sudo mkfs.ext4 /dev/sdd1

After few seconds, it will give you some more information about your drive:

mke2fs 1.41.14 (22-Dec-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=1 blocks, Stripe width=1 blocks
183148544 inodes, 732566272 blocks
36628313 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
22357 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848, 512000000, 550731776, 644972544

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 35 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Now you can use your drive, such as mounting or putting it into /etc/fstab

sudo mount /dev/sdd1 /mnt/
df
/dev/sdd1             2.8T   42G  2.6T   2% /mnt

That’s it! Enjoy your new disk.

–Derrick

Every time Firefox releases a major upgrade, we love it because it comes with lots of cool features. At the mean time, we hate it because lots of our favorite add-ons are not working with the new Firefox. Sometimes, the owners of the add-ons might be too busy, and it might take a while to release a compatible version. That’s why I put some of my favorite add-ons here. They are 100% compatible with the new Firefox, and I only modified the source codes that by-pass the version compatibly check.

In the other words, the add-on will work the way it was in the old version, and you will receive an upgrade if the owner releases a newer version.

So far I have made the following add-ons available in the latest version of Firefox:

If you need any add-on, please post in the comment below. I will try to make it available here.

Enjoy your new Firefox!

–Derrick

Many people are confused between the compression and deduplication because they are so similar. Both of them are designed to reduce the size of the data being stored in the storage. Let me explain the difference between them in simple English.

1. This is how your data looks like originally (Assuming only one unique file):

2. This is how your data look like after being stored in a ZFS pool with compression enabled.

3. This is how your data look like after being stored in a ZFS pool with deduplication enabled.

4. Let say we are storing three identical files, i.e.,

5. ZFS: Compression Only

6. ZFS: Deduplication Only

7. ZFS: Compression + Deduplication

Of course, enabling both compression and deduplication will save lots of free space. However, it comes with a very high price tag. If you like to enable deduplication, you need to make sure that you have at least 2GB of memory per 1TB of storage. For example, if your ZFS pool is 10TB, you need to have 20GB of memory installed in your system. Otherwise, you will experience a huge performance hit.

Hope this article helps you to understand the difference between compression and deduplication.

–Derrick

Today, I sent a message to a company through their contact page on their website. It is a standard contact page, i.e., you need to fill in your contact information and the details of your request, and they will follow up with you later. An hour later, I received a reply from them, saying that they would like more information from me. Therefore, I reply to their email. Few seconds later, I got the following:

Delivery has failed to these recipients or distribution lists:

Assistly@company.com
Your message wasn't delivered because of security policies. Microsoft Exchange will not try to redeliver this message for you. Please provide the following diagnostic text to your system administrator.
Sent by Microsoft Exchange Server 2007

Diagnostic information for administrators:

....

(Another 100 lines of error messages)

Initially, I thought I made a mistake when typing the email address. Therefore, I redid it and verified every single letter in the email address. Unfortunately, I got the same message again. After trying it for 5 times, I gave up, and this company loses a sale.

There are few things we’ve learned here. First, never display very technical error messages to the customers. They are very boring for non-engineers. Second, don’t assume that every customer is patient. Not every one is willing to re-send the same email for 5 times. Third, it takes many years to build a city but it only takes few hours to destroy it. With today’s technology, it is too easy to spread out the bad words.

I think this is not a right way to run a custom request management system (or ticket system). It should never yell to the customer. Instead, it should let the stuffs to evaluate the customer’s reply rather than letting the Microsoft Exchange Server to do it.

I don’t recommend using Microsoft Exchange for your business. It just hurts your business.

–Derrick