ZFS Compression: lz4 VS lzjb

Posted on by

ZFS offers a new compression method in the latest version: lz4. It claims to be better than lzjb. Since lzjb is pretty good already, I am curious to find out how good will lz4 be comparing to lzjb. According to my tests, lz4 is performing better than lzjb in terms of spacing saving and I/O, but not too much.

lz4 VS lzjb: Space Saving

Long story short, here is what I did. I set up two servers with brand new ZFS settings. One server is set to lzjb, and the other server is set to lz4. Both servers have exact the same hardware, software, operating system etc. Both of them are loaded with the same set of data (10TB) via rsync and rsyncd. These data are real data that I use everyday, which includes office documents (PDF, Word, Excel), photos(Raw and JPEG), music(mp3), video(mpeg), zip, source codes, binary applications, database (MySQL, Redis), webserver files (PHP, HTML, CSS), etc. Notice that some of these files are already compressed (such as jpeg, zip etc), and the file types are not evenly distributed (e.g., 40% of the files are jpeg, 25% are docs, 10% are zip, 5% are something else). I want to make a test in a more realistic scenario.

Also, all ZFS settings are set BEFORE loading the data. This will ensure that all data on the same server share the same compression settings. Below is the summary of the used space. Keep in mind that the space saving test is independent to the hardware, such as CPU type, memory, speed of the disks, etc. Just like running the same command to compress the same set of files in two systems. We expect the result will have the same size. The only difference will be the time spent on compressing the data.

#ZFS with lzjb
df
Filesystem    512-blocks        Used     Avail Capacity  Mounted on
storage/data 23007987368 22996620466  11366902   100%    /storage/data

#Used space: 22996620466 blocks = 10965.6 GB

#ZFS with lz4
df
Filesystem    512-blocks        Used      Avail Capacity  Mounted on
storage/data 31527470077 22942284913 8585185164    73%    /storage/data

#Used space: 22942284913 blocks = 10939 GB

I found that for 10TB of data, the server with lzjb uses 26GB more space than lz4, which translate to 0.23% (26GB out of 10TB). The difference is quite small and not too significant. However, when your ZFS is nearly full or you are too busy to upgrade your system, the extra saving may matter a lot.

lz4 VS lzjb: Time Saving

I am also curious to know how much time I will save by switching from lzjb to lz4. Therefore I did another simple test. First, I generated a 1GB of random data and wrote to the lzjb system. After that, I destroyed the zpool and rebuilt the zpool with lz4. This will eliminate other factors such as hardware, network traffic etc, because both tests were done within the same system. 

#ZFS with lzjb
time dd if=/dev/random of=/storage/data/file.out bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 23.725955 secs (44195313 bytes/sec)

real    0m24.144s
user    0m0.024s
sys     0m18.326s

#ZFS with lz4
time dd if=/dev/random of=/storage/data/file.out bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 22.589257 secs (46419234 bytes/sec)

real    0m22.802s
user    0m0.016s
sys     0m18.273s

P.S. The /dev/random in FreeBSD is nothing like the one in Linux. It is very fast.

By just upgrading from lzjb to lz4, the time is reduced from 24.144 to 22.802 seconds, which translate to 5.5% (1.342s out of 24.144s) improvement.

This result also agrees with my overall experience: The improvement is small and not noticeable. In fact, I barely notice any performance improvement after switching from lzjb to lz4, which primarily includes transferring the files between Windows and FreeBSD-based ZFS system via Samba on a gigabit network. The I/O speed are about the same. However, if you are talking about a busy server with lots of traffic, a 5% improvement will be something.

Anyway, this is what I recommend. If lz4 is available in your ZFS version, use it. It can’t be wrong.

sudo zfs set compression=lz4 myzpool

–Derrick

Our sponsors:

ZFS Performance Boost/Improvement: How I push the I/O speed to 126MBps

Posted on by

This article is part of my main ZFS tutorial: How to improve ZFS performance. That article covers everything you need. If you already have the basic knowledge, or you just want to know how I push the I/O speed to 120+ MBps, you can skip that article and read this one.

Long story short, here is the result of my iostat:

sudo zpool iostat -v

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
storage     7.85T  12.1T      1  1.08K  2.60K   126M < ---126 MBps!
  raidz2    5.51T  8.99T      1    753  2.20K  86.9M
    ada0        -      -      0    142    102  14.5M
    ada1        -      -      0    143    102  14.5M
    ada3        -      -      0    146    307  14.5M
    ada4        -      -      0    145    409  14.5M
    ada5        -      -      0    144    204  14.5M
    ada6        -      -      0    143    204  14.5M
    ada7        -      -      0    172    409  14.5M
    ada8        -      -      0    171    511  14.5M
  raidz1    2.34T  3.10T      0    349    409  39.5M
    da0         -      -      0    169    204  13.2M
    da1         -      -      0    176      0  13.2M
    da2         -      -      0    168    102  13.2M
    da3         -      -      0    173    102  13.2M
----------  -----  -----  -----  -----  -----  -----

I measured the speed while I was transferring 10TB of data from one FreeBSD-based ZFS server to another FreeBSD-based ZFS server, over a consumer-level gigabit network. Basically, every components I used are consumer-level. The hard drives I used are standard PATA and SATA hard drives (i.e., not even the light speed SSD hard drive). The data I transferred are real data (rather than all zeros or ones generated through dd).

First of all, here are the settings I used. Both server and client have similar configurations.

sudo zpool history
History for 'storage':
2014-01-22.20:28:59 zpool create -f storage raidz2 /dev/ada0 /dev/ada1 /dev/ada3 /dev/ada4 /dev/ada5 /dev/ada6 /dev/ada7 /dev/ada8 raidz /dev/da0 /dev/da1 /dev/da2 /dev/da3
2014-01-22.20:29:07 zfs create storage/data
2014-01-22.20:29:15 zfs set compression=lz4 storage
2014-01-22.20:29:19 zfs set compression=lz4 storage/data
2014-01-22.20:30:19 zfs set atime=off storage

#where the ad* and da* are nothing more than standard SATA drives:
#ZFS Drives
#8 x 2TB standard SATA hard drives connected to the motherboard
#4 x 1.5TB standard SATA hard drives connected to a PCI-e RAID card

da0: 1430799MB (2930277168 512 byte sectors: 255H 63S/T 182401C)
da1: 1430799MB (2930277168 512 byte sectors: 255H 63S/T 182401C)
da2: 1430799MB (2930277168 512 byte sectors: 255H 63S/T 182401C)
da3: 1430799MB (2930277168 512 byte sectors: 255H 63S/T 182401C)
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 512bytes)
ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada3: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada4: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada5: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada6: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada6: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada7: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada7: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada8: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada8: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)


#System Drive
#A PATA 200MB drive from 2005
ada2: maxtor STM3200820A 3.AAE ATA-7 device
ada2: 100.000MB/s transfers (UDMA5, PIO 8192bytes)
ada2: 190782MB (390721968 512 byte sectors: 16H 63S/T 16383C)

Here are the corresponding hardware components:

#CPU
hw.machine: amd64
hw.model: Intel(R) Core(TM) i7 CPU         920  @ 2.67GHz
hw.ncpu: 8
hw.machine_arch: amd64


#Memory - 3 x 2GB
real memory  = 6442450944 (6144 MB)
avail memory = 6144352256 (5859 MB)

#Network - Gigabit network on the motherboard / PCI-e
#If your gigabit network card is PCI, that will be your bottleneck.
re0: realtek 8168/8111 B/C/CP/D/DP/E/F PCIe Gigabit Ethernet

Other than that, I didn’t really specify any special settings in my OS. Everything else are default:

#uname -a
FreeBSD 9.2-RELEASE-p3 FreeBSD 9.2-RELEASE-p3 #0: Sat Jan 11 03:25:02 UTC 2014     [email protected]:/usr/obj/usr/src/sys/GENERIC  amd64

The way how to transfer 10TB of data is through rsync and rsyncd. First of all, I set up a rsync as daemon on the server. Keep in mind that rsync as a daemon (service) is not the same as rsync. If you want to know how I set up the rsyncd, please visit How to Improve rsync Performance.

After you have set up the rsyncd on the server, I run the following on the client.

rsync -av server_ip::rsync_share_name /my_directory/

#Example
rsync -av 192.168.1.101::storage /mydata/

Notice that I didn’t enable the compression option. That’s because my files are pretty much compressed (jpeg, zip, tar, 7z etc). If your file types are different, you may want to enable the compression, i.e.,

rsync -avz server_name::rsync_share_name /my_directory/

Give it a try and see whether it improves the I/O speed.

Here are few things I have learned to improve the ZFS performance.

Keep the ZFS structure small and simple. For example, keep the number of disks in your vdev small. Previously, I set up a giant big pool with 14 disks RAIDZ in one single vdev. This is a bad idea. There is a maximum number of disks for RAIDZ. That’s why I split my disks into two groups:

#Group 1 - RAIDZ2
#8 x 2TB standard SATA hard drives connected to the motherboard

#Group2 - RAIDZ
#4 x 1.5TB standard SATA hard drives connected to a PCI-e RAID card

zpool create -f storage myzpool group1 raidz group2

Keep the ZFS settings clean. Enable the only thing you need, and disable the junk settings. In my case, I enable the best compression (lz4) and disable the access time:

2014-01-22.20:29:15 zfs set compression=lz4 storage
2014-01-22.20:30:19 zfs set atime=off storage

Keep the ZFS clean. I see the performance gain on a brand new ZFS than a few years old ZFS, even both of them contain the same data. Unfortunately, there is no easy way to clean the ZFS. The only way is to destroy the current ZFS first, create a new one and bring the data back. Typically, it takes about 2 days to transfer 10TB of data over a gigabit LAN. So it is really not too bad to spend about 5 days to rebuild your ZFS.

That’s it. Again, if you want to learn about tricks, please visit How to improve ZFS performance for more information.

–Derrick

Our sponsors:

How to Remote Desktop To Linux From Windows Using XRDP

Posted on by

If you are looking for ways to remote desktop to Linux from Windows, and you are sick of VNC, you are in the right place.

I have been looking for a solution to do something very simple. I want to remotely access the desktop of my Linux servers from Windows machine. Of course, it must be at a usable level.

I’ve tried different applications before, incluing XMing, XManager (X11 forwarding), VNC, 2X, NoMachine, ranging from open-source to commercial. Unfortunately, they offer nothing close to the Windows to Windows Remote Desktop experience.

Why XMing is bad?

XMing allows you to run a remote application (e.g., Firefox) on your desktop. It works great as long as you keep the session available. Once you close your session, there is no way to go back to the previous session. For example, suppose I am browsing a website using the Firefox on the remote server via XMing. If I decide to close the XMing (without closing the browser), there is no way for me to go back to the page I left off later. Of course, in reality, it doesn’t make too much sense to use a Firefox on a remote server. I was referring to some software that takes days to run, such as R, Matlab etc.

By the way, it is an application level, meaning that it does not work with the desktop level.

Why VNC is bad?

VNC is nearly perfect. It offers something similar to Windows Remote Desktop, i.e., I can access to the desktop level. If I work on something, I can get my previous session back even my session is closed. There are only two disadvantages: Slow, and not for multiple users.

VNC uses a stone-age protocol to delivery the remote desktop experience: Screen capture. When there is a change on the remote server, it takes a screen shot and send it to the client. You can imagine how slow it will be over the Internet. Although they released a better algorithm, which works directly at the graphic card level, i.e., only send the modified area to the client instead of everything. However the new protocol is still far from usable. The Microsoft Remote Desktop just blows VNC far far away.

Another thing I don’t like VNC is that it is at the machine/server level, not the user level. VNC uses its own user authorization. When you set up a VNC service, you define your own password. That’s it. In the other words, if I open a desktop on the server as derrick. Everybody will be able to see my desktop content via VNC as long as they enter the correct password. Unless I go to the server physically and switch to a different user, otherwise there is no way to protect the content of my desktop.

The good thing about VNC is that it is cross-platform. It works on OS X, Windows, Linux etc. Wait a minute. Java is cross-platform too. Do you want to use it?

Why XManager is bad?

XManager is better than VNC. It uses some special protocol such that the speed is not an issue. However, it is suffered in the session problem like XMing. If I remote desktop to the server using XManager, and I close my session later, there is no way for me to go back to my previous sessions. In the other words, I will lose all opened applications and contents.

And no, the session is still running silently on the server. They are just orphans. XManager doesn’t allow you to connect to the old session when you try to establish a new connection.

Solution: xrdp

I am so happy that I finally found something useful and usable. That’s xrdp. Here is a step-by-step tutorial on how to install it on Fedora/CentOS Linux. The instruction will be similar to Debian/Ubuntu Linux. The set up time should take no more than 5 minutes.

#Install xrdp
sudo yum install -y xrdp

If your Linux is too old, or your default repository does not have xrdp available, you may want to run the following:

#Don't forget to use the right (i686 or x86_64) architecture.

sudo rpm -Uvh  http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
sudo rpm -Uvh  http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
sudo yum update -y

sudo yum install -y xrdp

Next, we want to start the xrdp service

sudo /etc/rc.d/init.d/xrdp start

You may want to enable this service during the boot up:

sudo chkconfig  xrdp on

Don’t forget to open the port 3389 in your firewall:

sudo nano /etc/sysconfig/iptables

-A INPUT -m state --state NEW -m tcp -p tcp --dport 3389 -j ACCEPT

Don’t forget to restart the firewall:

sudo service iptables stop
sudo service iptables start

Now, run the remote desktop on Windows, and enter the IP address of the server to see what happen.

–Derrick

Our sponsors:

[FreeBSD]mount: /dev/da0p2: R/W mount of / denied. Filesystem is not clean – run fsck. Forced mount will invalidate journal contents: Operation not permitted

Posted on by

When I tried to mount a hard drive on FreeBSD today, I got the following error message.

sudo mount -t ufs /dev/da0p2 /mnt/
mount: /dev/da0p2: R/W mount of / denied. Filesystem is not clean - run fsck. Forced mount will invalidate journal contents: Operation not permitted

Here is how to fix it:

sudo mount -r -t ufs /dev/da0p2 /mnt/

where ufs refers to the file system of FreeBSD.

That’s it.

–Derrick

Our sponsors:

[FreeBSD] ZFS FAULTED / kernel: (ada0:ahcich4:0:0:0): lost device

Posted on by

I am so happy that I finally solved a problem today. This problem had existed in my server farms for few months already. In the past few months, I had absolutely no idea what was causing the issues. Here is my story:

I built a file server using FreeBSD and ZFS. The hardware components are consumer grade hardware. Basically, it is a desktop computer with multiple hard drives. The hard drives are connected to the SATA ports on the motherboard. It is a very simple setup.

After I set up a ZFS pool, I started to load the data and stress tested the system. I noticed that the ZFS pool turned into a fault state after 15 mins, e.g., 

#sudo zpool status -v
  pool: storage
 state: FAULTED
status: One or more devices could not be opened.  There are insufficient
        replicas for the pool to continue functioning.
action: Attach the missing device and online it using 'zpool online'.
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            ada4    ONLINE       0     0     0
            ada0    REMOVED      0     0     0 
            ada2    ONLINE       0     0     0
            ada5    ONLINE       0     0     0
            ada3    ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

In short, a drive is missing. So I tried to run dmesg 

Oct 17 20:24:00 kernel: (ada0:ahcich4:0:0:0): lost device
Oct 17 20:24:00 kernel: (ada0:ahcich4:0:0:0): removing device entry
Oct 19 20:42:14 kernel: (ada0:ahcich3:0:0:0): lost device
Oct 19 23:19:18 kernel: (ada0:ahcich4:0:0:0): lost device
Oct 19 23:19:18 kernel: (ada0:ahcich4:0:0:0): removing device entry
Oct 19 23:19:18 kernel: (ada0:ahcich4:0:0:0): lost device
Oct 19 23:19:18 kernel: (ada0:ahcich4:0:0:0): removing device entry
Oct 19 23:19:18 kernel: (ada0:ahcich4:0:0:0): lost device
Oct 19 23:19:18 kernel: (ada0:ahcich4:0:0:0): removing device entry
Oct 19 23:19:18 kernel: (ada0:ahcich4:0:0:0): lost device
Oct 19 23:19:18 kernel: (ada0:ahcich4:0:0:0): removing device entry
Oct 19 23:19:18 kernel: (ada0:ahcich4:0:0:0): lost device
Oct 19 23:19:18 kernel: (ada0:ahcich4:0:0:0): removing device entry
Oct 19 23:33:02 kernel: (ada0:ahcich4:0:0:0): Synchronize cache failed
Oct 20 22:53:40 kernel: (ada0:ahcich4:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 40 30 af 4e 40 7a 00 00 00 00 00
Oct 20 22:53:40 kernel: (ada0:ahcich4:0:0:0): CAM status: ATA Status Error
Oct 20 22:53:40 kernel: (ada0:ahcich4:0:0:0): ATA status: 61 (DRDY DF ERR), error: 04 (ABRT )
Oct 20 22:53:40 kernel: (ada0:ahcich4:0:0:0): RES: 61 04 00 00 00 40 00 00 00 00 00
Oct 20 22:53:40 kernel: (ada0:ahcich4:0:0:0): Retrying command

I also verified the content in /dev/, e.g.,

#ls /dev/ad0

crw-r-----  1 root  operator   0x6f Oct 18 22:10 /dev/ada0

Interestingly, the device is still available. This is very confusing because the system sees the hardware (with some error), but the application could not see the hardware. So I decided to perform the following tests:

  • Reboot
  • Shutdown and reboot
  • Replacing the SATA cables
  • Connecting the hard drive to a different SATA port
  • Replacing the hard drive with a new one that is certified by the manufacturer
  • Replacing the motherboard
  • Replacing the hard drive power cable
  • Replacing the power supply
  • Updating the firmware of the motherboard

Unfortunately, none of these fixed the problem. I also checked the S.M.A.R.T. status of the hard drives but I didn’t see any issues. One thing I noticed is that the error is not consistent. It happens randomly on different SATA port and hard drive. That doesn’t make any sense to me because I saw the same thing on 3 different motherboards. Finally, I decided to try one last thing: I connected the hard drives through an USB port. Guess what, my FreeBSD box stopped complaining.

So I decided to a different approach. I used the same hardware setup (the hard drives were connected using SATA instead of USB) and loaded a Windows. The system worked perfectly fine and stable. I didn’t experience any issue at all. So I believed the problem had nothing to do with the hardware. It must be the software settings issue.

Finally, I decided to try one last thing, something I hadn’t paid attention before: BIOS Settings

AHCI/IDE

Typically there is a setting to control how the motherboard interacts with the hard drives: IDE or AHCI. I usually stick with the default settings. In my case, the default value of my motherboard is IDE. After I changed the settings to AHCI, I found that the problem is gone. Yes, it’s gone and my headache is gone too.

That was easy!

–Derrick

Our sponsors:

How to Improve rsync Performance

Posted on by

I need to transfer 10TB of data from one machine to another machine. Those 10TB of files are living in a large RAID which span across 7 different disks. The target machine has another large RAID which span across 12 different disks. It is not easy to copying those files locally. Therefore, I decide to copy the files over the LAN.

There are four options popping up in my head: scp, rsync, rsyncd (rsync as daemon) and netcat.

scp

scp is handy, easy to use but comes with two disadvantages: slow and not fault-tolerant. Since scp comes with the highest security, all data are encrypted before the transfer. It will slow down the overall performance because of the extra encryption stuffs (which makes the data larger), and extra computational resource (which uses more CPU). If the transfer is interrupted, there is no easy way to resume the process other than transferring everything again. Here are some example commands:

#Source machine
#Typical speed is about 20 to 30MB/s
scp -r /data target_machine:/data

#Or you can enable the compression on the fly
#Depending on the type of your data, if your data is already compressed, you may see no or negative speed improvement
scp -rC /data target_machine:/data

rsync

rsync is similar to scp. It comes with the encryption (via SSH) such that the data is safe. It also allows you to transfer the newer files only. This will reduce the amount of data being transferred. However, it comes with few disadvantages: long decision time, encryption (which increase the size of overhead) and extra computational resource(e.g., data comparison, encryption and decryption etc). For example, if I use rsync to transfer 10TB of files from one machine to another machine (where the directory on the target machine is blank), it can easily take 5 hours to determine which files will need to be transferred before the actual data transfer is initialized.

#Run on the target machine
rsync -avzr -e ssh --delete-after source_machine:/data/ /data/

#Use a less secure encryption algorithm to speed up the process
rsync -avzr --rsh="ssh -c blowfish" --delete-after source_machine:/data/ /data/

#Use an even less secure algorithm to get the top speed
rsync -avzr --rsh="ssh -c arcfour" --delete-after source_machine:/data/ /data/

#By default, rsync compares the files using checksum, file size and modification date.
#Reduce the decision process by skipping the hash check
rsync -avzr --rsh="ssh -c arcfour" --delete-after --whole-file source_machine:/data/ /data/

Anyway, no matter what you do, the top speed of rsync in a consumer-grade gigabit network is around 45MB/s. On average, the speed is around 25-35MB/s. Keep in mind that this number does not include the decision time, which can be few hours.

rsyncd (rsync as a daemon)

Thanks for the comment of our reader. I got a chance to investigate the rsync as a daemon. Basically, the idea of running rsync as a daemon is similar to rsync. On the server, we run rsync as a service/daemon. We specify which directory we want to “export” to the clients (e.g., /usr/ports). When the files get changed on the server, it records the changes so that the when the clients talk to the server, the decision time will be faster. Here is how to set up rsync server on FreeBSD

sudo nano /usr/local/etc/rsyncd.conf

And this is my configuration file:

pid file = /var/run/rsyncd.pid

#Notice that I use derrick here instead of other systems users, such as nobody
#That's because nobody does not have permission to access the path, i.e., /data/
#Either you make the source directory available to "nobody", or you change the daemon user.
uid = derrick
gid = derrick
use chroot = no
max connections = 4
syslog facility = local5
pid file = /var/run/rsyncd.pid

[mydata]
   path = /data/
   comment = data

Don't forget to include the following in /etc/rc.conf, so that the service will be started automatically.

rsyncd_enable="YES"

#Let's start the rsync service:

sudo /usr/local/etc/rc.d/rsyncd start

To pull the files from the server to the clients, run the following:

rsync -av myserver::mydata /data/

#Or you can enable compression
rsync -avz myserver::mydata /data/

To my surprise, it works much better than running rsync alone. Here are some data I collected during transferring 10TB files from ZFS to ZFS:

Bandwidth measured on the client machine: 70MB/s

zpool IO speed on the client side: 75MB/s

P.S. Initially, the speed was about 45-60MB/s, after I tweak my Zpool, I can get the top speed to 75-80MB/s. Please check out here for references.

I notice that the decision time is much faster than running rsync alone. Also the process is much more stable, with zero interruption, i.e., 

rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(521) [receiver=3.1.0]
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(632) [generator=3.1.0]
rsync: [receiver] write error: Broken pipe (32)

NetCat

NetCat is similar to cat, except that it works at the network level. I decide to use netcat for the initial transfer. If it is interrupted, I will let rsync to kick in the process. Netcat does not encrypt the data, so the overhead is very small. If you transfer the file within a local network and you don’t care about the security, netcat is a perfect choice.

There is only one disadvantage of using netcat. It can only handle one file at a time. It doesn’t mean you need to run netcat for every single file. Instead, we can tar the file before feeding to netcat, and untar the file at the receiving end. As long as we do not compress the files, we can keep the CPU usage small. 

#Open two terminals, one for the source and another one for the target machine.

#On the target machine:
#Go to the directory, e.g., 
cd /data

#Run the following:
nc -l 9999| tar xvfp -

#On the source machine:
#Go to the directory, e.g.,
cd /data

#Pick a port number that is not being used, e.g., 9999
tar -cf - . | nc target_machine 9999

Unlike rsync, the process will start right the way, and the maximum speed is around 45 to 60MB/s in a gigabit network.

Conclusion

Candidates Top Speed (w/o compression) Top Speed (w/ compression) Resume Stability Instant Start?
scp 40MB/s 25MB/s No Low Instant
rsync 25MB/s 50MB/s Yes Medium Long Preparation
rsyncd 30MB/s 70MB/s Yes High Short Preparation
netcat 60MB/s (tar w/o -z) 40MB/s (tar w/ -z) No Very High Instant

–Derrick

Our sponsors:

SFTP Without SSH Shell Access

Posted on by

I got an interesting question today. Our client needs to access to the server via SFTP, but they want to disable to the access to the SSH shell. Assuming that you are fully aware of the potential issues of doing it, such ask security risk etc, here is how to do it:

#Create a user account as usual, e.g.,
sudo adduser sftpuser

#Edit the User Password Profile
sudo vipw


#Type i to switch to the editor mode

#Replace the shell of the user to the following based on your OS:
Linux: /usr/libexec/openssh/sftp-server
FreeBSD: /usr/libexec/sftp-server


#Your line will look something like this:
#Linux:
sftpuser:x:1001:1001::/home/sftpuser:/usr/libexec/openssh/sftp-server

#FreeBSD:
sftpuser:x:$XX3edc8989Ra.:1001:1001::0:0:SFTP User:/home/sftpuser:/usr/libexec/sftp-server

#Type :wq to save and quit.

You may want to include the sftp command in your shell lists, i.e.,

sudo nano /etc/shells

#Include the following (Linux):
/usr/libexec/openssh/sftp-server

#Include the following (FreeBSD):
/usr/libexec/sftp-server

That’s it!

–Derrick

Our sponsors:

[FreeBSD]Problem to Update cURL-7.31.0

Posted on by

When I tried to update the cURL to cURL 7.31.0 today for my FreeBSD, it stopped and gave the following error messages:

configure: using CFLAGS: -O2 -pipe -DLDAP_DEPRECATED -fno-strict-aliasing
configure: CFLAGS error: CFLAGS may only be used to specify C compiler flags, not macro definitions. Use CPPFLAGS for: -DLDAP_DEPRECATED
configure: error: Can not continue. Fix errors mentioned immediately above this line.
===>  Script "configure" failed unexpectedly.
Please report the problem to [email protected] [maintainer] and attach the
"/usr/ports/ftp/curl/work/curl-7.31.0/config.log" including the output of the
failure of your make command. Also, it might be a good idea to provide an
overview of all packages installed on your system (e.g. a /usr/sbin/pkg_info
-Ea).
*** [do-configure] Error code 1

Stop in /usr/ports/ftp/curl.
*** [build] Error code 1

Stop in /usr/ports/ftp/curl.

===>>> make failed for ftp/curl
===>>> Aborting update

===>>> Update for ftp/curl failed
===>>> Aborting update

===>>> Killing background jobs
Terminated

===>>> You can restart from the point of failure with this command line:
       portmaster  ftp/curl

===>>> Exiting

It is very simple the fix this problem. However, the prerequisite is to give up the support to LDAP. If you are not sure whether your cURL needs LDAP or not, you probably don’t need it.

cd /usr/ports/ftp/curl
sudo make config

Remove the LDAP related features, such as LDAP, LDAPS. Try to re-build cURL again.

sudo make clean
sudo make

If everything looks good (i.e., not complaining any more), you can resume the update process, e.g.,

sudo portmaster -Da

Now your FreeBSD should be happy.

–Derrick

Our sponsors:

[FreeBSD]Portsnap / gunzip: can’t stat: files/… .gz: No such file or directory

Posted on by

My FreeBSD got an unknown fever today. When I updated my ports using portsnap, it gave the following error message:

#sudo portsnap fetch
Looking up portsnap.FreeBSD.org mirrors... 6 mirrors found.
Fetching snapshot tag from your-org.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Thu May 23 09:08:53 CDT 2013 to Thu May 23 09:25:25 CDT 2013.
Fetching 0 metadata patches. done.
Applying metadata patches... done.
Fetching 0 metadata files... done.
gunzip: can't stat: files/992a1325cdc9a00a3543aa38fdf58903cdf70eaee02b8bb8aebea5505ac7b3f8.gz: No such file or directory
Fetching 0 patches. done.
Applying patches... done.
Fetching 0 new ports or files... done.
Building new INDEX files... gunzip: can't stat: /var/db/portsnap/files/09f65f8a730283fd31d068a5927ed46d95e37540f89090c257d7809b75116293.gz: No such file or directory
gunzip: can't stat: /var/db/portsnap/files/e3d3219617c1ea87cdfac7c8df0a52d611b191be8a80fd97f511277dff4cce77.gz: No such file or directory
gunzip: can't stat: /var/db/portsnap/files/8c2576279258f0d1b8762df8fc1e0cb4bcfcd23b6b09cdb4e7d68886af35ed7d.gz: No such file or directory
done.

Apparently, something in /var/db/portsnap/ is broken. Many people will try to remove /var/db/portsnap/ and run the command again. Do not do it. It will make portsnap failed. Instead, do the following:

sudo cp -r /var/db/portsnap /var/db/portsnap_backup
sudo rm -Rf /var/db/portsnap/tag /var/db/portsnap/files/*
sudo portsnap fetch extract
sudo portsnap update

Now your portsnap should be happy.

–Derrick

Our sponsors:

[Solved]FreeBSD: Problem to Update glib20

Posted on by

I decided to install Java on my FreeBSD box today. It wasn’t a very good experience. Primary the system is not automated. It requires a lot of manual works.

Anyway, after waiting for couple hours, I found that the process was stuck on glib. The system could not build the /usr/ports/devel/glib20. Here is the error message: 

===>  Building for glib-2.34.3
gmake  all-recursive
gmake[1]: Entering directory `/usr/ports/devel/glib20/work/glib-2.34.3'
Making all in .
gmake[2]: Entering directory `/usr/ports/devel/glib20/work/glib-2.34.3'
gmake[2]: Nothing to be done for `all-am'.
gmake[2]: Leaving directory `/usr/ports/devel/glib20/work/glib-2.34.3'
Making all in m4macros
gmake[2]: Entering directory `/usr/ports/devel/glib20/work/glib-2.34.3/m4macros'
gmake[2]: Nothing to be done for `all'.
gmake[2]: Leaving directory `/usr/ports/devel/glib20/work/glib-2.34.3/m4macros'
Making all in glib
gmake[2]: Entering directory `/usr/ports/devel/glib20/work/glib-2.34.3/glib'
gmake  all-recursive
gmake[3]: Entering directory `/usr/ports/devel/glib20/work/glib-2.34.3/glib'
Making all in libcharset
gmake[4]: Entering directory `/usr/ports/devel/glib20/work/glib-2.34.3/glib/libcharset'
gmake[4]: Nothing to be done for `all'.
gmake[4]: Leaving directory `/usr/ports/devel/glib20/work/glib-2.34.3/glib/libcharset'
Making all in update-pcre
gmake[4]: Entering directory `/usr/ports/devel/glib20/work/glib-2.34.3/glib/update-pcre'
gmake[4]: Nothing to be done for `all'.
gmake[4]: Leaving directory `/usr/ports/devel/glib20/work/glib-2.34.3/glib/update-pcre'
Making all in .
gmake[4]: Entering directory `/usr/ports/devel/glib20/work/glib-2.34.3/glib'
  CC       gstrfuncs.lo
  CC       gthreadpool.lo
gstrfuncs.c:330: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'get_C_locale'
gstrfuncs.c: In function 'g_ascii_strtod':
gstrfuncs.c:700: warning: implicit declaration of function 'strtod_l'
gstrfuncs.c:700: warning: implicit declaration of function 'get_C_locale'
gstrfuncs.c: In function 'g_ascii_formatd':
gstrfuncs.c:902: error: 'locale_t' undeclared (first use in this function)
gstrfuncs.c:902: error: (Each undeclared identifier is reported only once
gstrfuncs.c:902: error: for each function it appears in.)
gstrfuncs.c:902: error: expected ';' before 'old_locale'
gstrfuncs.c:904: error: 'old_locale' undeclared (first use in this function)
gstrfuncs.c:904: warning: implicit declaration of function 'uselocale'
gstrfuncs.c: In function 'g_ascii_strtoull':
gstrfuncs.c:1148: warning: implicit declaration of function 'strtoull_l'
gstrfuncs.c: In function 'g_ascii_strtoll':
gstrfuncs.c:1195: warning: implicit declaration of function 'strtoll_l'
gmake[4]: *** [gstrfuncs.lo] Error 1
gmake[4]: *** Waiting for unfinished jobs....
gmake[4]: Leaving directory `/usr/ports/devel/glib20/work/glib-2.34.3/glib'
gmake[3]: *** [all-recursive] Error 1
gmake[3]: Leaving directory `/usr/ports/devel/glib20/work/glib-2.34.3/glib'
gmake[2]: *** [all] Error 2
gmake[2]: Leaving directory `/usr/ports/devel/glib20/work/glib-2.34.3/glib'
gmake[1]: *** [all-recursive] Error 1
gmake[1]: Leaving directory `/usr/ports/devel/glib20/work/glib-2.34.3'
gmake: *** [all] Error 2
*** Error code 1

Stop in /usr/ports/devel/glib20.
*** Error code 1

Stop in /usr/ports/devel/glib20.

This is not a popular problem. After I google a while, I found a very similar problem here. Basically, the author suggests that the problem may be caused by an uncleaned run of freebsd-update. In short, you need to run the program twice, i.e.,

sudo freebsd-update fetch install
sudo reboot
sudo freebsd-update install

Obviously, it didn’t work for me. So I decided to try my last solution: pkg_add

sudo pkg_add -r glib20

and I tried to resume the installation:

cd /usr/ports/java/jdk16
sudo make install

It worked!

Hope this little trick is helpful to you.

–Derrick

Our sponsors: