Category: Troubleshooting

panic: ffs_blkfree: freeing free block

Posted on by

In the last few days, my FreeBSD box drove me crazy with the kernel panic. The error message is shown below:

dev = ad14s1f, block = 1, fs = /usr
panic: ffs_blkfree: freeing free block
cpuid = 4
KDB: stack backtrace:
...

I am not sure what caused this kernel panic, but it happened few times a day for a week. Every time it happened, I had to reboot my machine manually. After some investigations, I found the following solution:

1. Boot into the single user mode.
2. Run the following command to force (-f) the system to check the system. It is likely that you will get a lot of error, so adding a -y option (which assume yes to all the questions) will make your life easier:

fsck -f -y

After the check is done, reboot the system to normal mode:

reboot

The reason why my system had this kind of issue was due to the filesystem inconsistency. Even the fsck was trigger after each system was crashed, it could not detect or repair the problem (fsck will not fix any problem on a mounted partition on a running system.). That’s why we need to do it in the single user mode.

–Derrick

Our sponsors:

FreeBSD Startup Script – How to Start a Script on Boot in FreeBSD (An Alternative Solution to /etc/rc.local)

Posted on by

I want to start a script when booting my FreeBSD box automatically. Unlike Linux, FreeBSD does not honor the script you put in /etc/rc.local. Usually it is recommended to use the rc service (See here for details), however I think it is too complicated to convert my script to a rc compatible version. I decide to explore more simple solutions. I found that it can be done via Cron.

Suppose I have a script in the following location. I decide to run it when booting the machine automatically:

/script/myscript.sh

First, I need to include this script in the Cron:

sudo nano /etc/crontab

And add the following to the end of the file:

@reboot root /script/myscript.sh

That will make the system to execute this script.

However, the system may not run your script correctly because the Cron job uses a different shell and the path information may be missing. You can fix it by modifying the SHELL and PATH variables in /etc/crontab:

In my case, I uses Bash and I like my script executed by Bash:

SHELL=/usr/local/bin/bash

And my script needs to execute some commands that locate in /usr/local/bin:

PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin

That’s it! Have fun with FreeBSD.

–Derrick

Our sponsors:

The devel/automake110 port has been deleted: No longer required by any port…Aborting update

Posted on by

My FreeBSD box must be sick today. For some reasons, when I ran the portsnap command (i.e., portsnap fetch update), it removed the entire port directory (i.e., rm /usr/ports -Rf). Therefore, I decided to build the entire port tree:

sudo su
cd /var/db/portsnap
rm -Rf *
portsnap fetch extract

I decided to run portmaster to update my system, I got the following error:

===>>> The devel/automake110 port has been deleted: No longer required by any port
===>>> Aborting update
Terminated

Well, the problem looks pretty simple. FreeBSD found an out-dated package installed in my system, and it could not find any corresponding port (Already deleted in the repository), that’s why it stops there and doesn’t know what to do next.

To solve this problem, the solution is very simple:

First, let’s see what version of automake we have in our system:

pkg_info | grep auto

which return:

autoconf-2.62       Automatically configure source code on many Un*x platforms
autoconf-2.68       Automatically configure source code on many Un*x platforms
autoconf-wrapper-20101119 Wrapper script for GNU autoconf
automake-1.10.1     GNU Standards-compliant Makefile generator (1.10)
automake-1.11.1     GNU Standards-compliant Makefile generator (1.11)
automake-wrapper-20101119 Wrapper script for GNU automake

Apparently, the older version of autoconf and automake cause the issues. Why not remove them?

sudo pkg_delete autoconf-2.62 automake-1.10.1

I re-ran the portmaster to update the system. Everything worked fine again!

–Derrick

Our sponsors:

Apache won’t start after upgrading to 2.2.21 in FreeBSD 8.2

Posted on by

Apache 2.2.21 was rolled out today. Since it has some security fixes, I decide to put a new version on my server farm. I didn’t know that it becomes a small headache today morning.

The upgrade was running smooth, no error messages, no complaints, until I restart the Apache server manually

sudo apachectl restart

And guess what, my website didn’t show up on the browser. Instead, I saw the following picture on my browser:

That was not fun at all. So I decided to investigate what was wrong to the upgrade.

sudo tail /var/log/httpd-error.log

And I found something interesting:

httpd: Syntax error on line 100 of /usr/local/etc/apache22/httpd.conf:
Cannot load /usr/local/libexec/apache22/mod_negotiation.so into server:
/usr/local/libexec/apache22/mod_negotiation.so: Undefined symbol "ap_set_accept_ranges"

Therefore, I edited my Apache configuration and commented out that module:

#LoadModule negotiation_module libexec/apache22/mod_negotiation.so

Saved the file and restarted the server again. Now the problem is gone!

sudo apachectl stop
sudo apachectl start

When I tried to visited the website again, I found that the speed is bit slow. I think it has something to do with the mod_negotiation.so. I decided to rebuild the Apache.

cd /usr/ports/www/apache22
sudo make config

Make sure that the Negotiation: Enable mod_negotiation is checked.

sudo make

Running sudo make is enough. You don’t need to re-run sudo make install or sudo make reinstall.

This will rebuild the necessary libraries including the missing mod_negotiation.so.

Go back to the configuration file and enable the module again:

LoadModule negotiation_module libexec/apache22/mod_negotiation.so

Restart the server:

sudo apachectl restart

That’s it! Happy Apaching!

–Derrick

Our sponsors:

What OS is good for SPARC64?

Posted on by

Recently, I received a SPARC64 machine (Sun Blade 100) as a gift. If you are not sure what is SPARC64, it is a CPU primary manufactured by Sun/Oracle. It is quite different than Intel x86 / AMD 64, so anything that is compiled to run on these two platforms will not work on SPARC64. Besides, Sun/Oracle is currently shifting toward to Intel now, which means the SPARC64 will become less popular in the long run. Since I don’t want to let my Sun Blade becomes a paper-weight / foot stool, I decide to explore some options on what I can do with it.

Notice that this list only includes some popular operating systems I’ve tested only. I am pretty sure that there are lots of more choices available.

If you don’t want to go through the entire list, here are some quick recommendations:
Good: FreeBSD, NetBSD, OpenBSD, OpenSolaris
Okay: Debian
Avoid: Fedora, Ubuntu

The following operating system are listed in alphabetically:

Debian Linux

Recommendation: Yes if you don’t care about the performance
Support Status: Active
Comment: Although Debian provides a 64-bit version, the applications are run in 32-bit mode (Source). It can not only cause performance issue because the application does not utilize the full power of the CPU. For example, a 32-bit application can only handle unsigned integer from 0 to 2^31-1 (2147483647).

Fedora Linux

Recommendation: No
Support Status: Seems Discontinued.
Comment: As of today, the latest version of Fedora is 15. However, the latest version for SPARC is 12/13. Seems like the support in this architecture is inactive.
Source: Architectures/SPARC

FreeBSD

Recommendation: Yes
Support Status: Active
Comment: The command freebsd-update is not supported in SPARC64. In the other words, you have to rebuild the kernel from scratch in order to upgrade the system to a newer version

NetBSD

Recommendation: Yes
Support Status: Active
Download: Here

OpenBSD

Recommendation: Yes
Support Status: Active

OpenIndiana

Recommendation: Yes
Support Status: Active

OpenSolaris

Recommendation: No
Support Status: Discontinued

Ubuntu Linux

Recommendation: No
Support Status: Inactive
Comment: Ubuntu already dropped support to SPARC64. Forget about it.
Source: Ubuntu drops support for ia64 and sparc

Our sponsors:

cups-client-1.4.8 is marked as broken: gnutls does not support threads yet; disable the GNUTLS.

Posted on by

I saw this message twice on two machines on two different days, end up using the same way to fix it.

Case 1: Samba

My Samba was down today, so I tried to start it manually:

/usr/local/etc/rc.d/samba start

And I got the following error message:

Removing stale Samba tdb files:  done
nmbd already running? (pid=1074).
Starting smbd.
/libexec/ld-elf.so.1: Shared object "libcups.so.2" not found, required by "smbd"
/usr/local/etc/rc.d/samba: WARNING: failed to start smbd

Initially, I thought the problem was about missing the file. So I did a search and soft-link it:

find / -name "libcups.so*"

Surprisingly, there is no such file in my system. I think I don’t have this file. So I tried to compile the Samba again:

cd /usr/ports/net/samba35
sudo make

and I got the following message:

===>  cups-client-1.4.8 is marked as broken: gnutls does not support threads yet disable the GNUTLS.
*** Error code 1

Stop in /usr/ports/print/cups-client.

Case 2: Upgrading Port

When I upgraded my FreeBSD system via portmaster, I found the following messages:

portmaster -Da

===>  cups-client-1.4.8 is marked as broken: gnutls does not support threads yet disable the GNUTLS.
*** Error code 1

Stop in /usr/ports/print/cups-client.

Initially, I tried to by-pass upgrading the cups-client using:

portmaster -Dai

However, it is very annoying to do it everyday, therefore I decide to solve this problem.

Solution

From the error message, it seems that the problem is caused by the additional option in cups-client: GNUTLS. So I decide to disable this option in the GNUTLS:

cd /usr/ports/print/cups-client
sudo make config

Uncheck “GNUTLS: Build with GNUTLS instead of OpenSSL”

and try to compile it again:

sudo make

Now re-run the portmaster:

sudo portmaster -Da

The problem is gone.

Our sponsors:

How to clean up the dmesg message

Posted on by

Recently, I found that my dmesg log are filled of junk. Or I should say it contains lots of errors and complains that I already fix. However, I don’t want to reboot the machine to get rid of these error messages. How can I do it? The solution is very simple.

For FreeBSD and other BSD members, run the following:

sudo sysctl kern.msgbuf_clear=1

For Fedora / Ubuntu Linux, run the following instead:

sudo dmesg -c

Easy?

–Derrick

Our sponsors:

PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/local/lib/php/20090626/tokyo_tyrant.so’ – /usr/local/lib/php/20090626/tokyo_tyrant.so: Undefined symbol “ps_globals” in Unknown on line 0

Posted on by

Every time I upgrade the PHP and related extensions, I always run into the same problem with some PHP-PECL modules. Recently with the new release of PHP 5.3.7, there is no exception for the trouble. So what kind of trouble I have today? The Tokyo Tyrant (The middleman between my program and Tokyo Cabinet).

#php -v

PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20090626/tokyo_tyrant.so' - /usr/local/lib/php/20090626/tokyo_tyrant.so: Undefined symbol "ps_globals" in Unknown on line 0

This problem was caused by a version mis-match. PHP is running at a later version, which the Tokyo Tyrant extension is running at an earlier version. To solve this computer problem, all you need is to re-install the PHP-Tokyo-Tyrant extension, i.e.,

cd /usr/ports/databases/pecl-tokyo_tyrant
suod make deinstall clean
sudo make install

And later, you may or may not get the following error message:

...
cc -shared  .libs/tokyo_tyrant.o .libs/tokyo_tyrant_funcs.o .libs/connection.o .libs/session_funcs.o .libs/server_pool.o .libs/failover.o .libs/session.o  -Wl,--rpath -Wl,/usr/local/lib -Wl,--rpath -Wl,/usr/local/lib -L/usr/local/lib -ltokyotyrant -llua -ltokyocabinet /usr/local/lib/liblzo2.so -llzma -lbz2 -lz -lrt -lm  -Wl,-soname -Wl,tokyo_tyrant.so -o ./.libs/tokyo_tyrant.so
/usr/bin/ld: cannot find -llua
*** Error code 1

Stop in /usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0.
*** Error code 1

Stop in /usr/ports/databases/pecl-tokyo_tyrant.

The problem is related to Lua. By default, the extension enable the Lua option, which I have a hard time to make it run properly on FreeBSD box. Since my website does not use the Lua with Tokyo Cabinet, I simply disable it:

sudo nano /usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/Make

Look for the keyword: lua
And you will find something like the following:

TOKYO_TYRANT_SHARED_LIBADD = -R/usr/local/lib -L/usr/local/lib -R/usr/local/lib -L/usr/local/lib -ltokyotyrant -llua -ltokyocabinet -lbz2 -lz -lrt -lm -ltokyocabinet -llzo2 -llzma -lbz2 -lz -lrt -lm

Delete the option: “-llua”, i.e.,

TOKYO_TYRANT_SHARED_LIBADD = -R/usr/local/lib -L/usr/local/lib -R/usr/local/lib -L/usr/local/lib -ltokyotyrant -ltokyocabinet -lbz2 -lz -lrt -lm -ltokyocabinet -llzo2 -llzma -lbz2 -lz -lrt -lm

and re-start the installation:

cd /usr/ports/databases/pecl-tokyo_tyrant
sudo make install clean

Now re-run the php, and you may see the following error:

#php -v

PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20090626/tokyo_tyrant.so' - /usr/local/lib/php/20090626/tokyo_tyrant.so: Undefined symbol "ps_globals" in Unknown on line 0
Segmentation fault: 11

That’s because the Tokyo Tyrant module is placed at a wrong position in the PHP extension file. Edit the file:

sudo nano /usr/local/etc/php/extensions.ini

and move the following code to the END of the file (Yes, order does matter):

extension=tokyo_tyrant.so

And re-check your php:

#php -v

PHP 5.3.7 with Suhosin-Patch (cli) (built: Aug 23 2011 10:33:12)
Copyright (c) 1997-2011 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2011 Zend Technologies

Now everything is all set.

Unfortunately, you pretty much need to do the same thing every time when upgrading your PHP on FreeBSD box.

–Derrick

Our sponsors:

HighPoint RocketRAID 2640X4SGL Beeping

Posted on by

I noticed some beeping sound coming from my server today. After I opened the server case, I found that the problem came from my newly install HighPoint RocketRAID card. During the beep, the server could not operate normally. First, the system dumped some error messages, and followed by core dump, and the computer ended up rebooting by itself. After the system was restarted, during the first few seconds, the system was doing fine. After waiting for couple minutes, the error happened again. So I know that it was not some random error, and it was repeatable.

Here are my step-by-step guide on how to solve this problem:

Beeping doesn’t mean defective

After reading the manual, I found that beeping is part of the monitoring process. If the card detects any failure in the RAID system, it will start beeping. If you have any doubt, try to run the card on a different machine. That’s the best way to verify whether the card is defective or not.

Did you make any change to the system recently

Making a system stable is not easy, because you don’t know whether the hardware are conflicting with each other, or the newly installed module will cause the system unstable or not. For example, I found that the beeping happened after I enabled the Linux Emulation module in FreeBSD. After I turned that off, things are working fine.

Are the cables securely attached?

SATA harddrive pretty much is the standard today. It is fast and doesn’t take up lots of space. However, unlike the old IDE cable, it is very easy to get loosen. After I unplugged all the attached SATA cables from the RAID card, and put them back afterward, the problem seemed gone.

Flash the firmware

Although this is not likely, but the problem can sometimes caused by an out-dated firmware. (I have experience a similar thing on my motherboard before). Try to flash the card with the latest firmware.

If none of these methods work for you, sounds like your card is defective.

–Derrick

Our sponsors:

How to install HighPoint RocketRAID 2640X4SGL on FreeBSD 8.X

Posted on by

This tutorial is for FreeBSD 8. If you are looking for the driver for FreeBSD 9, you can forget about it. According to HighPoint, there are no plans to update the RocketRaid 26XX driver to FreeBSD 9. Since it is a binary driver, there are very limited things we can do. FYI, here is the reply from their support team:

New Priority: Medium

Status: Active -> Awaiting Customer Action

Dear customer there are no plans to update the RocketRAID driver to support FreeBSD 9.0.

Note: Our RocketRAID 27xx series controllers have native support in FreeBSD 9.0.

Regards

HighPoint Support Team

I will never purchase any product from this HighPoint.

Recently I decide to upgrade my file server. I decide to get a PCI-Express SATA controller, and attach my hard-drives to it. After researching some cards, I finally decide to buy this one:

HighPoint RocketRAID 2640X4SGL

What it differentiates with other similar cards is the number of PCI-express channel. It comes with 4 channels while the other cards only carry one. In the other words, it will make transfer speed higher.

Anyway, long story short. FreeBSD does not recognize this card by default. You need to install a driver to support it. Unfortunately, the driver is version specific, and the driver on the web only supports up to FreeBSD 7. I have two choices – Switch to the legacy Microsoft Windows / Linux, or downgrade to FreeBSD 7.0. One way or the other, I need to say Goodbye to ZFS.

After some searches and hacks, finally I found the hidden driver. Here you are: Driver For FreeBSD 8.X

1. Extract the driver

tar -zxvf rr26xx-bsd-8.0-v1.2.tgz

2. Copy the driver (64-bit)

cp rr26xx-8.0-amd64.ko /boot/kernel/rr26xx.ko

3. Add the following in /boot/defaults/loader.conf

rr26xx_load="YES"

4. Reboot

reboot

4. After the reboot is done, you can see the hard-drives (/dev/da*) by running the following commands:

dmesg | grep da

That’s it! Have fun.

–Derrick

Our sponsors: