Today, I noticed that the MySQL server on my FreeBSD server is down for unknown reason, and today is Thanksgiving.

Initially, I tried to restart the MySQL server using the following command, but it didn’t help to restart the MySQL:

sudo /usr/local/etc/rc.d/mysql-server restart

and it gives me the following error message:

mysql not running? (check /var/db/mysql/icesquare.com.pid).

That’s interesting. So I check the pid file and I found that it is not available. After scratching my head for a while, I noticed one thing:

df
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/ad4s1d    7.7G    7.7G     -1M   100%    /var

Obviously, the /var directory is full. No wonder why the MySQL could not create any file, which explains why it stopped working.

Since the server is already up and running, it will be too late to increase the size of /var without reinstalling the entire system. The simplest way is to soft link the /var directory to somewhere else. Since a lot of system applications depends on /var, I don’t want to move the entire /var into a different places. So I decide to softlink /var/log into /usr/var/log.

The steps are easy:

sudo su
mkdir -p /usr/var/
mv /var/log /usr/var/
ln -s /usr/var/log /var/

To verify your work, it should look something like that:

ls /var
lrwxr-xr-x   1 root    wheel       13B Nov 22 11:48 log -> /usr/var/log/
df
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/ad4s1d    7.7G    1.1G    6.0G    15%    /var

Now, if you try to start the MySQL server again, it may or may not work. If you experience any difficulties, try to reboot the server. After the server is reboot, MySQL should be up and running again.

Lesson learned: Always create single partition. 🙂

–Derrick

Our sponsors:

In case you are not aware this bad news. FreeBSD.org was compromised on September 17, 2012. Good news is that the compromise was caused by a human error (a stupid developer leaked a SSH private key somewhere), which has nothing to do with the code quality of the FreeBSD. 🙂 Bad news is that this may affect you if you use port. 🙁

The compromise is believed to have occurred due to the leak of an SSH key from a developer who legitimately had access to the machines in question, and was not due to any vulnerability or code exploit within FreeBSD.

We unfortunately cannot guarantee the integrity of any packages available for installation between 19th September 2012 and 11th November 2012, or of any ports compiled from trees obtained via any means other than through svn.freebsd.org or one of its mirrors. Although we have no evidence to suggest any tampering took place and believe such interference is unlikely, we have to recommend you consider reinstalling any machine from scratch, using trusted sources.

In short, the port tree was affected from September 19, 2012 to November 11, 2012. If you downloaded the ports in between this period, you are likely a victim of this incident. 🙁 Here is how to clean up the mess:

Stop using cvsup/csup to update your port. Use portsnap instead.

If you already used portsnap, do the following:

sudo portsnap fetch extract
sudo portsnap upgrade

If you have no idea what is portsnap, here is a quick tutorial:

First, install the portmaster and portupgrade. Port-Upgrade downloads the new ports to your server and Port-Master updates your applications based on the downloaded ports.

sudo pkg_add -r portupgrade portmaster

Download all ports. Notice that this is one-time work.

sudo portsnap fetch extract

If you decide to update your port tree, run the following:

sudo portsnap fetch update

Or if you like to run it in a cron job, run the following instead:

sudo nano /etc/crontab

#Run every three hours
0       */3     *       *       *       root    portsnap -I cron update && pkg_version -vIL= >/dev/null 2>&1

Now you have an updated port tree. Next you will need to update your applications based on your local port tree:

sudo portmaster -Dai

That’s it.

And remember, NEVER put your SSH private key in a public area.

–Derrick

Our sponsors: