Today, I noticed that the MySQL server on my FreeBSD server is down for unknown reason, and today is Thanksgiving.
Initially, I tried to restart the MySQL server using the following command, but it didn’t help to restart the MySQL:
sudo /usr/local/etc/rc.d/mysql-server restart
and it gives me the following error message:
mysql not running? (check /var/db/mysql/icesquare.com.pid).
That’s interesting. So I check the pid file and I found that it is not available. After scratching my head for a while, I noticed one thing:
df
Filesystem Size Used Avail Capacity Mounted on /dev/ad4s1d 7.7G 7.7G -1M 100% /var
Obviously, the /var directory is full. No wonder why the MySQL could not create any file, which explains why it stopped working.
Since the server is already up and running, it will be too late to increase the size of /var without reinstalling the entire system. The simplest way is to soft link the /var directory to somewhere else. Since a lot of system applications depends on /var, I don’t want to move the entire /var into a different places. So I decide to softlink /var/log into /usr/var/log.
The steps are easy:
sudo su mkdir -p /usr/var/ mv /var/log /usr/var/ ln -s /usr/var/log /var/
To verify your work, it should look something like that:
ls /var
lrwxr-xr-x 1 root wheel 13B Nov 22 11:48 log -> /usr/var/log/
df
Filesystem Size Used Avail Capacity Mounted on /dev/ad4s1d 7.7G 1.1G 6.0G 15% /var
Now, if you try to start the MySQL server again, it may or may not work. If you experience any difficulties, try to reboot the server. After the server is reboot, MySQL should be up and running again.
Lesson learned: Always create single partition. 🙂
–Derrick
Our sponsors:
Hello Derrick 🙂
I’m not proficient with FreeBSD so please forgive my ignorance but what’s the problem with having a separate /var and why can’t you resize it?
Can’t you simply boot yout server to a recovery/live image and then move/resize your partitions to give more space to /var?
While this is my first post here -I think!- I frequently read your blog, nice work!
Hi Martin,
Thanks for your comment. Actually there is a way to re-size the partition without rebuilding the whole system: growfs
However, since my server is already in production and I wanted to minimize the down-time and impact to the existing users, I decided to go with a simple solution: soft-link.
For growfs, some people report that it is unstable on newer FreeBSD (> 8, current version is 9) and it may cause the kernel to crash. So I rather stick with the old school method.
Thanks for vising this blog. 🙂
–Derrick
“However, since my server is already in production and I wanted to minimize the down-time and impact to the existing users, I decided to go with a simple solution: soft-link.
For growfs, some people report that it is unstable on newer FreeBSD (> 8, current version is 9) and it may cause the kernel to crash. So I rather stick with the old school method.”
I see, thanks for the explanation.
“Thanks for vising this blog. ”
Np, thank _you_ for posting your experiences, it really makes a difference to some of us 😉