[FreeBSD]How to run PECL Tokyo Tyrant in PHP 5.4

After upgrading to PHP 5.4.3 (which is the first version of PHP 5.4 available in the FreeBSD port), I found that the pecl-tokyo_tyrant could not be compiled. Here is the error messages:

cd /usr/ports/databases/pecl-tokyo_tyrant
sudo make
===>  pecl-tokyo_tyrant-0.6.0 cannot install: doesn't work with PHP version : 5 (Doesn't support PHP 5).
*** Error code 1

Stop in /usr/ports/databases/pecl-tokyo_tyrant.
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:1827: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'php_tokyo_tyrant_class_methods'
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:1867: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'php_tokyo_tyrant_table_class_methods'
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:1911: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'php_tokyo_tyrant_query_class_methods'
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:1936: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'php_tokyo_tyrant_iterator_class_methods'
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c: In function 'php_tokyo_tyrant_query_object_new':
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2001: error: 'zend_class_entry' has no member named 'default_properties'
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c: In function 'php_tokyo_tyrant_iterator_object_new':
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2052: error: 'zend_class_entry' has no member named 'default_properties'
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c: In function 'php_tokyo_tyrant_object_new_ex':
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2090: error: 'zend_class_entry' has no member named 'default_properties'
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c: In function 'zm_startup_tokyo_tyrant':
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2166: error: 'php_tokyo_tyrant_class_methods' undeclared (first use in this function)
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2166: error: (Each undeclared identifier is reported only once
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2166: error: for each function it appears in.)
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2171: error: 'php_tokyo_tyrant_table_class_methods' undeclared (first use in this function)
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2176: error: 'php_tokyo_tyrant_query_class_methods' undeclared (first use in this function)
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2182: error: 'php_tokyo_tyrant_iterator_class_methods' undeclared (first use in this function)
*** Error code 1

Stop in /usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0.
*** Error code 1

Stop in /usr/ports/databases/pecl-tokyo_tyrant.

Basically, it means that the pecl-tokyo_tyrant is not happy with the new version of PHP 5.4. I’ve tried several methods, including using pkg_add, compiling from source, copying a working tokyo_tyrant.so etc, unfortunately none of these methods works.

Finally, I found that the maintainer of this package has made a new version available(0.6.2) at github. Here is how I did it:

First, go to here to grab the source code. Make sure that you download the tag.gz version because it has the file attribute available.

Second, let’s remove the pecl-tokyo_tyrant from your system. You can either do it in the port or pkg_delete:

From port:

cd /usr/ports/databases/pecl-tokyo_tyrant
sudo make deinstall

Or using pkg_delete:

sudo pkg_delete pecl-tokyo_tyrant-0.6.0

Now, extract the files and compile it from source:

tar -zxvf mkoppanen-php-tokyo_tyrant-0.6.0-2-gb0d7e0c.tar.gz
cd mkoppanen-php-tokyo_tyrant-b0d7e0c
sudo phpize
sudo ./configure
sudo make
sudo make install

and don’t forget to include the .so file in your configuration:

sudo nano /usr/local/etc/php/extensions.ini

Add the following (if not available) to the end of the file:

extension=tokyo_tyrant.so

Finally, let’s restart the Apache:

sudo apachectl stop
sudo apachectl start

That’s it! Have fun with Tokyo Tyrant with PHP 5.4.

–Derrick

Our sponsors:

PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/local/lib/php/20090626/tokyo_tyrant.so’ – /usr/local/lib/php/20090626/tokyo_tyrant.so: Undefined symbol “ps_globals” in Unknown on line 0

Every time I upgrade the PHP and related extensions, I always run into the same problem with some PHP-PECL modules. Recently with the new release of PHP 5.3.7, there is no exception for the trouble. So what kind of trouble I have today? The Tokyo Tyrant (The middleman between my program and Tokyo Cabinet).

#php -v

PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20090626/tokyo_tyrant.so' - /usr/local/lib/php/20090626/tokyo_tyrant.so: Undefined symbol "ps_globals" in Unknown on line 0

This problem was caused by a version mis-match. PHP is running at a later version, which the Tokyo Tyrant extension is running at an earlier version. To solve this computer problem, all you need is to re-install the PHP-Tokyo-Tyrant extension, i.e.,

cd /usr/ports/databases/pecl-tokyo_tyrant
suod make deinstall clean
sudo make install

And later, you may or may not get the following error message:

...
cc -shared  .libs/tokyo_tyrant.o .libs/tokyo_tyrant_funcs.o .libs/connection.o .libs/session_funcs.o .libs/server_pool.o .libs/failover.o .libs/session.o  -Wl,--rpath -Wl,/usr/local/lib -Wl,--rpath -Wl,/usr/local/lib -L/usr/local/lib -ltokyotyrant -llua -ltokyocabinet /usr/local/lib/liblzo2.so -llzma -lbz2 -lz -lrt -lm  -Wl,-soname -Wl,tokyo_tyrant.so -o ./.libs/tokyo_tyrant.so
/usr/bin/ld: cannot find -llua
*** Error code 1

Stop in /usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0.
*** Error code 1

Stop in /usr/ports/databases/pecl-tokyo_tyrant.

The problem is related to Lua. By default, the extension enable the Lua option, which I have a hard time to make it run properly on FreeBSD box. Since my website does not use the Lua with Tokyo Cabinet, I simply disable it:

sudo nano /usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/Make

Look for the keyword: lua
And you will find something like the following:

TOKYO_TYRANT_SHARED_LIBADD = -R/usr/local/lib -L/usr/local/lib -R/usr/local/lib -L/usr/local/lib -ltokyotyrant -llua -ltokyocabinet -lbz2 -lz -lrt -lm -ltokyocabinet -llzo2 -llzma -lbz2 -lz -lrt -lm

Delete the option: “-llua”, i.e.,

TOKYO_TYRANT_SHARED_LIBADD = -R/usr/local/lib -L/usr/local/lib -R/usr/local/lib -L/usr/local/lib -ltokyotyrant -ltokyocabinet -lbz2 -lz -lrt -lm -ltokyocabinet -llzo2 -llzma -lbz2 -lz -lrt -lm

and re-start the installation:

cd /usr/ports/databases/pecl-tokyo_tyrant
sudo make install clean

Now re-run the php, and you may see the following error:

#php -v

PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20090626/tokyo_tyrant.so' - /usr/local/lib/php/20090626/tokyo_tyrant.so: Undefined symbol "ps_globals" in Unknown on line 0
Segmentation fault: 11

That’s because the Tokyo Tyrant module is placed at a wrong position in the PHP extension file. Edit the file:

sudo nano /usr/local/etc/php/extensions.ini

and move the following code to the END of the file (Yes, order does matter):

extension=tokyo_tyrant.so

And re-check your php:

#php -v

PHP 5.3.7 with Suhosin-Patch (cli) (built: Aug 23 2011 10:33:12)
Copyright (c) 1997-2011 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2011 Zend Technologies

Now everything is all set.

Unfortunately, you pretty much need to do the same thing every time when upgrading your PHP on FreeBSD box.

–Derrick

Our sponsors:

[Solved]Shared object “libtokyocabinet.so.8” not found

After upgrading the Tokyo Cabinet on my FreeBSD 8 system, I could not start Tokyo Cabinet:

ttserver
/libexec/ld-elf.so.1: Shared object "libtokyocabinet.so.8" not found, required by "ttserver"

After some investigations, I solved the problem.

1. First, go to the library directory. In my case, it is /usr/local/lib.

cd /usr/local/lib

2. Display all the libtokyocabinet.so:

ls libtokyocabinet.so*

which returns:

lrwxr-xr-x  1 root  wheel    24B Jan  9 20:17 libtokyocabinet.so -> libtokyocabinet.so.9.4.0
lrwxr-xr-x  1 root  wheel    24B Jan  9 20:17 libtokyocabinet.so.9 -> libtokyocabinet.so.9.4.0
-rwxr-xr-x  1 root  wheel   778K Jan  9 20:17 libtokyocabinet.so.9.4.0

So the solution to this problem is pretty obvious. The application is looking for libtokyocabinet.so.8 but a newer version of this library is available. A soft-link is good enough to solve this problem:

sudo ln -s libtokyocabinet.so libtokyocabinet.so.8

Here is the expected result:

ls libtokyocabinet.so*
lrwxr-xr-x  1 root  wheel    24B Jan  9 20:17 libtokyocabinet.so -> libtokyocabinet.so.9.4.0
lrwxr-xr-x  1 root  wheel    18B Jan  9 20:26 libtokyocabinet.so.8 -> libtokyocabinet.so
lrwxr-xr-x  1 root  wheel    24B Jan  9 20:17 libtokyocabinet.so.9 -> libtokyocabinet.so.9.4.0
-rwxr-xr-x  1 root  wheel   778K Jan  9 20:17 libtokyocabinet.so.9.4.0

Now I run the Tokyo Cabinet again, and the problem is gone!

ttserver
2010-01-09T20:26:12-06:00       SYSTEM  --------- logging started [13244] --------
2010-01-09T20:26:12-06:00       SYSTEM  server configuration: host=(any) port=1978
2010-01-09T20:26:12-06:00       SYSTEM  opening the database: *
2010-01-09T20:26:12-06:00       SYSTEM  service started: 13244
2010-01-09T20:26:12-06:00       INFO    timer thread 1 started
2010-01-09T20:26:12-06:00       INFO    worker thread 1 started
2010-01-09T20:26:12-06:00       INFO    worker thread 2 started
2010-01-09T20:26:12-06:00       INFO    worker thread 3 started
2010-01-09T20:26:12-06:00       INFO    worker thread 4 started
2010-01-09T20:26:12-06:00       INFO    worker thread 5 started
2010-01-09T20:26:12-06:00       INFO    worker thread 6 started
2010-01-09T20:26:12-06:00       INFO    worker thread 7 started
2010-01-09T20:26:12-06:00       INFO    worker thread 8 started
2010-01-09T20:26:12-06:00       SYSTEM  listening started

Our sponsors:

Updated: What technologies (or toys) I am playing recently?

Recently I am playing the following technologies (toys):

  1. Tornado Web – Not sure how long will Apache + PHP last. Time to learn Python.
  2. Tokyo Cabinet & Tyrant – Another one of my long term plans to step away from SQL.
  3. PHP API for Tokyo Tyrant — I decide to write my own because I cannot find a good one.
  4. Moving my development platform from Windows to FreeBSD. My servers have been already on FreeBSD for many years. I think now is a good time to move my desktop systems to FreeBSD as well. Why not Linux? Oh Well…
  5. LUA – I may pick up Lua if time is permitted.

Looks like I will have a busy time other than shoveling the snow this Winter.

–Derrick

Our sponsors:

Tokyo Cabinet is 31 times faster than MySQL!!!

Today I was comparing the performance of Tokyo Cabinet, MySQL and Memcached using PHP. The result of Tokyo Cabinet is really impressive!

(Note that these results are generated from my computer. If you perform your own benchmarks, then you might find different results due to different system configurations. In fact, I encourage you to perform your own benchmarks. )

1. Quick Result

Total time used to write 10000 records and retrieve them back by each candidate  (The lower the better):
Tokyo Cabinet: 18.87s
Memcached: 11.309s
MySQL: 562.21s

A completed result will be given below.

2. Testing environment

Hardware

  • Intel Pentium II 450MHz
  • 160 MB RAM

Software

  • FreeBSD 7.2
  • Apache 2.2.11
  • PHP 5.2.9
  • Tokyo Cabinet 1.4.20
  • Tokyo Tyrant 1.1.26
  • MySQL 5.1.34
  • Memcached 1.2.6

3. Test scenario

For each candidate, I measure its performance using the following strategies through PHP:

1.) The time used to create all of the necessary objects.

2.) The time used to connect to the database.

3.) The time used to store 10000 randomly generated data.

4.) The time used to read 10000 randomly generated data.

5.) The time used to close the connection.

4. Test results

Tokyo Cabinet

time index ex time %
Start 1243136772.72818600 0.00%
create 1243136772.72838700 0.000201 0.00%
connect 1243136772.73224200 0.003855 0.02%
vanish 1243136772.73526800 0.003026 0.02%
put 1243136781.53287200 8.797604 46.60%
get 1243136791.60574800 10.072876 53.35%
close 1243136791.60755900 0.001811 0.01%
Stop 1243136791.60782300 0.000264 0.00%
total 18.879637 100.00%


Memcached

time index ex time %
Start 1243131185.86078100 0.00%
create 1243131185.86092500 0.000144 0.00%
connect 1243131185.86688300 0.005958 0.05%
put 1243131191.49702900 5.630146 49.78%
get 1243131197.16868100 5.671652 50.15%
close 1243131197.16982400 0.001143 0.01%
Stop 1243131197.17003400 0.000210 0.00%
total 11.309253 100.00%

MySQL

time index ex time %
Start 1243136138.19049000 0.00%
create 1243136138.19064700 0.000157 0.00%
connect 1243136138.22529900 0.034652 0.01%
put 1243136150.70760600 12.482307 2.22%
get 1243136700.40063700 549.693031 97.77%
close 1243136700.40078900 0.000152 0.00%
Stop 1243136700.40104500 0.000256 0.00%
total 562.210555 100.00%

The overhead time (create objects, making connections, closing connections etc) of all candidates are about the same. The only difference is storing the record and retrieving the record. Both Tokyo Cabinet and MySQL took about 10 seconds to store 10000 records in the database. However, Tokyo Cabinet took about 10 seconds to retrieve the records while MySQL took about 550 seconds! That’s about 31 times longer!

benchmark-tokyocabinet-memcached-mysql

5. Materials

Here is the material I used to make this benchmark test.
Click here to download the package.

Please feel free to let me know if you have any question or comment.

Our sponsors:

Oracle, MySQL and Berkeley DB, what’s next?

When Oracle acquired Berkeley DB, the quality of Berkeley DB has gone worse after that.

Now Oracle acquired Sun who owns MySQL, will the history repeat again?

(By the way, I am going to fall in love with Tokyo Cabinet.)

Tokyo Cabinet is now the new black.

Our sponsors: