Tokyo Cabinet

[FreeBSD]How to run PECL Tokyo Tyrant in PHP 5.4

Posted on May 19, 2012 by Derrick

After upgrading to PHP 5.4.3 (which is the first version of PHP 5.4 available in the FreeBSD port), I found that the pecl-tokyo_tyrant could not be compiled. Here is the error messages:

cd /usr/ports/databases/pecl-tokyo_tyrant
sudo make

===>  pecl-tokyo_tyrant-0.6.0 cannot install: doesn't work with PHP version : 5 (Doesn't support PHP 5).
*** Error code 1

Stop in /usr/ports/databases/pecl-tokyo_tyrant.

/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:1827: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'php_tokyo_tyrant_class_methods'
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:1867: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'php_tokyo_tyrant_table_class_methods'
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:1911: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'php_tokyo_tyrant_query_class_methods'
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:1936: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'php_tokyo_tyrant_iterator_class_methods'
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c: In function 'php_tokyo_tyrant_query_object_new':
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2001: error: 'zend_class_entry' has no member named 'default_properties'
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c: In function 'php_tokyo_tyrant_iterator_object_new':
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2052: error: 'zend_class_entry' has no member named 'default_properties'
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c: In function 'php_tokyo_tyrant_object_new_ex':
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2090: error: 'zend_class_entry' has no member named 'default_properties'
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c: In function 'zm_startup_tokyo_tyrant':
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2166: error: 'php_tokyo_tyrant_class_methods' undeclared (first use in this function)
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2166: error: (Each undeclared identifier is reported only once
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2166: error: for each function it appears in.)
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2171: error: 'php_tokyo_tyrant_table_class_methods' undeclared (first use in this function)
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2176: error: 'php_tokyo_tyrant_query_class_methods' undeclared (first use in this function)
/usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/tokyo_tyrant.c:2182: error: 'php_tokyo_tyrant_iterator_class_methods' undeclared (first use in this function)
*** Error code 1

Stop in /usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0.
*** Error code 1

Stop in /usr/ports/databases/pecl-tokyo_tyrant.

Basically, it means that the pecl-tokyo_tyrant is not happy with the new version of PHP 5.4. I’ve tried several methods, including using pkg_add, compiling from source, copying a working tokyo_tyrant.so etc, unfortunately none of these methods works.

Finally, I found that the maintainer of this package has made a new version available(0.6.2) at github. Here is how I did it:

First, go to here to grab the source code. Make sure that you download the tag.gz version because it has the file attribute available.

Second, let’s remove the pecl-tokyo_tyrant from your system. You can either do it in the port or pkg_delete:

From port:

cd /usr/ports/databases/pecl-tokyo_tyrant
sudo make deinstall

Or using pkg_delete:

sudo pkg_delete pecl-tokyo_tyrant-0.6.0

Now, extract the files and compile it from source:

tar -zxvf mkoppanen-php-tokyo_tyrant-0.6.0-2-gb0d7e0c.tar.gz
cd mkoppanen-php-tokyo_tyrant-b0d7e0c
sudo phpize
sudo ./configure
sudo make
sudo make install

and don’t forget to include the .so file in your configuration:

sudo nano /usr/local/etc/php/extensions.ini

Add the following (if not available) to the end of the file:

extension=tokyo_tyrant.so

Finally, let’s restart the Apache:

sudo apachectl stop
sudo apachectl start

That’s it! Have fun with Tokyo Tyrant with PHP 5.4.

–Derrick

Our sponsors:

PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/local/lib/php/20090626/tokyo_tyrant.so’ – /usr/local/lib/php/20090626/tokyo_tyrant.so: Undefined symbol “ps_globals” in Unknown on line 0

Posted on August 23, 2011 by Derrick

Every time I upgrade the PHP and related extensions, I always run into the same problem with some PHP-PECL modules. Recently with the new release of PHP 5.3.7, there is no exception for the trouble. So what kind of trouble I have today? The Tokyo Tyrant (The middleman between my program and Tokyo Cabinet).

#php -v

PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20090626/tokyo_tyrant.so' - /usr/local/lib/php/20090626/tokyo_tyrant.so: Undefined symbol "ps_globals" in Unknown on line 0

This problem was caused by a version mis-match. PHP is running at a later version, which the Tokyo Tyrant extension is running at an earlier version. To solve this computer problem, all you need is to re-install the PHP-Tokyo-Tyrant extension, i.e.,

cd /usr/ports/databases/pecl-tokyo_tyrant
suod make deinstall clean
sudo make install

And later, you may or may not get the following error message:

...
cc -shared  .libs/tokyo_tyrant.o .libs/tokyo_tyrant_funcs.o .libs/connection.o .libs/session_funcs.o .libs/server_pool.o .libs/failover.o .libs/session.o  -Wl,--rpath -Wl,/usr/local/lib -Wl,--rpath -Wl,/usr/local/lib -L/usr/local/lib -ltokyotyrant -llua -ltokyocabinet /usr/local/lib/liblzo2.so -llzma -lbz2 -lz -lrt -lm  -Wl,-soname -Wl,tokyo_tyrant.so -o ./.libs/tokyo_tyrant.so
/usr/bin/ld: cannot find -llua
*** Error code 1

Stop in /usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0.
*** Error code 1

Stop in /usr/ports/databases/pecl-tokyo_tyrant.

The problem is related to Lua. By default, the extension enable the Lua option, which I have a hard time to make it run properly on FreeBSD box. Since my website does not use the Lua with Tokyo Cabinet, I simply disable it:

sudo nano /usr/ports/databases/pecl-tokyo_tyrant/work/tokyo_tyrant-0.6.0/Make

Look for the keyword: lua
And you will find something like the following:

TOKYO_TYRANT_SHARED_LIBADD = -R/usr/local/lib -L/usr/local/lib -R/usr/local/lib -L/usr/local/lib -ltokyotyrant -llua -ltokyocabinet -lbz2 -lz -lrt -lm -ltokyocabinet -llzo2 -llzma -lbz2 -lz -lrt -lm

Delete the option: “-llua”, i.e.,

TOKYO_TYRANT_SHARED_LIBADD = -R/usr/local/lib -L/usr/local/lib -R/usr/local/lib -L/usr/local/lib -ltokyotyrant -ltokyocabinet -lbz2 -lz -lrt -lm -ltokyocabinet -llzo2 -llzma -lbz2 -lz -lrt -lm

and re-start the installation:

cd /usr/ports/databases/pecl-tokyo_tyrant
sudo make install clean

Now re-run the php, and you may see the following error:

#php -v

PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20090626/tokyo_tyrant.so' - /usr/local/lib/php/20090626/tokyo_tyrant.so: Undefined symbol "ps_globals" in Unknown on line 0
Segmentation fault: 11

That’s because the Tokyo Tyrant module is placed at a wrong position in the PHP extension file. Edit the file:

sudo nano /usr/local/etc/php/extensions.ini

and move the following code to the END of the file (Yes, order does matter):

extension=tokyo_tyrant.so

And re-check your php:

#php -v

PHP 5.3.7 with Suhosin-Patch (cli) (built: Aug 23 2011 10:33:12)
Copyright (c) 1997-2011 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2011 Zend Technologies

Now everything is all set.

Unfortunately, you pretty much need to do the same thing every time when upgrading your PHP on FreeBSD box.

–Derrick

Our sponsors:

[Solved]Shared object “libtokyocabinet.so.8” not found

Posted on January 9, 2010 by Derrick

After upgrading the Tokyo Cabinet on my FreeBSD 8 system, I could not start Tokyo Cabinet:

ttserver
/libexec/ld-elf.so.1: Shared object "libtokyocabinet.so.8" not found, required by "ttserver"

After some investigations, I solved the problem.

1. First, go to the library directory. In my case, it is /usr/local/lib.

cd /usr/local/lib

2. Display all the libtokyocabinet.so:

ls libtokyocabinet.so*

which returns:

lrwxr-xr-x  1 root  wheel    24B Jan  9 20:17 libtokyocabinet.so -> libtokyocabinet.so.9.4.0
lrwxr-xr-x  1 root  wheel    24B Jan  9 20:17 libtokyocabinet.so.9 -> libtokyocabinet.so.9.4.0
-rwxr-xr-x  1 root  wheel   778K Jan  9 20:17 libtokyocabinet.so.9.4.0

So the solution to this problem is pretty obvious. The application is looking for libtokyocabinet.so.8 but a newer version of this library is available. A soft-link is good enough to solve this problem:

sudo ln -s libtokyocabinet.so libtokyocabinet.so.8

Here is the expected result:

ls libtokyocabinet.so*
lrwxr-xr-x  1 root  wheel    24B Jan  9 20:17 libtokyocabinet.so -> libtokyocabinet.so.9.4.0
lrwxr-xr-x  1 root  wheel    18B Jan  9 20:26 libtokyocabinet.so.8 -> libtokyocabinet.so
lrwxr-xr-x  1 root  wheel    24B Jan  9 20:17 libtokyocabinet.so.9 -> libtokyocabinet.so.9.4.0
-rwxr-xr-x  1 root  wheel   778K Jan  9 20:17 libtokyocabinet.so.9.4.0

Now I run the Tokyo Cabinet again, and the problem is gone!

ttserver
2010-01-09T20:26:12-06:00       SYSTEM  --------- logging started [13244] --------
2010-01-09T20:26:12-06:00       SYSTEM  server configuration: host=(any) port=1978
2010-01-09T20:26:12-06:00       SYSTEM  opening the database: *
2010-01-09T20:26:12-06:00       SYSTEM  service started: 13244
2010-01-09T20:26:12-06:00       INFO    timer thread 1 started
2010-01-09T20:26:12-06:00       INFO    worker thread 1 started
2010-01-09T20:26:12-06:00       INFO    worker thread 2 started
2010-01-09T20:26:12-06:00       INFO    worker thread 3 started
2010-01-09T20:26:12-06:00       INFO    worker thread 4 started
2010-01-09T20:26:12-06:00       INFO    worker thread 5 started
2010-01-09T20:26:12-06:00       INFO    worker thread 6 started
2010-01-09T20:26:12-06:00       INFO    worker thread 7 started
2010-01-09T20:26:12-06:00       INFO    worker thread 8 started
2010-01-09T20:26:12-06:00       SYSTEM  listening started

Our sponsors:

Updated: What technologies (or toys) I am playing recently?

Posted on November 16, 2009 by Derrick

Recently I am playing the following technologies (toys):

Tornado Web – Not sure how long will Apache + PHP last. Time to learn Python.
Tokyo Cabinet & Tyrant – Another one of my long term plans to step away from SQL.
PHP API for Tokyo Tyrant — I decide to write my own because I cannot find a good one.
Moving my development platform from Windows to FreeBSD. My servers have been already on FreeBSD for many years. I think now is a good time to move my desktop systems to FreeBSD as well. Why not Linux? Oh Well…
LUA – I may pick up Lua if time is permitted.

Looks like I will have a busy time other than shoveling the snow this Winter.

–Derrick

Our sponsors:

Tokyo Cabinet is 31 times faster than MySQL!!!

Posted on May 23, 2009 by Derrick

Today I was comparing the performance of Tokyo Cabinet, MySQL and Memcached using PHP. The result of Tokyo Cabinet is really impressive!

(Note that these results are generated from my computer. If you perform your own benchmarks, then you might find different results due to different system configurations. In fact, I encourage you to perform your own benchmarks. )

1. Quick Result

Total time used to write 10000 records and retrieve them back by each candidate (The lower the better):
Tokyo Cabinet: 18.87s
Memcached: 11.309s
MySQL: 562.21s

A completed result will be given below.

2. Testing environment

Hardware

Intel Pentium II 450MHz
160 MB RAM

Software

FreeBSD 7.2
Apache 2.2.11
PHP 5.2.9
Tokyo Cabinet 1.4.20
Tokyo Tyrant 1.1.26
MySQL 5.1.34
Memcached 1.2.6

3. Test scenario

For each candidate, I measure its performance using the following strategies through PHP:

1.) The time used to create all of the necessary objects.

2.) The time used to connect to the database.

3.) The time used to store 10000 randomly generated data.

4.) The time used to read 10000 randomly generated data.

5.) The time used to close the connection.

4. Test results

	time index	ex time	%
Start	1243136772.72818600	–	0.00%
create	1243136772.72838700	0.000201	0.00%
connect	1243136772.73224200	0.003855	0.02%
vanish	1243136772.73526800	0.003026	0.02%
put	1243136781.53287200	8.797604	46.60%
get	1243136791.60574800	10.072876	53.35%
close	1243136791.60755900	0.001811	0.01%
Stop	1243136791.60782300	0.000264	0.00%
total	–	18.879637	100.00%

Memcached

	time index	ex time	%
Start	1243131185.86078100	–	0.00%
create	1243131185.86092500	0.000144	0.00%
connect	1243131185.86688300	0.005958	0.05%
put	1243131191.49702900	5.630146	49.78%
get	1243131197.16868100	5.671652	50.15%
close	1243131197.16982400	0.001143	0.01%
Stop	1243131197.17003400	0.000210	0.00%
total	–	11.309253	100.00%

MySQL

	time index	ex time	%
Start	1243136138.19049000	–	0.00%
create	1243136138.19064700	0.000157	0.00%
connect	1243136138.22529900	0.034652	0.01%
put	1243136150.70760600	12.482307	2.22%
get	1243136700.40063700	549.693031	97.77%
close	1243136700.40078900	0.000152	0.00%
Stop	1243136700.40104500	0.000256	0.00%
total	–	562.210555	100.00%

The overhead time (create objects, making connections, closing connections etc) of all candidates are about the same. The only difference is storing the record and retrieving the record. Both Tokyo Cabinet and MySQL took about 10 seconds to store 10000 records in the database. However, Tokyo Cabinet took about 10 seconds to retrieve the records while MySQL took about 550 seconds! That’s about 31 times longer!