After a while of using statnet I decided to switch my VPS provider. The main reason behind this was the low bandwidth statnet offered: 150GB monthly. I often had to shutdown some of my projects in order to stay within the limit, which was obviously an inconvenience. The new provider I chose is intovps. It may not exactly be what I was wishing for (unmetered), but still 150GB to 0.5TB is a huge improvement. After doing some math I realized I could spend all the month uploading/downloading at 200kB/s. Should be enough

One thing I was worried about though was performance. Seeing as I was being handed three times the bandwidth of my old provider for the same price I felt obligated to check where the catch is. And I did find one:

BYTE UNIX Benchmarks (Version 4.1-wht.2)
System -- Linux robus.info 2.6.18-128.2.1.el5.028stab064.7 #1 SMP Wed Aug 26 15:47:17 MSD 2009 i686 GNU/Linux
/dev/simfs            20971520   2971716  17999804  15% /
Start Benchmark Run: Sun May  9 00:18:09 CEST 2010
 00:18:09 up 5 min,  1 user,  load average: 0.07, 0.03, 0.00
End Benchmark Run: Sun May  9 00:33:55 CEST 2010
 00:33:55 up 21 min,  1 user,  load average: 13.30, 5.93, 3.01

                     INDEX VALUES
TEST                                BASELINE     RESULT      INDEX

Dhrystone 2 using register variables 376783.7 2477189.0      65.7
Double-Precision Whetstone              83.1     1496.4      180.1
Execl Throughput                       188.3     1970.4      104.6
File Copy 1024 bufsize 2000 maxblocks 2672.0    58391.0      218.5
File Copy 256 bufsize 500 maxblocks   1077.0    17051.0      158.3
File Read 4096 bufsize 8000 maxblocks 15382.0  339885.0     221.0
Pipe Throughput                     111814.6   427653.0       38.2
Pipe-based Context Switching         15448.6   134913.0       87.3
Process Creation                       569.3     5974.8      104.9
Shell Scripts (8 concurrent)            44.8      384.6       85.8
System Call Overhead                114433.5   349000.1       30.5
                                                                 =========

     FINAL SCORE                                              98.7

I can’t say this is bad, but it is worse than my earlier server, which cut off at around 150. Another thing I noticed were the hard limits on CPU. I was not allowed to consume more than 50% at any time. It’s still sufficient for my needs; but I was wondering at what cost this would come.

So the first test I ran was a siege on the main page. ( siege -b robus.info -c 50)

Transactions:                    1172 hits
Availability:                 100.00 %
Elapsed time:                   4.80 secs
Data transferred:               1.43 MB
Response time:                  0.18 secs
Transaction rate:             244.17 trans/sec
Throughput:                     0.30 MB/sec
Concurrency:                   45.02
Successful transactions:        1172
Failed transactions:               0
Longest transaction:            0.48
Shortest transaction:           0.00

Okay, but I already made some improvements earlier, so that the page was only read from the cache. But hey, at least I see that nginx, which I installed for the first time, was able to run at 250 transactions/sec.

The next test I ran was a siege on wordpress; its results were terrible: around 1.5 trans/sec.  After making some performance improvements it cut off pretty well though:

Transactions:                    1320 hits
Availability:                 100.00 %
Elapsed time:                   7.02 secs
Data transferred:               5.94 MB
Response time:                  0.25 secs
Transaction rate:             188.03 trans/sec
Throughput:                     0.85 MB/sec
Concurrency:                   47.60
Successful transactions:        1367
Failed transactions:               0
Longest transaction:            2.38
Shortest transaction:           0.00

The next test will only be my subjective feeling on how fast my little Nasza-klasa fetcher will be. Though currently the database is still uploading; so this will have to wait (ULing ~2GB of data at 32kB/s, *sigh*).

Since the server switching already brought quite a few changes (lighttpd -> nginx), which went suprisingly smooth after I had figured out how to set up PHP properly (converting rewrites was as simple as deleting the quotes, duh) – I was also considering migrating MySQL->PostgreSQL, though this is obviously going to be a much harder task. First of all, I’ll have to figure out if WordPress even supports it; and even if it does – I suppose its long outdated and not maintained, as most non-mysql projects are. Oh well, Im getting ahead of myself.

Anyways, I’m considering the switch because MySQL has gotten much slower after the tables exceeded 1 million rows, hell, even dropping the database took 17 seconds while I was moving. And as far as I’ve been told PostgreSQL cuts off better performance-wise. It would also be a relief to finally work on a real relational database, since the data inconsistency has been growing ever since I split the users table. I’ll prolly write another post on this as soon as I get some benchmarks.

While porting some of my projects to the new directory layout, I noticed something strange. Some time ago I was complaining about how vnc4server kept asking for root’s settings. Suprisingly, this time vnc4server used www-data’s settings – as it always should have! The only difference between my old server I can name is the newer debian version (4.0 -> 5.0). I’m absolutely clueless on how this could have affected it, but I’m happy things finally work the way they should have all along.

I’ve spent AGES trying to create some automated tool for website screenshots. Eventally I came across a tool called Webshot. It worked fine, until I tried automating it. Obviously, I can’t run a GUI tool on a server; neither can I run a windows console app on linux. After searching for a while I once again came across a working tool – CutyCapt. The tool itself works fine, but I’ve had trouble running it on a headless server. The app’s site suggests to run it with Xvfb, however…

robus@robus.info:/$ Xvfb -screen 0 1024x768x24
(EE) XKB: Couldn't open rules file /usr/share/X11/xkb/rules/base
Could not init font path element /usr/share/fonts/X11/cyrillic, removing from list!
[config/hal] couldn't initialise context: (null) ((null))

As you can see it works just fine on 24bit. But…

robus@robus.info:/$ Xvfb -screen 0 1024x768x32
Fatal server error:
Couldn't add screen 0

One might ask why bother with 32bit? Well…

24 bit

vs

32bit

The difference is obvious after looking at these two pictures. How to achieve this, however, hasnt been obvious most of the time. I’ve literally searched for ages (1, 2,3, just to name a few places) and posted my own forum question. Sadly, this hasn’t yielded an answer either. I was even determined enough to compile everything on my own and tinker with the code, but I couldn’t find Xvfb’s source anywhere (X11 I suppose?).

Now the only reasonable alternative I came across was vnc4server. Though the main problem I’ve encountered when using it is…

capt: cannot connect to X server localhost:55199.0

This happened around 65% of the time. It really didn’t make any sense, since I didn’t change the website I was fetching once. Now this would’ve saved tons of my time if I knew this earlier: It won’t work if the screen number is >32000. Sigh!

Finally ready to let go Xvfb I developed this little python script:

import os
import sys
from random import randint
sys.argv[1] = randint(1000,32000)
for i in xrange(2):
    os.system("/bin/bash -c \"vnc4server :%s -ac -depth 32\"" % (sys.argv[1]))
    if (os.system("/bin/bash -c \"export DISPLAY=localhost:%s.0; ./capt --max-wait=5000 --url=%s --out=%s --user-styles=http://yui.yahooapis.com/2.8.0r4/build/reset/reset-min.css --min-width=1024;\"" %(sys.argv[1],sys.argv[2],sys.argv[3]))) == 0:
        os.system("/bin/bash -c \" vnc4server -kill :%s\"" % (sys.argv[1]))
        exit(0)
    os.system("/bin/bash -c \" vnc4server -kill :%s\"" % (sys.argv[1]))
    sys.argv[1] = randint(1000,32000)

And voila, all of the sudden everything works the way its supposed to. Im honestly suprised nobody has posted a tutorial on how to do this yet, especially as I’ve seen several questions around. Though, I’ve encountered another problem when trying to integrate this with a php script.

vnc4server: Wrong type or access mode of /root/.vnc.
capt: cannot connect to X server localhost:13798.0

This really didn’t make any sense either. Since the web-server was run by www-data, and as I later checked – so was the python script, why on earth was vnc4server still trying to access /root/? Eventually I gave up and chmoded the entire directory 777. This suprisingly didn’t help either. Finally I tried to chown /root/ to www-data altogether, which finally made it work (.vnc would’ve been enough). Once again, knowledge nowhere related to what I was doing – but it would’ve saved so much time if I had known it earlier!

I hope this post is going to be helpful to the next person annoyed with Xvfb’s unwillingness to run at 32bit. Also, you can see the script I’ve made live here.