BLOG: Working on TheCLQ.com / AllGameServers.com

[BattleMode]

All this looks very cool. Amazing how you come up with this so fast. Is there any particular reason on your server that you went with 3GB of RAM instead of an even 4? I would think for an IMDB that you would want as much as possible.

The version of CentOS I am using can not address the 4th GB and my provider has no 64 bit experience. At some point the server will probably collapse and then I will figure out what the next step will be.

[BattleMode]

Unfortunately my provider made a small mistake when configuring CentOS (did not want JBOD, got it anyway). That is unfortunate since I intended to get the scanner up and running this weekend. Next week I will leave for vacation to England for two weeks, so this could delay the moment the scanner starts about 4 weeks. Oh well.

Got my new EEE 900, so now I will spend some time playing with that instead .

[LiquidAftermath]

Hey everyone... I dont even know if this is where I should post... But I used to use the CLQ back when I played Half-Life 1... I went to look at the stats the other day and the site was shut down very very sad day for me. I was wondering if you have any of the old stats, cause I would really like to get mine, I know you have heard that a million times. But hey its worth a shot, Half-Life was a big part of my life when I was starting with computers... No I work and develop for Valve. So it would be nice to see my old stats again if thats at all possible. Thanks.

Kyle Hastings
**(PKS)**LiquidAftermath
Was #1 At one Time

[BattleMode]

Hey everyone... I dont even know if this is where I should post... But I used to use the CLQ back when I played Half-Life 1... I went to look at the stats the other day and the site was shut down very very sad day for me. I was wondering if you have any of the old stats, cause I would really like to get mine, I know you have heard that a million times. But hey its worth a shot, Half-Life was a big part of my life when I was starting with computers... No I work and develop for Valve. So it would be nice to see my old stats again if thats at all possible. Thanks.

Kyle Hastings
**(PKS)**LiquidAftermath
Was #1 At one Time

You work for Valve? Cool! I actually have a backup of the old database. Have not been able to restore it yet (5 year old backup, none of the software is around any more) but I intend to try anyway. Just hope those tapes still work.

[LiquidAftermath]

You have no idea how awesome that would be for all the OLD SCHOOL gamers lol... I loved the CLQ you did a AWESOME JOB WITH IT! I remember when I got a PM from a friend telling me "HEY!!! Your Ranked #1 in the WORLD!!!!" It was an awesome feeling.

[BattleMode]

My provider fixed the machine they made for me. It is now exactly they way I wanted it to be. 1 TB storage for scan data and 320 GB storage for the rest (website, hyperstore, etc.).

I leave on vacation in 2 days (2 weeks England) but perhaps I can do 1 or 2 things in the meantime. First thing to do will be to move all the stuff I put on the BBS server to my new AGS/CLQ server.

[BattleMode]

Got my patent pending HyperStore(tm) working on the AGS/CLQ server.

SAVE / LOAD 1000 small objects
1000 SAVES: 58 ms
1000 LOADS: 41 ms
1000 SURE SAVES: 141 ms
1000 SLOW LOADS: 91 ms

SAVE / LOAD 1000 big objects
1000 SAVES: 182 ms
1000 LOADS: 145 ms
1000 SURE SAVES: 655 ms
1000 SLOW LOADS: 186 ms

[BattleMode]

Transferred the temporary http://allgameservers.com and http://www.theclq.com sites to the new servers.

[BattleMode]

Moved the pageview counter from http://counter.digits.com to hyperstore.

[BattleMode]

Got the scanner working on the new server:

Code: Select all

    [83.98.163.86:7777] => Array
        (
            [status] => up
            [name] => #1 BBS Titan RPG vCTF
            [maxplayers] => 32
            [players] => 9
            [now] => 1217035276
            [retries] => 0
            [map] => VCTF-JeffsMap-Beta2
            [playerlist] => Array
                (
                    [k@0$] => 3
                    [GHOST] => 0
                    [pepe_559] => 0
                    [CALLER] => 0
                    [DbD2] => 0
                    [Suicide_Squid] => 1
                    [Leeroy_Jenkins] => 0
                    [BUDGIE] => 0
                    [kille_spider] => 2
                )

        )

[BattleMode]

It took my UT2004 server collector 28 seconds to locate 1379 UT2004 servers.

It took my server classification tool 2869 seconds to classify them:

CLASS 1 SERVERS: 173
CLASS 2 SERVERS: 1133
CLASS 3 SERVERS: 0
CLASS 4 SERVERS: 0
CLASS 5 SERVERS: 0
CLASS 6 SERVERS: 0
CLASS 7 SERVERS: 73

[BattleMode]

One of the problems to solve is massive parallel scanning of lots of servers. Made a small threading library to help with that. Did some thorough testing and everything seems to work fine.

[OpT_fr]

do you have to synchronize all these scanning ?

[BattleMode]

do you have to synchronize all these scanning ?

No, but I need to do a lot of scanning so I need high parallelism.

[BattleMode]

A lot of things have changed since I made the first CLQ 10 years ago. Most noticeably all kinds of anti virus and anti hack stuff has been added to Linux that gets in the way if you want to introduce massive parallelism.

I could work around this with a customized Linux kernel, but choose to work around this instead.

I now have a working "superscalar" scanner that can scan 1304 active UT2004 servers in 5 seconds. CPU load is about 5% for this, it scales nicely so all cores are used. All scan data is preprocessed, bzip compressed and stored on disk. This will allow me to achieve the 15.000 servers I wanted to scan per minute with great ease. I might even increase my ambitions and scan servers more often than I intended.

There is still a lot of room for improvement. It is highly configurable allowing me to change it in the future if I start scanning larger amounts of servers. Also the scanner is not connected to the processor yet (which might be caused by the fact I did not build the processor yet ) so the database is not optimal yet.

The amount of data needed for this is 4 megabytes (compressed data). This means a single server scan takes up 3 kilobytes. This means the 1 TB disk can store 330.000.000 server scans. With 15000 scans per minute this means I can store 22000 minutes (or 15 days) of data on that disk. Hm, that is not very much actually. OTOH I do not really need all that history, I only need to store the resulting statistics. Oh well, I will figure out this part later on. I negotiated 2.25 TB of traffic per month for this with my provider, didn't realize I might actually use it all .

[BattleMode]

Already gave classification some though.

Current setup will become:

CLASS 1: Server up and 2 or more players (max missed scans: 2)
CLASS 2: Server up (max missed scans: 2)
CLASS 3: max missed scans: 10
CLASS 4: max missed scans: 20
CLASS 5: max missed scans: 30
CLASS 6: new server never scanned
CLASS 7: never worked or > 30 missed scans

[BattleMode]

Integrated the first part if the data processor with the scanner. Servers are now classified in real time. Hyperstore is used for all data processing. Also fixed some hard to track bugs in my threading stuff.

[BattleMode]

Improved the superscalar scanner. It now scans over 1000 servers in less than 3 seconds.

[BattleMode]

The UT2004 scanner is operational !!!

From now on all UT2004 stats will be collected.

Classification as implemented:

CLASS 1: Server up and 2 or more players (max missed scans: 2) - once per minute
CLASS 2: Server up (max missed scans: 2) - once per minute (will become slower)
CLASS 3: max missed scans: 10 - once per 10 minutes
CLASS 4: max missed scans: 20 - once per 100 minutes
CLASS 5: max missed scans: 30 - once per 1000 minutes
CLASS 6: new server never scanned - manual scanning only
CLASS 7: never worked or > 30 missed scans - manual scanning only


Current values:

CLASS 1 SERVERS: 187
CLASS 2 SERVERS: 1063
CLASS 3 SERVERS: 78
CLASS 4 SERVERS: 51
CLASS 5 SERVERS: 0
CLASS 6 SERVERS: 0
CLASS 7 SERVERS: 0

[BattleMode]

Put some global statistics on the homepage now: http://allgameservers.com