BLOG: Working on TheCLQ.com / AllGameServers.com

[BattleMode]

Succeeded in reducing the page generation time of the current homepage to between 400 and 600 μs (that's microseconds not milliseconds). This will of course become a lot slower when more content is added, but it means the foundation I am building on is now very efficient. The two tools I use to make everything as fast as possible are memcached and eAccelerator (and then some smart code to utilize this all).

The amount of collected data worries me a bit. 1200 servers already resulted in 5 gigabytes of compressed data in less than a day. With 100.000 servers that means I can fill a terabyte in 2 days. For now I can ignore this, but I will have to find ways to work around this. One trick would be to transform the data from time => servers to servers => time so the data compressor can achieve much higher compression. That would be a very IO intensive operation though.

Another puzzle I have not cracked yet is HyperStore vs pure MySQL. Since I just made HyperStore so I am not sure yet where exactly it will become slower that MySQL. There are too many unknown variables here to determine that. I will probable have to take an educated guess and accept that I will have to rewrite some stuff in the future.

Some general idea I have now is:

Real time data (updated every 60 seconds):
* server data (map, players, score) including real time ranking based on current number of players
* player data (where is the player playing, might be multiple servers for common names)
I think a visitor should be able to select favorite servers and players and get a real time overview of those (e.g. how active are my favorite servers, and where are my friends playing at the moment). This will also allow things like content snippets clans could add to their web pages. The 60 second timeslot will be split into two segments, 30 seconds of collecting server data (the superscalar scanner can scan at least 10.000 servers in that time, probably even more) and 30 seconds of data processing.

Delayed data (probably recalculated once every hour or even less often):
* server rankings, e.g. average number of players
* player rankings
* map rankings

[BattleMode]

Hm made some stupid mistake. Easiest way to solve it was to delete all data and start from scratch again. Oh well, all part of the process.

[BattleMode]

Arived in England yesterday. Have a EEE 900 with Ubuntu with me so I can play a little here (not games, but PHP hacking). WiFi is not very reliable so SSH is not working very well. Used SSH to make a small web based PHP editor so I can fool around a little.

BTW this driving on the left side of the road takes some serious getting used to. BTW2 the weather is perfect today.

[thomaster]

Well, good work then Personally I never touch a computer, when I'm on holiday...

[BattleMode]

Well, good work then Personally I never touch a computer, when I'm on holiday...

Some stats from the scanner (currently I only scan UT2004):

TOTAL PLAYERS: 1010
TOTAL SERVERS: 1216
TOTAL SCANS: 5260
CLASS 1 SERVERS: 114
CLASS 2 SERVERS: 1102
CLASS 3 SERVERS: 0
CLASS 4 SERVERS: 24
CLASS 5 SERVERS: 139
CLASS 6 SERVERS: 0
CLASS 7 SERVERS: 0

[BattleMode]

Site now has a live server top 25: http://allgameservers.com

Updated once per minute.

[BattleMode]

Implemented live server statistics now.

Live #1 BBS Titan RPG vCTF: http://allgameservers.com/ags/server.ph ... 63.86:7777

Live Titan Internet Freon Server #2: http://allgameservers.com/ags/server.ph ... 2.199:7880

Live clickable server top 25: http://allgameservers.com

[BattleMode]

Made "active players" more accurate now by eliminating bots etc. I think I got most of these now. I do not entirely trust those "you knows it" servers but having no access to the actual game that part will have to wait a few weeks.

[BattleMode]

I implemented a transformation to be able to store and process raw scan data more efficiently. Currently the scanner collects 18 MB raw data per hour. By transforming and compressing the data I can store this into 400 kB of disk space. I achieved 44x compression. This means I can store roughly 40 TB (tera bytes) of raw data on the 1 TB data storage disk I have in the server.

The transformation of 1 hour of data takes 11 seconds. Assuming I currently scan about 1% of all game servers in the world this means I would need 1100 seconds to transform one hour of data once I start scanning everything.

The transformation is the toughest part of all processing since it requires a lot of disk I/O. I am assuming actual processing (determining ranks etc.) will take less time, especially since the highly compressed data minimizes disk I/O and after transformation the order of the data on disk and in memory is optimized for per server processing.

The high compression I achieved now also means I can store 3 years of raw data of ALL game servers in the world on a single 1 TB disk. This will allow for some interesting functionality in the future. It will also allow me to change or tune algorithms in the future and recalculate everything.

I also implemented configurable hashing so in the future I will be able to process data in segments to minimize memory usage. The current set fits into memory but this will change once I start scanning 100 times more servers.

[BattleMode]

Discovered a very nasty bug in the bzdecompress function. Will take me some time to work around it I am afraid. It also meant my stats where wrong (when I reached > 1000x compression I started to get suspicious).

[BattleMode]

Figured out the problem. It seems the php decompression function, unlike the command line decompressor, can not handle multiple independently compressed blocks. Discovering this also revealed an efficiency error in my code which I now fixed..

Transforming 1 hour of scan data now takes 8 seconds (was 11 before) which is perfect for the large scale data processing I intend to do in the future (about 100x more data). Also my estimate that 1 TB of storage will hold 40 TB of raw scan data (about 3 years for 100.000 servers) still stands. I made a thorough check script to make sure the transformation does not lose any data and no data gets lost there.

So summarizing the scanner achieves 4.4x compression, the transformation increases this to 44x compression,

[BattleMode]

Made a small module for handling multidimensional data (aka star queries or OLAP, or more like ROLAP in my case). I needed this to force myself to lock into a certain data model and to prepare for 1001 different ways to look at data. I do not know how well MySQL will handle this, but there are alternatives like Cubulus in case it does not.

Also reorganized my code a bit. Bundled all configuration related stuff and all testing related stuff (I usually make 1 test module for each important module).

This means I am nearly ready for the next step, player statistics.

[BattleMode]

I implemented the first processor now. It processes 1 hour of server data (>1000 servers) in 5 seconds. It determines server popularity (no nice web page yet, it's all behind the screens).

The multidimensional data module I made yesterday can determine the server top 25 in a certain week (CLQ was always week based) in 11ms. This is without caching.

The basic processor I now made should be easily extendible to all kinds of other statistics as well.

So the picture is now complete:
- the scanner collects data and determines real time statistics (e.g. how many people are playing NOW)
- the transformer prepares data for long term storage and makes it ready for processing
- the processor determines week based statistics (e.g. how many hours did people play on a server in a certain week)

Just need to
- add many more statistics
- make them available on the website
- probably the site could use a tad more graphic design
- add many other games
- etc. etc.

[BattleMode]

The very first stats .

Period: Mon Jul 28th - Sun Aug 3rd 2008

Number of player-hours for each server. Most bots have been excluded but special bots like those on the "You Knows It" servers have yet to be eliminated as well.

#1 3265 UMGLEAGUE.COM | ONS #1 [Rent Your UT3 Server Today]
#2 3211 Omnip)o(tentS {ONS}
#3 2940 ++[EZE]Sneaky Ass-RiPpIn Sniper Server++
#4 2833 Nachos and Beer
#5 2698 Death Warrant:Monster Mash RPG - Chicago
#6 2645 [YKI] ++++ < Camper And Sniper > ++++ - You Knows It!
#7 2599 [CTU] <<< Sniper Heaven >>>
#8 2523 [YKI] ++++ < King_Dingys > ++++ Onslaught Deviance
#9 2514 [YKI] ++++ < Oldskool Onslaught > ++++ - You Knows It!
#10 2390 Death Warrant:Monster Mash RPG - Chicago Server 2
#11 2327 [UTAN] Multiplay.co.uk :: Public Onslaught #1 (Normal)
#12 2214 *** assclan.de *** --> [ICTF Playground]
#13 2138 FKs #4 Debrecen/Tank Maps w/ AB Mod! - FKMOD.COM
#14 2027 AoB clan server
#15 1737 CollegeDate.com/UT2004 Demo
#16 1687 Monster Madness - RPG WoP Invasion
#17 1567 Death Warrant: Spam Vikings (Dedicated to DW>Damn_Weasel)
#18 1554 {V][M} VCTF Server http://www.vehiclemasters.com
#19 1538 <<<<>>>>
#20 1536 #1 BBS Titan RPG vCTF
#21 1530 game2.techcom.cz: the enjoyable onslaught
#22 1523 >>> ESG Sniping MatriX FreeZeTag <<< http://www.esg-land.com
#23 1483 <{CTC}> VCTF Server - http://www.vctf.us
#24 1465 TeamFluff.com's Excessive Fun - Excessive Weapons
#25 1459 -FA2K- ****TEAM ARENA MASTER**** -FA2K-

[BattleMode]

Improved the data model and experimented with various MySQL statements. Now reduced the time needed to determine the server top 25 in a certain week from 11ms to 0.7 ms (without caching). The underlying data structure is universal so I should be able to use it for any type of ranking. Even though the number of sql transactions doubled and the number of MySQL round trips became 100x more I still succeed in processing 1 hour of data (1000+ servers) in 5 seconds.

This will become more once I add more types of statistics and a LOT more once I start scanning 100,000 servers instead of 1000.

I used INSERT INTO ... ON DUPLICATE KEY UPDATE for this. I never used it before (it is MySQL specific) but I have to say it works very well.

I also did some thinking on the player ranking. Based on what I learned from CLQ 1.0 and some gaming experience I think I will try the following systems (feedback etc. will help me determine what works best but I have to start somewhere). No matter what I choose it will never make everyone happy. in part I will solve this by using 2 different ranking systems, in part I will need a lot of feedback and experimenting here, in part it will be an impossible to solve problem.

CLQ1 (which will be close to the original system) will be based on how fast players collect points. The faster they do that, the more points they will get. So it will be based on peak performance which has as advantage that fooling around or having a bad day does not affect your ranking. It also means you cannot wreck someone else's score (cheat) by playing using their name.

Based on a score sample SCORE and a time frame of SECONDS the increase in the internal value fro CLQ1 will be:

TOTAL += SCORE/SECONDS*60) ^ 3) / 60 * SECONDS

Using ^3 means the faster someone collects points in a certain time frame, the more significant it will be. I used to exclude the top part in CLQ 1.0 but I think this will no longer be needed since I can determine the time frame much more accurately than I used to and I will no longer combine all servers but determine statistics for each individual server.

For presentation I will use:

1 + 1000 * Log10 (TOTAL)

The Log10 is used to equalize the scale no matter the style of the game. In other words, it looks better.

CLQ2 will be new. It will be based on the in game ranking (e.g. who won the game, but also during the game, e.g. once per minute). It will take the score of the top player and ignore every player who has less than 20% of the score of this top player (those players will not get any points). I considered using ELO for this but given that I cannot validate player names it would make it too easy to cheat. I need methods that do not allow people to nuke other players scores.

Based on COUNTING players with 20% or more of the score of the #1 player and POSITION being the position of the player the internal value for CLQ2 will be:

TOTAL += (11 - POSITION / (COUNTING * 10)) ^ 3 * SQRT (COUNTING)

In general this means winning a game from 20 players is better than winning from 15 players.

The same presentation will be used as for CLQ1.

[BattleMode]

I implemented the actual processing and calculation of several statistics now (in addition to the global server ranking I already made):
CLQ1 player ranking per server
CLQ2 player ranking per server
Player playtime per server
Player points per server
Map playerseconds per server

The CLQ1 and CLQ2 rankings seem to match common sense for the servers I know (RPG ones).

Please note the site does not show any of this yet, it's all internal.

All this takes over 6000 MySQL tables which are generated automatically by my DynamicData module.

I tried 1 combined table for map popularity on all servers but that turned out to be a disaster to calculate (even though essentially what I am calculating already is the same data). Keeping the tables small seems to be essential. I am curious how all this is going to behave with larger data sets. It will probably need some major tuning later on.

Next step will be to do a huge amount of calculations and see what happens. Actually showing the stats on the site might be useful as well.

[BattleMode]

As I expected once more data gets into the database processing gets slower.

Processing 1 hour of data (1000+ servers all scanned once per minute) took up to 77 seconds. That might seem fast, but if you scale from 1000 to 100.000 servers this means there are not enough seconds left in an hour to process everything.

Made the processor scalable now. It can now process 24 hours of data in 1 cycle of 323 seconds. This means in the future I can scale the processing speed to match the database size.

This is not a new problem, the old CLQ suffered a lot from this problem, up to the point where it took almost a week to process 1 week of data.

[BattleMode]

Fixed a bug in the bulk processing logic, which somehow concluded it was 1901. I checked and that turned out to be incorrect. Seems it works fine now.

Next step will be to show internal data on the web site, Both to make it useful and to see if all data seems logical. The processing logic is quite complicated so there will probably be more mistakes.

[BattleMode]

Available immediately: AllGameServers.com beta prototype 1.

What is it?
It is a website scanning game servers and collecting statistics. It is roughly comparable to The CLQ which I made 10 years ago but also to sites like game-montior.com and serverspy.com.

Where is it?
http://allgameservers.com

Detailed BBS Titan RPG statistics: http://allgameservers.com/ags/server.ph ... 63.86:7777
Detailed Titan Freon statistics: http://allgameservers.com/ags/server.ph ... 2.199:7800

What is next?
It is a prototype. Some things are working most things are not. E.g. at the moment only UT2004 servers are scanned, I intend to scan every FPS game server on the planet. Also there is no graphical design yet, pure information only.

Your feedback would be appreciated.

[Suicidal]

arent u meant to be on HOIDAY?!?!? agreed it is raining here atm but.... lol