BLOG: Working on TheCLQ.com / AllGameServers.com

[BattleMode]

arent u meant to be on HOIDAY?!?!? agreed it is raining here atm but.... lol

Others read books, I hack PHP

[BattleMode]

Got home now. Changed the EEE for the real PC again, man did I miss fast Internet.

Anyways, improved the look and feel of the site now: http://allgameservers.com

[BattleMode]

Added support for game types.

Blacklisted 5 servers which are using bots pretending to be humans. All statistics are still kept but the player counter is always set to 0.

The "minutes" rank is a good way to locate fake players: http://allgameservers.com/ags/server.php?sort=minutes&addr=193.200.158.146:4444

Servers I blacklisted are:
[CTU] <<< Sniper Heaven >>>
++[EZE]Sneaky Ass-RiPpIn Sniper Server++
[YKI] ++++ < Oldskool Onslaught > ++++ - You Knows It!
[YKI] ++++ < King_Dingys > ++++ Onslaught Deviance
[YKI] ++++ < Camper And Sniper > ++++ - You Knows It!

[BattleMode]

Fixed a very tough bug. The "Red Team" player is now finally removed from the ranking system.

Some internal documentation:

Code: Select all

OVERVIEW OF DATA:

HyperStore (updated once per minute):
  globalvars: onceperminutecounter PROC_TransformOne STATS_ProcessOne
  servers: scanclass type missed lastscandata prelastscandata firstscan lastscan

DynamicData (updated once per hour):
  DD_RecreateTable ("server__player_seconds", array("server", "week"))
  DD_CreateTable ("server_{$address}_players_clq1", array ("player", "week"))
  DD_CreateTable ("server_{$address}_players_clq2", array ("player", "week"))
  DD_CreateTable ("server_{$address}_players", array ("player", "week"))
  DD_CreateTable ("server_{$address}_players_points", array ("player", "week"))
  DD_CreateTable ("server_{$address}_players_pph", array ("player", "week"))
  DD_CreateTable ("server_{$address}_maps", array ("map", "week"))
  DD_CreateTable ("server_maps", array ("map", "week")) not used yet

Disk:
  /data/scans/2008/07/28/..: raw scan data (multi files per minute): address=>scandata, scandata containing now
  /data/scans/2008/07/28/proc/..: processes scan data (multi files per hour, based on $hashsize): address=>now=>scandata

Objects:
  scandata: always: status name maxplayers players retries map playerlist  optional: now servertype gametype

OVERVIEW OF LOGIC:

  Cycle:
    QSTAT_InitScanAndStore ($specs, $threads=1): initiates the background scanning
      QSTAT_ScanNow ($scanspecs, $timeout=60): does the REAL scanning
        QSTAT_ScanAndStore ($specs, $timeout=60)
          PROC_RT ($address, $scandata): process scandata, update hs:servers
    PROC_Transform ($dir, $hashsize=1)
    STATS_ProcessMulti ($multi=1)   
      STATS_Process1 ($dir, $last)
        STATS_ProcessServer1 ($address, $scandata, $last)

  SERVERS_FindNewNow(): to be called manually does clearcache + findnew + manualscan 6 + manualscan 7
  SERVERS_ManualScan ($amount=25, $scanclass=6): only updates status and scanclass (to 1, 2 or 7)
  PROC_FIX ($scandata, $address): repairs scandata (e.g. removes bots and non players)

[BattleMode]

Implemented individual player statistics now. No search function yet so you have to click on the name of the player or modify the URL by hand.

Some examples:

Player HaiFeiz on the #1 BBS Titan RPG vCTF server:
http://allgameservers.com/ags/player.ph ... er=HaiFeiz

Player Deathborn.T32 on the Titan Internet Freon Server #1 server:
http://allgameservers.com/ags/player.ph ... thborn.T32

Player Cruz0r on the Titan Internet Freon Server #1 server:
http://allgameservers.com/ags/player.ph ... yer=Cruz0r

[BattleMode]

Implemented GeoIP matching now and added all flags of the world to the server. You can now see in which country all servers are:

http://allgameservers.com/ags/game.php

[BattleMode]

Wasted some time on explaining why I think people are wasting time on string theory : http://ut2004.titaninternet.co.uk/forum ... post281984.

Time to move forward. The biggest thing missing from AGS atm is support for multiple games. So that is my first priority now.

[BattleMode]

The second game is implemented. I decided to do Quake 3.

http://allgameservers.com/ags/game.php?game=quake3

Please note its all BETA so there are plenty of lose ends there.

I tried to make implementing games as generic as possible. So adding new games to the mix should be relatively easy to do.

[BattleMode]

Implemented search for games and for servers.

E.g. with http://allgameservers.com/ags/servers.php?query=%231+b you can find all BBS servers.

With http://allgameservers.com/ags/servers.php?query=titan you can find all Titan servers.

[BattleMode]

Did some experiments with different approaches for the "find player" engine.

This is probably the most difficult part of the entire site.

On one hand a single player could have be on 100.000+ servers over time (e.g. a player with a very generic name), at the same time a single server can have 100.000+ players over time (given that any name ever used becomes a new player in the database). So the data set is very huge.

The second problem is that I want this data to be reasonably up to date. Which is tough with 100.000+ concurrent players.

The third problem is that I want a smart search engine. When a name is very common I want the search engine to look at things like when the player played on a server last and how much he played there.

The fourth problem is that the search page will probably be the most popular of the entire site (the search player page of The CLQ used to be the most popular page). So It has to respond extremely fast.

The fifth problem is that I want the database to be complete. Did you just start playing 2 hours ago? It should be able to find you. Did you play 2 years ago and then never again? It should be able to find you. (This will of cause require 2 years of data collection, although I also plan to import my old CLQ database at some point.)

Problems like these are the reason I do projects like this .

[BattleMode]

After the usual fight with the MySQL "query optimizer" (I won) I got searching the way I want it.

You can now locate a player in the database in a fraction of a second by entering the name or part of the name in the search engine. It will tell you all names like what you are looking for including stats (minutes, last seen). After you select a player name you get to see all servers (again including minutes, last seen) and you can go to the detailed stats.

This completes the search engine part.

Example: http://allgameservers.com/ags/players.p ... battlemode

Next will be testing, debugging some more tuning and then operation big bang. Adding 100.000 servers and see where it starts to hurt.

[thomaster]

This one seems "unreal" for me: "Total pageviews: 200,818,345"

[BattleMode]

This one seems "unreal" for me: "Total pageviews: 200,818,345"

I use the CLQ counter for that, so it includes those counts as well.

[BattleMode]

I am trying to get subversion to work with CentOS 5.2. Unfortunately this brings me into my least favorite Linux area: package hell. I love Linux, I really do, but I really hate those "Missing Dependency" errors. Installing a vmware image of CentOS now so I can do some experimenting without risking nuking my server.

BTW the reason I am doing all this is because I want to research UT3 support.

[BattleMode]

Got everything the way I wanted it to work now. Turned out I used the wrong repository for rpmforge and also I learned about yum priorities now. Time to do some research on new game types and see if I can scan them.

I also have the challenge of UTF-8 (e.g. chinese) characters in servers and players. My current scanner simply ignores them but that is not going to work with counterstrike.

[BattleMode]

I succeeded in scanning a UT3 server (including players and score) now. I am NOT happy with the query port fiasco that is UT3 (had a lot of problems with that with my own game server) so I will have to figure something out there.

[BattleMode]

Succeeded in scanning a COD4 server. As expected this will be much easier then UT3.

[BattleMode]

Succeed in scanning a TF2 server.

[BattleMode]

Have TF2, COD4 and UT3 working in the scanner. This does not mean they are on the site yet. Before I go large scale I want the functional side to be more or less finished.

[BattleMode]

Got UTF8 support working on the scanner level now. This means special characters, Chinese etc. will be supported.