Warcraft II Forum

Warcraft II => Mods & Development => Topic started by: BusyOrc on February 19, 2016, 11:31:53 AM

Title: War2 Analytics
Post by: BusyOrc on February 19, 2016, 11:31:53 AM
Here's a project I've been working on :)

Server Analytics

http://analytics.war2.co/ (http://analytics.war2.co/)

User Analytics

http://analytics.war2.co/user.php (http://analytics.war2.co/user.php)


Game Search (still under construction, only works for search 1-2 players so far and has lots of bugs still)

http://analytics.war2.co/game_search.php (http://analytics.war2.co/game_search.php)


Screenshot generator

http://analytics.war2.co/game_report.php?id=119150 (http://analytics.war2.co/game_report.php?id=119150)


Its not fully baked in some areas but I'm getting bored with working on it for the last few months so figure I will let you guys have some fun with it.
Title: Re: War2 Analytics
Post by: Certified MENSA Genius Brain (smart) on February 19, 2016, 11:39:13 AM

1st   Garden of War.pud   31,530
2nd   Garden of war BNE.pud   8,507
3rd   Chop Farms [PROPER].pud   5,890
4th   Mini Chop Farms [[PROPER]].pud   3,923
5th   Mini Chop Farms [PROPER].pud   3,281
6th   Chop Farms (PROPER) CORRECT.pud   2,223

7th   Nowhere to run or hide.pud   1,720
8th   Mini Chop Farm [PROPER].pud   1,442
9th   Chop Farms {PROPER} CORRECT.pud   1,203


These dummies need to make up their minds and figure out which version it is they want to play
Title: Re: War2 Analytics
Post by: GaNzTheLegend on February 19, 2016, 11:44:31 AM
never knew people played that much NWTR still.
Title: Re: War2 Analytics
Post by: BusyOrc on February 19, 2016, 11:46:50 AM
NWTR must be all Unu lol
Title: Re: War2 Analytics
Post by: Certified MENSA Genius Brain (smart) on February 19, 2016, 12:07:31 PM
Yeah Unu hosts nothing but nwtr and if I'm online and it's the only game hosted I'll join it because nwtr is pretty good.
Title: Re: War2 Analytics
Post by: {Lance} on February 19, 2016, 12:35:30 PM
If you are not using caching for this,  I highly suggest it so that it does not overload iL's server resources.  His servers are not that good hardware wise.  Cache results in your own DB and have the search request from your page check your own DB first, then if nothing exists or it's been awhile since the last update for that particular query,  check iL's servers as sparingly as possible.  Especially for the "user search" feature which I am assuming abuses the ladder stats page or the game reports,  both of which are quite resource intensive when given large search parameters.  It seems like you already have caching for the "stats" portion,  but I'm not so sure about the search features because they take a conspicuously long time to complete,  I assume because it's reading from iL's servers rather than your own DB.

The UI you wrap around the items is pretty nice though.  I do like the look you have for the game reports with a clone of the war2 end screen and such.  Nice front end design work.
Title: Re: War2 Analytics
Post by: BusyOrc on February 19, 2016, 12:47:20 PM
Thanks Lance,

As for effecting server resources, I do not want to cause any problems! I am trying to have as small of a footprint as possible here.

All the queries are performed against my own DB.

I wrote a python script that runs every few minutes, downloads the list of game reports from reportb.war2.ru, checks to see which game reports its already done before, then it parses the new game reports that it hasnt seen.

Also another python script downloads the server status page every few minutes and saves info to my DB about antihack status, and country of users, and records session times.

I used to run a python script in the middle of the night that would log into the server and /finger every profile, parse and add to the DB, but it was getting me IP banned despite efforts to delay each command by 5+ seconds, so I stopped doing that, and it was probably a little too resource intensive anyways, although it didnt seem to cause any problems.


So now all I'm doing is opening reportb.war2.ru and server.war2.ru/status every few minutes.
Title: Re: War2 Analytics
Post by: {Lance} on February 19, 2016, 12:55:36 PM
Nice,  the slow down I noticed was really in the search queries.  They do seem to take a very long time and with a proper DB setup, they should take no time at all.  Are those queries (such as a username query) coming from your DB?  If so,  might want to make sure you are using indexing.  That could speed up the results A LOT because to me it seemed so slow that I thought it was querying il"s server on the fly and that wouldnt be good for the health of his servers ;) 
Title: War2 Analytics
Post by: EviL~Ryu on February 19, 2016, 12:56:55 PM
See what happens when you use your powers for good and not evil [emoji14]

Let me know if you wanna collaborate on some trivia reporting projects I have in mind.


Sent from my Motorola DynaTAC 8000X using Tapatalk
Title: Re: War2 Analytics
Post by: BusyOrc on February 19, 2016, 01:10:11 PM
Lance, I could definitely use some help with speeding this site up.

The server is 8gb RAM, 500GB sata hdd, 2 Xeon CPUs, there's no reason it should be so slow I would think.

backend: Linux/CentOS, Apache, MySQL, PHP, Python (for scraping all the data and adding to my DB)

frontend: Javascript, JQuery, and Google Charts API. The "Download SS" is actually a cool HTML5 Canvas script that turns the screenshot table into a PNG


I just barely started to get into programming and databases last spring, I didn't even know what a "framework" like Laravel was until after I started this site, so its just a mess of different PHP files strung together with "include"s.

Python scripts run on a 3min cronjob for collecting game reports and server status info and adding to my own DB.

I've made some changes in MySQL because it is so slow I've been trying to make it better, I increased some table size memory or something that helped alot for when I do subqueries that select huge tables "AS table_1" for example.

Also I changed to "innodb" just because I read its better but like I said I dont know much like how to cache, but Ill look into it



I need help writing the game search also.

Figuring out the query to write was very complex, joining multiple databases, but I have it figured out now. The problem is the number of possible combinations of inputs is insane.

Do I literally need to have an ifelse statement for every possible combination of inputs? If so I will literally have to write thousands of ifelse statements to expand to 8 players



Title: Re: War2 Analytics
Post by: BusyOrc on February 19, 2016, 01:20:21 PM
See what happens when you use your powers for good and not evil [emoji14]

Let me know if you wanna collaborate on some trivia reporting projects I have in mind.


Sent from my Motorola DynaTAC 8000X using Tapatalk

Yeah I think I'm going to change my name to GooD~Archer lol. But yeah I'm trying to contribute, hope some of the code I wrote can actually be useful for the community
Sure
Title: Re: War2 Analytics
Post by: {Lance} on February 19, 2016, 01:23:15 PM
Innodb actually isnt all that it's cracked up to be and will cause you more problems than it solves.  It generates 1 BIG ass file and rather than deleting from it,  it just keeps piling more info into it.  Example,  if you "Delete" a row in the table,  it does NOT actually delete anything.  All it does is add a "delete" query to the big ass file which then denotes that a row has been deleted and it just "appears" like it was deleted when it really wasnt.  It was just marked as deleted.  This is called journaling and is only useful if you have to "rollback" data at all.  You can use the journal to go backwards in time.  Similar to version control for source code files (like git).  It does have some speed enhancements that affects input,  but as far as actual querying,  no, there is not much difference.  Certainly not enough difference to ignore the fact that your innodb file is going to be TERABYTES in size in no time flat and the only way to "fix" that is to completely start over.  Do a mysqldump of the DB,  drop the DB, wipe the innodb file manually, and then re-import it.  There are ways to split that innofb file into multiple pieces, but it's a very advanced thing and very likely way above anything you could handle.  Stick with MyISAM for your tables.  Your diskIO and disk space will thank you for it.

The key for joining tables is to make sure that you are using "WHERE" ONLY on fields that have an "index".  If even 1 field does NOT have an index,  then your query can only run as fast as the slowest field and that can be a real bummer.  Thats usually the single quickest method to magically speed up any query.
Title: Re: War2 Analytics
Post by: BusyOrc on February 19, 2016, 01:41:01 PM
Interesting... I have alot of reading to do to get this site optimized.

The server stats charts are really cool. It was one of the hardest parts of programming.

Isn't anyone surprised to see how the server has actually been growing somewhat from Aug - December, increasing from 11k games per month to 15k games in December.

January's numbers are skewed because of a gap of game reports that were missing from RU for some reason (i think server migration?)


ALSO IL/LANCE:

The game report texts can be pretty shitty. They have problems with unicode characters so my game parser only works if the person uses an English war2 client, because otherwise the report gets all garbled and theres no way to parse it. I dont know if you can normalize them to be english some how, but it would help alot.

Also the times in the game report, it would be awesome if they could be normalized to UTC. It was very difficult to figure out how I could convert the times to UTC, and I ended up going off of one of the times that is always in Eastern timezone (because the server time is eastern I guess) and I was able to convert that to UTC and use it as the game time
Title: Re: War2 Analytics
Post by: Certified MENSA Genius Brain (smart) on February 19, 2016, 02:02:51 PM
See what happens when you use your powers for good and not evil [emoji14]

Let me know if you wanna collaborate on some trivia reporting projects I have in mind.


Sent from my Motorola DynaTAC 8000X using Tapatalk

Yeah I think I'm going to change my name to GooD~Archer lol. But yeah I'm trying to contribute, hope some of the code I wrote can actually be useful for the community
Sure
(https://upload.wikimedia.org/wikipedia/commons/3/35/PenitientGirl_Rotari.JPG)
Title: Re: War2 Analytics
Post by: EviL~Ryu on February 19, 2016, 02:03:50 PM
Lmao, this picture again.


Sent from my Motorola DynaTAC 8000X using Tapatalk
Title: Re: War2 Analytics
Post by: EviL~Ryu on February 19, 2016, 02:58:24 PM
Arch, are you a reports developer at your job?


Sent from my Motorola DynaTAC 8000X using Tapatalk
Title: Re: War2 Analytics
Post by: BusyOrc on February 19, 2016, 03:21:42 PM
No I just work on an ecommerce website for my family business now. Before that I was a Network/VoIP engineer though, CCNP for the Dept of Treasury

Just started playing with programming and databases last year
Title: Re: War2 Analytics
Post by: BusyOrc on February 19, 2016, 03:34:42 PM
Would appreciate if people would quit trying to hack the site, there is nothing interesting to be found just a database of war2 games

Its on its own virtual server, theres nothing on it besides war2.

Going to have to just shut it down if it doesnt stop getting flooded
Title: Re: War2 Analytics
Post by: iL on February 19, 2016, 03:37:40 PM
Archer, anything changed since you have been permbanned?
I don't see anything new.
You firstly come and tell that you are good and want to help and then make attacks against server to hang it up. Or to steal password or w/e.
Then you tell stories that it was not you or w/e. And then continue your attacks.

Now it's the next round of the same story. I can also assert that you made an attack to the server recently.
My relation to you have not been changed.

So, how that happened that this person have been unbanned? I don't want him to be a part of our community and to be involved into any projects here.
Nominal profit from him is much less than real harm.
Title: Re: War2 Analytics
Post by: BusyOrc on February 19, 2016, 03:42:45 PM
iL,
I never DDoS your server, not even once. Since what happened with war2.me I have become the scapegoat for everything that happens at RU.

Ever since war2.me I have said sorry for the ladder and account hacking.
I didnt make any attacks on the server, I've just been working on my analytics site and playing games but have not caused any harm to the server. While working on the analytics site I was careful not to strain RU server resources by using my own DB for game reports etc...
Title: War2 Analytics
Post by: EviL~Ryu on February 19, 2016, 03:44:49 PM


Now it's the next round of the same story. I can also assert that you made an attack to the server recently.


...

Lol, I did bitch slap his bot load to the void [emoji6].


Sent from my Motorola DynaTAC 8000X using Tapatalk
Title: Re: War2 Analytics
Post by: BusyOrc on February 19, 2016, 03:49:02 PM
http://analytics.war2.co/user.php?user=il (http://analytics.war2.co/user.php?user=il)
Title: Re: War2 Analytics
Post by: Player on February 19, 2016, 03:53:56 PM
Permanent ban.
Title: Re: War2 Analytics
Post by: BusyOrc on February 19, 2016, 03:54:48 PM
http://analytics.war2.co/user.php?user=player (http://analytics.war2.co/user.php?user=player)
Title: Re: War2 Analytics
Post by: BusyOrc on February 19, 2016, 03:55:32 PM
http://analytics.war2.co/game_report.php?id=114710&user=Player (http://analytics.war2.co/game_report.php?id=114710&user=Player)
Title: Re: War2 Analytics
Post by: BusyOrc on February 19, 2016, 03:57:57 PM
Give me a second chance. Let me give code to server admins for my screenshot generator, analytics etc :/
Title: War2 Analytics
Post by: EviL~Ryu on February 19, 2016, 04:00:41 PM
Give me a second chance. Let me give code to server admins for my screenshot generator, analytics etc :/

Don't bother, most are too stupid to even understand it lol [emoji23]

Dellam still allowed to roam this community, don't see why he can't have a similar chance. Make him do community service, under a watchful eye.


Sent from my Motorola DynaTAC 8000X using Tapatalk
Title: Re: War2 Analytics
Post by: iL on February 19, 2016, 04:13:47 PM
I didnt make any attacks on the server, I've just been working on my analytics site and playing games but have not caused any harm to the server.
Sure, and aslo the same words.

Just an example: you PM me your ideas that server should be closed for new registrations, i answer that i don't have such plans and right after that get tons of bots creating 100000 new accounts per hour. Coincidence? Of course.
Last example: tons of useless bot activity every minute from your home ip. I had to ban it on firewall. Then you appeared on the forum with questions about you connection. Coincidence?
I didn't check logs for possible attacks later, i have no so much free time for you and your attacks.

So, i want to develop something useful, not to waste my time to close holes in obviously holey software againt you and your coincidences.
So i prefer to not have you around. I hope to finish my antihack one day and ban you from the server forever.
Title: Re: War2 Analytics
Post by: BusyOrc on February 19, 2016, 04:23:54 PM
IL!

come one man, I see what youre saying but listen

the fact I said closing registration might be a good idea, then 10,000 bots attack with new accounts was actually coincidence.

I actually STILL wonder how the person programmed a bot to register an account, I do not know how it is done, because when connecting on the raw socket, you can only log in, you cant create an account.

I honestly DONT EVEN KNOW how to make a flood that creates accounts!!!!

The bot you banned from my IP was my analytics scraper. I said earlier in this post, I wrote a python script that logs into an account (using the RAW socket network login), then it did a /finger on all the accounts to extract profile information.

But my goal was to extract profiles, if I wanted to make it flood the server, I couldve done that, but I dont want to cause any problems.

I stopped running that python script and havent done it since then!


Title: Re: War2 Analytics
Post by: BusyOrc on February 19, 2016, 04:27:55 PM
BTW! I even posted more info about the attacker last time a flood happened, because my analytics software actually recorded who was online right before it happened

SO i narrowed down your suspect list dramatically and even posted that as a thread, but I still havent figured out who the flooder is or why they do it! I want to figure it out more than anyone because Iwant to clear my name over this shit

The flooder fucked up my database too, I had to go in there and delete all the flood account names and  shit.
Title: Re: War2 Analytics
Post by: BusyOrc on February 19, 2016, 04:33:11 PM
heres the link, I had already been working on analytics.war2.co for about a month when this flood happened and because my software tracks sessions of who is online at what time, I was able to get a list of everyone online right before the flood, and right after

http://forum.war2.ru/index.php/topic,1833.0.html (http://forum.war2.ru/index.php/topic,1833.0.html)

Title: Re: War2 Analytics
Post by: Delete mine too on February 19, 2016, 05:22:58 PM
Would appreciate if people would quit trying to hack the site, there is nothing interesting to be found just a database of war2 games

Its on its own virtual server, theres nothing on it besides war2.

Going to have to just shut it down if it doesnt stop getting flooded
Are you sure ? ;)
Title: Re: War2 Analytics
Post by: BusyOrc on February 19, 2016, 05:30:57 PM
Yes its completely just war2, the entire physical server is war2
Title: Re: War2 Analytics
Post by: Delete mine too on February 19, 2016, 06:02:11 PM
Yes its completely just war2, the entire physical server is war2
Man you heard the boss. You got to go, come back in a few years or be straight 100% anonymous. You are too smart and certified like why risk all that? Keep coding you're projects but I would not involve any of them with anything server.war2.ru related. I for one put you to an end when you "accused me / frame me". I should beat your face we could have met up. Just go away cya in a few years.
Title: War2 Analytics
Post by: EviL~Ryu on February 19, 2016, 06:16:25 PM

...I for one put you to an end when you "accused me / frame me". I should beat your face we could have met up.....

Lmao I remember that.


Sent from my Motorola DynaTAC 8000X using Tapatalk
Title: Re: War2 Analytics
Post by: shesycompany on February 19, 2016, 07:53:24 PM
dammm anyone need testo jelly shits like whiskey x 100 to dangerous for me!...gone back listen to me music..
this dude archer?
Title: Re: War2 Analytics
Post by: EviL~Ryu on February 19, 2016, 08:26:52 PM

..
this dude archer?




Sent from my Motorola DynaTAC 8000X using Tapatalk
Title: Re: War2 Analytics
Post by: Certified MENSA Genius Brain (smart) on February 19, 2016, 10:03:43 PM
So, how that happened that this person have been unbanned? I don't want him to be a part of our community and to be involved into any projects here.
Nominal profit from him is much less than real harm.
I don't think anyone ever unbanned him.  He simply came back, new IPs or new names, and nobody has kept up with banning him.

Personally, I think what he did that one time was abhorrent, and it was abhorrent in three parts: a) hack the server - abhorrent; b) steal accounts to abuse the ladder - abhorrent; c) post all the stolen information including passwords on his website - abhorrent.  Then he lies about it or whatever...

I don't know what he's done besides that, maybe there's more I didn't know.  I only know this analytics site is pretty cool, so when I see him show up and post this project and say he wants to be good now, my first reaction is not to immediately ban him again... even though, yes, what he did before surely was awful and possibly deserves forever ban.  But Lance is here now, Tupac is here, etc...  Anyone can re-ban him if they want and I won't stand in the way.  I just personally did not feel moved to re-ban him myself when I saw him post this thread with this project, that doesn't mean I disagree if any of the other admins want to ban him.
Title: Re: War2 Analytics
Post by: Nox on February 20, 2016, 01:31:45 AM
Best tool ever made by a war2 player and iL said permaban.

Nothing new here.

Please ban il for excessive super ultra master retardness.
Title: Re: War2 Analytics
Post by: Winchester on February 20, 2016, 06:18:35 AM
We're still waiting on your ban for super ultra master retardness Equinox.
Title: Re: War2 Analytics
Post by: tk[as] on February 20, 2016, 08:54:35 AM
braviet has been IN GAME nearly 30 days since this launched

http://analytics.war2.co/user.php?user=braviet (http://analytics.war2.co/user.php?user=braviet)


spb- is not far behind

our fearless leader blid has logged almost 3 hours.. so that's cool.
Title: Re: War2 Analytics
Post by: shesycompany on February 20, 2016, 08:58:45 AM
wow that site is cool wtg!!!! 8)
Title: Re: War2 Analytics
Post by: BusyOrc on February 20, 2016, 10:35:44 AM
braviet has been IN GAME nearly 30 days since this launched

[url]http://analytics.war2.co/user.php?user=braviet[/url] ([url]http://analytics.war2.co/user.php?user=braviet[/url])


spb- is not far behind

our fearless leader blid has logged almost 3 hours.. so that's cool.


Yea i ran a query a few days ago to find out the top 100 players sorted by minutes in game and Braviet was #1

I am planning on adding a "leaderboards" section of the site with top players All Time and Monthly sorted by minutes in game, and other categories maybe like gold, kills, etc

30 days in game is incredible tho, thats not even counting time between games or logged in talking to people

Thats 30 days since July 19 2015 so in the last 7months he spent 1month in game. Not sure if he even sleeps or eats or does he just play war2 lol
Title: Re: War2 Analytics
Post by: EviL~Ryu on February 20, 2016, 11:12:14 AM

We're still waiting on your ban for super ultra master retardness Equinox.

That made my morning [emoji23]


Sent from my Motorola DynaTAC 8000X using Tapatalk
Title: Re: War2 Analytics
Post by: Certified MENSA Genius Brain (smart) on February 20, 2016, 01:28:46 PM
braviet has been IN GAME nearly 30 days since this launched

[url]http://analytics.war2.co/user.php?user=braviet[/url] ([url]http://analytics.war2.co/user.php?user=braviet[/url])


spb- is not far behind

our fearless leader blid has logged almost 3 hours.. so that's cool.

It seems a bit flaky to me since it says brav's total time online is 17d but his time in game is 29d.

However looking at a few of my smurfs I've got 2d 15h in games.
Title: Re: War2 Analytics
Post by: {Lance} on February 20, 2016, 02:24:39 PM
That just means brav spends more time in games than actually in chat.  So its not unusual to me that game time could be higher than chat time.  For myself I bet that would be the case as well because I rarely spend more than 10 mins in chat waiting for a game.  If nothing shows up, I just hang it up and leave.  So if I spend 10 mins in chat but 20 in a game,  then yea, my time logged in would be 10 mins.  So the problem here isnt the stats, but rather the verbiage of what "Online time" actually is.  I believe it's in chat time, and not a total time on the server.
Title: Re: War2 Analytics
Post by: BusyOrc on February 20, 2016, 03:37:37 PM
That just means brav spends more time in games than actually in chat.  So its not unusual to me that game time could be higher than chat time.  For myself I bet that would be the case as well because I rarely spend more than 10 mins in chat waiting for a game.  If nothing shows up, I just hang it up and leave.  So if I spend 10 mins in chat but 20 in a game,  then yea, my time logged in would be 10 mins.  So the problem here isnt the stats, but rather the verbiage of what "Online time" actually is.  I believe it's in chat time, and not a total time on the server.


Online time has only been recording since around Dec 4 2015, but min in game has been recording since about 19 July 2015


Min in game is highly accurate because it adds the minutes in the game report up, but min online is less accurate because it works by checking server status page every 3 minutes to see whos online, so its accurate plus or minus 6 minutes per session

Also min in game has 1 important flaw, if a game lasts over 1 hr it doesnt record the time in game because I havent programmed it to parse "1 day 2 hours 5 mins" it can only handle cases like "5mins" right now

So min in game and online time should always be compared separately. Also, online time includes min in game time. So if someone comes online for 1hr, and is in game for 30min, then 1hr gets added to the online clock and 30min gets added to in game clock

Min in game is a very accurate way to measure growth/decline of the server tho, it is a the green line on the server status charts, one of my favorite parts of the site is the server analytics charts

Btw, another big flaw in the system, like i said earlier, its not able to parse game reports for anyone not using English war2 client, because the game reports cant be understood
Title: Re: War2 Analytics
Post by: Certified MENSA Genius Brain (smart) on February 20, 2016, 04:28:48 PM
Online time has only been recording since around Dec 4 2015, but min in game has been recording since about 19 July 2015
Oh so that brings a whole lot more of my smurfs into play lol.
Title: Re: War2 Analytics
Post by: {Lance} on February 20, 2016, 05:49:11 PM
Also min in game has 1 important flaw, if a game lasts over 1 hr it doesnt record the time in game because I havent programmed it to parse "1 day 2 hours 5 mins" it can only handle cases like "5mins" right now


http://php.net/manual/en/function.strtotime.php (http://php.net/manual/en/function.strtotime.php)  <-- this is what you are looking for to fix that bug.

As for the game report issues,  that is something I had fixed in war2.us but it has not yet made it into the war2.ru codebase.  It is something we are aware of though and there are several ways to fix it.  It may not even need to be fixed once version control has been instituted allowing only certain versions to connect and play.  The idea will be to allow other versions to connect, but not play.  Instead, feed them a message about the need to DL the correct version, etc.
Title: Re: War2 Analytics
Post by: tk[as] on November 30, 2016, 06:11:43 PM
I wonder why the analytics pages no longer work
Title: Re: War2 Analytics
Post by: Nedro on March 02, 2017, 09:07:56 AM
let's get this site back up. give the people what they want.