War2.ru Slogan
News: All hail our wise & benevolent admins!


Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
Welcome to the forums! We're glad to have you here! :) You can register your account here, then feel free to introduce yourself in the Server.War2.ru board & let us know who you are on the server.

Nerd's Corner 2851  109

Axe Thrower Posts: 303 Karma: +20/-0 ****

Lambchops

  • Axe Thrower
  • ****
  • *
  • Posts: 303
    • View Profile
Nerd's Corner
« on: October 31, 2016, 08:27:45 PM »
I found a bit of writing I did about NAT a while back. There's been some talk about this recently, so I modded it a little bit to make it more relevant to WC2, as I thought the more nerdy folks out there might find it helpful.

So NAT. What's it all about?

Lamby's super-abridged, mostly fictional, oversimplified, heavily biased synopsis: Network Address Translation is a messy, shitty, stop-gap measure that got dreamed up by psychotic, sadistic and under-medicated network engineer as a way to get more flexibility out of the already overcrowded IPV4 address space prior to the implementation of IPV6.

Most people know by now that IPV4 – that’s the 4 numbers with the dots in-between that we all recognise as an IP address - was just too small. There just wasn't enough numbers. Its a 32 bit value. 2^32 = just under 4.3 billion possible addresses. Seemed like a big number 50 years ago, but even 20 years ago people were realising there weren't enough to go around.

IPV6 has now been implemented, which has lots of address space - like crazy lots =). We still see IPV4 addresses on the surface, but the backbone of the internet is now all IPV6, and all those things like credit-card gobbling parking meters that all need their own address just have IPV6 addresses. IPV4 is still around to interface with all the existing technology.

So IPV4 addresses were just not cutting it – it's 4 measly bytes, and that limit was (and for the most part still is) hard coded into every bit of network infrastructure in the world. But! Thought the madman, most of the traffic is TCP and UDP, and they have this port number. It's another 2 bytes! What can we do with that? NOTHING you asshole, it already has an important function, leave it alone. But no…. We get Network Address Translation, and we'll probably be stuck with it until the end of days. Post nuclear holocaust the only things to survive will be cockroaches and NAT.

So yes, a port number is 16 bits (0 thru 65535) and only a fraction of them are actually used. Officially ports 0-1023 are 'system ports' and stuff above that can be registered by people who want to put their official stamp on them….. or you can just pick one that's not getting used too much ;) Wikipedia lists around 1300, including 3 entries for port 6112:

Code: [Select]
6112 TCP UDP dtspcd, execute commands and launch applications remotely,  Official
6112 TCP     Blizzard's Battle.net gaming service and some games, ArenaNet gaming service, Relic gaming service,  Unofficial
6112 TCP     Club Penguin Disney online game for kids,  Unofficial


Anyone spot the mistake? … battle.net port 6112 listed as TCP only. (nvm, I already fixed it – the guy maintaining the page sent me a rather curt little message about 'original research', but he left it because he knew I was right;)

So how many possible port numbers do we need? Well I think we can safely say its more than 256 (1 byte) and less than (256x256) 2 bytes, so allowing 2 bytes is about right. But how many port numbers are actually used every day? Depends who you ask… let's see well 53 is DNS, that’s a cert, and 80:HTTP, more to the point these days 443:HTTPS, 20-21:FTP maybe not daily, 37:TIME gets used behind the scenes, 25:SMTP (email) used lots, but not so much from home any more, these days its all secure web pages (443) and poor old 110:POP3 is well on the way out. I could go on,  hey I love that Doom still officially owns port 666, (a system port 8P) but in reality most of the traffic in an average network on an average day probably uses maybe 50 ports? Who knows, maybe 100? Ok so lets say, for the sake of argument, its 100…. but we have 65535 possible port numbers, so lets go ahead and divide that by 100 and now we can support a network of 655 individual nodes off a single IPV4 address.

How do we do this? Well we just have to introduce an agent at the crossover point between our NAT network and the real world.

The following description is, of course, complete and total rubbish for so many reasons… there are different types of NAT and different ways to implement things within them, and I'm no expert. This is just here to hopefully help people who know nothing at all about the subject to get their head around the general concept.

Think of it like a (crazy) guy with a clipboard who writes really fast and looks at every packet that goes past. First he sees an outbound packet from puter 20 on port 3000, he says “call it 10” and writes down 10 = 20:3000, then he changes the source port number to 10 and sends the packet on its way. Next there's an outbound packet from puter 44 on port 8008, so he writes down '11= 44:8008', changes the source port number to 11 and sends it on its way. Next there's a packet from puter 150 on port 3000…. '12 = 150:3000' …. off you go… he does this a lot.

The packets, with their re-assigned port numbers go to wherever they were going and (as is the nature of things) they get responses from remote computers somewhere in the world. Those computers do whatever they do with the payload – the chunk of data that this addressing system is carrying – then they send back their response. There's to and from IP addresses and a source port and a destination port. The remote computers received the packet on the destination port, and you guessed it, for the return journey its all switched around: the to address becomes the from address and vice-versa, and the source port becomes the destination port etc., and the return packets get sent back to the addresses that the initial packets were received from.

The thing is here, that as far as the remote computers are concerned, these packets all came from the same computer, because they were all using the same external IP address as the from address, so they all arrive back at the same place and are greeted by the loony with the clipboard who looks at the packets, then checks his clipboard and says “hmm, port 10… that's puter 20, port 3000”, so he changes the destination port to 3000 and the to address to puter 10's local IP address and sends it on its way.. etc. When he gets the port 12 return he sends that to puter 150, port 3000.

So 2 different computers on the local network have simultaneously requested data over port 3000 from 2 separate remote servers using the same external IP address and both receive their replies, on the same port, and are none the wiser that for most of the journey their packets actually had completely different addresses and ports on them. Likewise the remote server has no idea that the originating computer is not receiving the packets on port 10 or 12, but on port 3000. But because of the way NAT works it never matters because the nutter in the middle keeps untangling his own knots on the way back.

Why do it this way? When you think about the amount of data that is sent around just for one Game of Thrones episode, its colossal compared with this tiny little 2-byte port number… so why mess with it? There is, of course, a very good reason. It's because every piece of data in every layer of every packet is ordered according to some protocol or other, and somewhere along the line, some computer is going to process that data. So if you add data, or introduce some new protocol, it means the computer at the other end has to understand that change and correctly process it, or even just know to ignore it (Unless you're meta-tagging an mp3, which is just deliciously naughty – but that's another story;). NAT works by sneakily messing with the existing data so that the remote computers just process it in according to their existing protocols without having to be aware that any translation is taking place. In this way it can be introduced in a single location, to suit local purposes without having to upgrade the entire internet (which wasn't in the budget that quarter).

Well. That sound's really clever… so what's wrong with that? Well…. nothing …. and everything. Hmm, how to explain…

You know when you have to move house? You pack everything into boxes and it gets loaded onto a moving van, then unloaded at the other end and you try to reconstruct your home. To make this a bit easier, you no doubt write “kitchen” or “garage” on the boxes, maybe even “kitchen – glassware” and “kitchen – plastics”. This way you have an idea about where everything goes when you get to your new home. Also the guys from the moving company might even notice that a box says “glassware” and decide not drop it 8 feet off the back of the van onto the road with the rest of them (maybe?). Good idea.

  - - >  So your address is where your new house is, and the extra note about what the box is being used to move, and exactly where in the new house it needs to go when it arrives, can be thought of as the port number. < - -

So now what if 500 people are all moving interstate on the same day, and they are all using the same moving company? Bit of a worry.. ok then, let's up the ante a bit more... What if there was a little snafu and instead of printing address labels for 500 different addresses somebody printed 10,000 labels all with the same address on it and for whatever reason they had to use those labels, and the only place to write any notes on the boxes is in the one little space provided on the address label and that note space just is not big enough to write the correct address? How can this work? Well, says the madman with the clipboard. It seems everyone is writing “kitchen” on these things, hey I've got an idea! There's lots more words than just that…

How about if I write “unicycle” on this, that means “bathroom at the Jones house”, but if I label it “butt-plugs”, that means “kitchen at the Smith house”, and if I put down “proverb” that means garage at the Bloggs house. Then I can just keep a list here on my clipboard, and when when we get to the next state we can just look at the list and send everything where it needs to go.

As insanity would have it, it turns out this guy really is obsessive-compulsive enough to do this and he correctly decodes the labels on 10,000 boxes and sends them to the right house. He even crosses out “butt-plugs” and re-writes “kitchen” so Mrs. Smith knows where to take her (broken) glassware. Pretty neat trick right? Perhaps, at least until you have to cross some border where there's a security check, and the customs agent decides to check out your unicycle and discovers instead, bottles of shampoo and other liquid product, which we all know is what terrorists use to blow up planes so… oh dear… Mr. Customs agent rather ominously snaps on a rubber glove and asks you to step into a small cubicle…. and Mrs. Jones never does get her hand sanitizer back.

There are some important protocols that include some form of auditing of their own packets, the most obvious being IPsec or “Internet Protocol Security”. IPsec encrypts packets at the internet layer, which of course includes the port data which exists at the transport layer, so if NAT changes what it assumes is the source port number the packet has been corrupted and can't be decrypted.

The real problem is that the crazy-guy is messing with the language that everybody speaks. Nowhere in the universe, apart from in his twisted head, does “unicycle” mean “shampoo”. Its all well and good for stuff that he handles first, but when Mrs. Jones' mother tries to send them a pot-plant as a house-warming gift, the crazy guy re-labels it 'special fried rice' and sends it to Alaska.

And this is the problem with NAT for WC2. Battle.net is NOT a gaming server for WC2 games, its more like Craiglist. It just advertises the games. Initially the host's computer is the server, then it's mostly just peer-to-peer. The 'server' (BNet, PvPGN etc.) just relays messages and keeps track of the stats when the game ends. But, the other clients aren't replying to packets that the fruit-bat has already messed with, they are using the actual correct information that was posted in the game list, or supplied by the game host. So when a packet arrives at the NAT agent labeled “WC2 Game”, our friend looks at his clipboard and says, “oh, you're a medium sized flock of migrating geese” and points it towards the equator.



NAT is an IPV4 thing only. There is no IPV6 Network address translation, its just not needed, and here's why:


The IPV6 address space contains approximately:       
340,282,366,920,938,463,463,374,607,430,000,000,000 addresses.

The zeros on the end round it off to the nearest 10 billion addresses.
The entire IPV4 address space is under 4.3 billion addresses.

Assigning a trillion trillion IPV6 addresses to every person on the planet would account for less than 0.01% of the address space. Pretty sure nobody wanted to have to upgrade the internet again.

An IPV6 address uses the same amount of bandwidth as writing BOOBIES!
…… Damn I love numbers (and boobies :P)



and at this point I'd just like to send a quick shout-out to all the network designers out there:

 WE'VE HAD IPV6 FOR YEARS NOW, SO CAN YOU CHEAP ASSHOLES PLEASE STOP USING NETWORK BLOODY ADDRESS TRANSLATION?

Its not clever. Its shite. Stupid shite.
Berserker Posts: 577 Karma: +25/-24 *****

Yamon

  • Berserker
  • *****
  • *
  • Posts: 577
    • View Profile
Re: Nerd's Corner
« Reply #1 on: November 02, 2016, 01:02:36 PM »
Axe Thrower Posts: 303 Karma: +20/-0 ****

Lambchops

  • Axe Thrower
  • ****
  • *
  • Posts: 303
    • View Profile
Re: Nerd's Corner
« Reply #2 on: November 12, 2016, 11:34:07 AM »
Ogre Mage Posts: 2209 Karma: +45/-2 retired, be in music section ********

easycompany

  • Ogre Mage
  • ********
  • Posts: 2209
  • retired, be in music section
    • View Profile
Re: Nerd's Corner
« Reply #3 on: November 12, 2016, 11:58:23 AM »
lol the last one ;D
Ogre Mage Posts: 2209 Karma: +45/-2 retired, be in music section ********

easycompany

  • Ogre Mage
  • ********
  • Posts: 2209
  • retired, be in music section
    • View Profile
Re: Nerd's Corner
« Reply #4 on: November 16, 2016, 06:32:22 PM »
lamb your needed in the support section<
Ogre Mage Posts: 2209 Karma: +45/-2 retired, be in music section ********

easycompany

  • Ogre Mage
  • ********
  • Posts: 2209
  • retired, be in music section
    • View Profile
Re: Nerd's Corner
« Reply #5 on: December 04, 2016, 10:24:21 AM »


using free netzero over the weekend,had to have some tunes!

keepvid.com copy "video" shortcut,paste in this site,http://www.online-convert.com

put your bitrate etc,walla!!! hi speed 56k :P sounded pretty dam good on my 200mhz lappy, "put all my external modems up some where and lost :("

4 min oukast ms jackson at <1/4th 56k.

yeah it plays war2.ru its a beast! also suprised internet5.5 went to these sites :o

 :o video isnt that bad with mp4 sadly old procs cant handle it to well.

so theres a nerd bulletin post (bored out the ass :P)
Axe Thrower Posts: 346 Karma: +23/-2 LAWL! LICK MY RIPESTICKZ0R H0M0 :D ****

RipE[Eur0]

  • Axe Thrower
  • ****
  • *
  • Posts: 346
  • LAWL! LICK MY RIPESTICKZ0R H0M0 :D
    • View Profile
    • fuk the leftist scum.
Ogre Mage Posts: 2209 Karma: +45/-2 retired, be in music section ********

easycompany

  • Ogre Mage
  • ********
  • Posts: 2209
  • retired, be in music section
    • View Profile
Re: Nerd's Corner
« Reply #7 on: December 07, 2016, 06:35:50 PM »
good enuff video 360? i cant remember but def better than 240 shit on youtube is at 10k just 2x 56k,dam good enuff music is at 1.6 k lool raging  this wasnt around in 2002.works for 2g also.
Axe Thrower Posts: 303 Karma: +20/-0 ****

Lambchops

  • Axe Thrower
  • ****
  • *
  • Posts: 303
    • View Profile
Re: Nerd's Corner
« Reply #8 on: December 20, 2016, 01:56:50 AM »
Going to cut&paste some of the stuff I have posted in random threads to here, cos I have trouble finding it all....
Axe Thrower Posts: 303 Karma: +20/-0 ****

Lambchops

  • Axe Thrower
  • ****
  • *
  • Posts: 303
    • View Profile
Re: Nerd's Corner
« Reply #9 on: December 20, 2016, 01:59:00 AM »
there is one WC2 mod that is head and shoulders above everything else absolutely screaming out to be implimented.

That mod is breaking the 128x128 map size limit. Historically I know this has been looked at before, but I dont see why it couldn't be done.

It also leads to another 2 mods that I would like to see, and they are :
 - Increasing the game display from the rather miserly 14x14 window we currently have
 - Breaking the maximum per game unit cap - cannot create unit - this one could possibly be quite easy.

Internally locations are mostly stored as 2 WORD size variables and often passed or referenced as a single DWORD argument so most of the game code already supports a map size of up to 65536x65536. There are, however parts of the code that impliment WORD sized absolute pixel co-ordinates, which restricts things to 2048x2048, but even this would be pretty cool.

Implimentation would not be trivial, but it could be almost all done just by modifying the existing code without actually having to add any new code in.

We would have to do this (at least!)

- Mod existing PUD editors to allow the creation of large maps.

- redefine the valid .PUD check so the clients will accept larger maps

- Mod things like any internal co-ordinate checks that may declare co-ords >127 to be invalid, it would be nice if these would all be based on the map size in the PUD file, but as there are currnetly only 4 legal map sizes there will probably be sections that are hard coded to make calculations based on one of these 4 sizes. All this code would ned to be chased down and modded or re-written.

- Increase any buffer allocations to allow for bigger maps, possibly parts of the executable's .data and .data? sections will also have to be redefined to allow more space.

- Check and mod the display code to allow for larger maps. Scaling for the mini-map is an obvious one, as is the buffer that keeps the background updated (this one is why you can see where trees are chopped even if you don't currently have vision there).

The main view code probably wouldn't have to be changed at all apart from any checks that just flat-out reject large co-ords.

.... Possibly by this time we might know enough about the display code to increase the 14x14 window size by having the internal routines draw the units to a larger buffer, then resizing that image and pasting it onto the existing screen buffer. There are also a bag of issues related to translating mouse input from this resized window.

--> This also leads to running the game at higher screen resolutions, which would be nice, but that would be another BIG mod. Where do you draw the line? It would be nice for it to actually be done before we are all too senile to enjoy it ;)

Breaking the unit cap should just be a matter of increasing the buffer allocation and finding any bits of code that impliment:

if n_units>MAX_UNITS {say "BLEH!"}
Axe Thrower Posts: 303 Karma: +20/-0 ****

Lambchops

  • Axe Thrower
  • ****
  • *
  • Posts: 303
    • View Profile
Re: Nerd's Corner
« Reply #10 on: December 20, 2016, 02:00:25 AM »
The most I've gotten to is war2 sitting at black screen when I used my maps bigger then 128x128. I think I nulled out the minimap and it no longer crashed. But I was stuck there and clueless. What do you think about that?


I think you gave up to soon ;)

Nah I just ran out of time because of career changes. Plus tons of other projects kept me bottlenecked. All honestly I was tired of being the one trying.


Fair call mate. This I understand.

Having slept on it, I must also say thanks for bit of actual real info. Getting any actual info about how the client works that comes from somebody else's time and effort is always good. Nobody understands better than me that your 'blank screen' story was no doubt the result of many, many hours of hard work.

So here's some thoughts about where to go next:

There are really only 2 questions for this type of thing:

1) what is it doing?
2) how do I make it do what I want?

They inevitably lead back into each other. Working out a bit of 'What is it doing?'(1) leads you to do/mod something (2) which almost never has the effect you thought it would so you go back to (1) and work out what the actual effect was, then try something else(2) etc...

(1) can be a bitch. (2) is the fun part - coming up with elaborate schemes muhahahahahah... ;)

! No longer available



So, as tends to be the case, you are stuck at (1), however a blank screen should not be a deterrant. The game screen is the least likely place to get any real info about what the internals are doing when you are messing around with them. It generally returns only a boolean 1-working, 0-fucked up.

So what is it doing? If it is not raising any exceptions with the OS then at a guess I would think that either (a) it's stuck in a loop somewhere, or (b) 1 or more buffers have overflowed and corrupted .data or .data? section values related to the video and sound.... or possibly (a) as a result of (b)

You need to have a look at the running process. Belt this stuff around a bit ;)

    MASM32
    ReadProcessMemory
    OllyDbg
    IDA Pro

Your're welcome. :)
Axe Thrower Posts: 303 Karma: +20/-0 ****

Lambchops

  • Axe Thrower
  • ****
  • *
  • Posts: 303
    • View Profile
Re: Nerd's Corner
« Reply #11 on: December 20, 2016, 02:01:36 AM »
PE files contain sections.


Common sections are:

.text (AKA .code) which contains the compiled ML

.data   which contains variables and strings etc., that the program can use and modify as it runs

.data? is the same as .data, except that it initially contains no values (uninitialised data). The .data? section is a virtual section that has 0 size in an .exe file but is created when the image is mapped to a process' memory by the system loader. It's a way of pre-defining memory to be allocated, with the bonuse of that memory being inside the module's image.

.rsrc "resource" usually contains things like bitmaps, constant strings (that are not modified during execution), dialog templates etc...



Not all files have all sections, and indeed the names, despite being almost always as listed above, are just a convention. The actual function of a given section is defined by flags set in the Characteristics member of the IMAGE_SECTION_HEADER structure.

Everything you never wanted to know about windows .exe files is here. (You want the file called "pecoff.docx")

'COFF' stands for Common Object File Format. Executable files started out as a sub-set of COFF files, they share many internal structures. Things like .obj "object" files are COFF files - .obj files are produced by compilers which are then fed to linkers which create the finished .exe file. 


.... anyway so what I mean by this:
....buffers have overflowed and corrupted .data or .data? section values...

      is that when the image is mapped to memory certain areas will have been allowed for certain data, that has an expected maximum size, so there could, for instance be something like this:

Code: [Select]
.data?
mapArray   db 4000h dup (?) ;-- some map array or other
hGLthing   dd ?             ;-- handle to some important GL structure
otherStuff dd ?,?,?,?       ;-- all sorts of other important data

as the max map size is 128x128 = 16384 (0x4000) and the map size is checked when it is loaded, whatever routine that puts data from the map or whatever into that array doesn't bother doing any checking, so when we gymp the 128x128 map size checks, the program just goes right ahead and writes data that is supposed to all fit in that array straight over the top of all the variables that are stored after it.

This is where image data sections can be tricky, because all the references to them are hard coded into the ML instructions. From one angle this makes them very easy to debug because you can always immediately see what is being accessed by a given instruction.

When memory is dynamically allocated, its trickier to debug, but much easier to resize. the same thing could be achieved with:

Code: [Select]
.data?
ptrMapArray   dd ?           ;-- *POINTER* to some map array or other
hGLthing      dd ?           ;-- handle to some important GL structure
otherStuff    dd ?,?,?,?     ;-- all sorts of other important data

then somewhere in the initialization code there is:
Code: [Select]
.code
push 4000h
call GlobalAlloc
mov ptrMapArray,eax     ;  <-- functions return values in the eax register
so this just asks the OS to allocate space for the array then just stores a pointer to the allocated memory in the data section. Later when the program wants to access this array you might see something like:
Code: [Select]
mov edi,ptrMapArray
; ...... then further on
mov [edi+ebx],dl ; ... or whatever
in this case edi woud contain the pointer to the base address of the array, ebx would contain an offset into the array, and the low byte of edx would contain the value it is writing there.

At least that is what the source could look like. Once it is compiled then decompiled, you dont have the benefit of the meaningful variable names that the guy coding it in the first place used, so your:
Code: [Select]
mov edi,ptrMapArray  has become maybe:
Code: [Select]
mov edi,[4A789Ch]... where 0x4A789C is the actual address of that dword in the .data? section (this is what the linker does)

things like:
Code: [Select]
call GlobalAllocwill still be there because GlobalAlloc is exported by name from kernel32.dll and the disassembler will extract the proc name from the IMAGE_EXPORT_DIRECTORY structure in kernel32.

... so if our array is dynamically allocated, and we want to change the max map size from 128x128 to 256x256 all we need to do to resize this particular buffer is change:

Code: [Select]
push 4000h    ; (128x128)
call GlobalAlloc
to:
Code: [Select]
push 10000h    ; (256x256)
call GlobalAlloc

BUT....
....  if this buffer (and not a pointer to it) was itself allocated in the .data? section, to resize it we have to create more space in that section of the PE image.

This in itself is trivial - you just adjust the VirtualSize member of the IMAGE_SECTION_HEADER structure for the .data? section and the system loader will create a larger image - however this doesn't change the ML in the .text section, and as I mentioned the addresses of the variables, and importantly for this case, those after the array (i.e. 'hGLthing') have been hard-coded into the ML instructions by the linker, so the code is still going to read and write the value of that handle to the same location, and the code writing to the array is still going to corrupt it. In this case all we have done is allocate extra memory at the end of the .data? section that isn't used anyway.

To resize this kind of array you would have to increase the size of the .data? section by N bytes, then find every reference to anything in the .data? section that is above the base address of that array (there could be hundreds of addressed data locations with multiple reference to each) and increase all of these addresses, which are hard-coded into the instructions, by N bytes, and to make matters worse there could be pointer tables to some of these locations stored literally anywhere that would all need to be found and adjusted.

With the original source code, you just change your:
Code: [Select]
mapArray   db 4000h dup (?)     to:
Code: [Select]
mapArray   db 10000h dup (?)     then recompile and its done (approx. 5 seconds total)

WHen reverse engineering a complex program this is can be almost impossible, and WC2 was pushing the limits of CPU and RAM capacities back in 1996, so to get the maximum possible performance they used lots of little tricks involving pointer tables to reference things and also to control program flow. It really is a cool chunk of code (Lamby gets all dreamy-eyed and cracks a nerd-boner) but it can be hard to get your head around.

....anyway if faced with this situation I would opt for injecting a small amount of code somewhere that dynamically allocated a larger buffer, and just leave the buffer in the .data? section as a blank space. Then every instruction that accessed the original buffer would have to be located and modded to point to the dynamically allocated one, which could be a bit painful, but nowhere near as painful as doing this for every one of the possibly hundreds of memory locations in the data section that are above the address of the original buffer.... and a nice trick for this type of thing is using a debugger to put a memory-watch on the original buffer then pause execution and report the value of the EIP register whenever it is accessed ;)

And for specifically Blizz related allocation, much of the WC2 stuff is, as you probably know, stored using Mike O'Brians Mo'PaQ format (.mpq files on disk). These are loaded into dynamically allocate memory by various functions exported by ordinal from storm.dll, so if any of these need to be resized it would be done by adjusting the size of the relevent resource in the .mpq archive.


Some notes on the code bits here:
It's all ASM code because that's how I think when I'm doing this stuff, and it really is the only way to think when you're reverse-engineering anything.

DATA DECLARATIONS

Code: [Select]
.data?
hGLthing   dd ?
this just allocates 4 bytes in the .data? section and names that location "hGLthing "
'dd' is 'declare double' it allocates a DWORD
'dw' allocates a WORD (2 bytes)
'db' allocates a BYTE
(also 'dq' allocates a QWORD, but you won't find any of those in WC2)

with this:
Code: [Select]
mapArray   db 4000h dup (?)'dup' means duplicate. Its declaring bytes (db) and its declaring 0x4000 of them.
In all of these the '?' is the data value the location is initialised with.
Because these are in the .data? section which holds only uninitialised data they can't have any defined initial value do they must all be '?'.

In the .data section you can have initialised data so you might see:
Code: [Select]
.data
dwThisVar   dd 0FE339D2h
myByte   db 20h
thisString db "Hello World",0   ;  <-- an initialised zero terminated string
myWords dw 3,3,3,3     ; <-- 4 words all set to 3 (8 bytes all up)
ymWords dw 4 dup (3)   ; <-- exactly the same ( 4 words all set to 3 )

when compiled this .data would look like:
Code: [Select]
D2:39:E3:0F:
20:
48:65:6C:6C:6F:20:57:6F:72:6C:64:00
03:00:03:00:03:00:03:00:
03:00:03:00:03:00:03:00:
(off the to of my head - dont crucify me if I messed up, but you get the idea..)

WC2, was of course written in C, but it all compiles to ML anyway. Apart from ASM (which is for the most part 1:1 ML instructions) C is the lowest level language around and therefore closest related to ASM and the compiled .exe, also most C compilers support in-line ASM code, so performance critical sections of code can be written in ASM, within the C source and then all compiled at once.

Data containers declared in C end up roughly the same as they would have if declare in ASM, i.e. a C pointer variable '*ptrArray' or whatever, if declared globally would end up as a 'ptrArray dd ?' in the data section, whereas if it was declared locally in a procedure it would be created as a stack variable. Which is temporary use of the 'stack' to store values. The stack is a special area of memory that is heavily used by the CPU to shunt numbers around... the 'push' and 'pop' instructions move values onto and off the stack. It acts like a big pile of numbers (FILO) but the current stack location is held in the ESP register ... the stack pointer.

.. and OMG don't look now, but here here comes a bit of compilers, procs and stack frames
  :o

This leads to the actual compiler that was used to compile the source, as they have diferent methods for implimentng procedure calls and allocating local storage. You can allocate 4 bytes of stack space simply with:
Code: [Select]
sub esp,4but it is vital to ensure that the stack is 'balanced' - returned to its correct state before executing the next 'ret' - return instruction. In this case you could
Code: [Select]
add esp,4or just
Code: [Select]
pop eaxwhich would get the value off the stack and put it in the eax register. (push and pop automatically adjust the esp register). The other way to do this is to save the value of esp when you enter a procedure.

The Microsoft compiler does this by repurposing ebp - the base pointer to save the value in esp when entering a procedure, so it might produce something like:
Code: [Select]
push ebp   ; save the value in esp
mov ebp,esp ; use ebp to store the current stack pointer
sub esp,8  ; in this case 8 bytes of local storage has been allocated for the proc (most likely 2 DWORDS)
At this point the arguments that were passed to the procedure are also sitting on the stack behind the return address that was left there by the call instruction that called the proc and the original value of ebp, which we have just stored there. ebp holds the value of the stack pointer after this proc entry, so if it is not changed the passed arguments can be accessed directly using ebp provided we allow for the 8 bytes that these 2 DWORD values use, so the first argument can be accessed at [ebp+8] and assuming its a DWORD (which they almost always are) the second argument is at [ebp+0Ch] etc. At the same time the locals that were allocated with the
Code: [Select]
sub esp,8can be accessed at [ebp-4] and [ebp-8]. This area of the stack, from the start of the passed arguments to the end of any local allocations, for the life of the proc is often referred to as the stack frame for that procedure.

Meanwhile you can merrily push and pop your heart out or call other procs etc. (sending the stack pointer all over the shop) because the position of the stack pointer at the proc entry is stored in ebp, and the entry value of ebp is itself stored on the stack. So when it comes time to exit the proc they just do the following:
Code: [Select]
mov esp,ebp ; <-- restore the stack pointer from the base pointer
pop ebp ; <-- pop the base pointer value off the stack
ret         ; <-- pop the return address off the stack and jmp to that location
In actual fact, Mr Gates made such extensive use of this method that his mates at Intel eventually designed their CPUs with special instructions to acommodate it. The leave instruction does the combination of mov esp,ebp and pop ebp in one hit.

Anybody who has not nodded off by now may have noticed the other issue. Despite having balanced the stack from procedure entry to exit (thus taking care of any locals allocated), whatever arguments were passed to the proc when it was call are still sitting on the stack, as these were pushed there by the caller prior to calling the proc, so esp still has to be increased by an amount equal to the total size of the passed args to put things to rights, but this has to happen after the return value is removed from the stack. Alternate versions of the ret instruction (often denoted as retn) that take a numeric value are used for this, so at the end of a proc that is passed 2 DWORD args, you might see:
Code: [Select]
mov esp,ebp ; <-- restore the stack pointer from the base pointer
pop ebp ; <-- pop the base pointer value off the stack
retn  8     ; <-- pop the return address off the stack, add 8 to esp, then jmp to that address
OR...
Code: [Select]
leave <-- do the esp/ebp square dance
retn  8     ; <-- pop the return address off the stack, add 8 to esp, then jmp to that address

...sadly, however. Blizzard didn't use MASM to compile WC2. By the looks of it, it's Borland code,....

<EDIT>
...Actually, on further investigation, it looks like they used M$ VisualC 2.0  Presumably there's compiler options for proc layouts/calling conventions etc. Interesting, but changes nothing anyway - it is what it is ;)
</EDIT>


....not that there's anything wrong with that as far as executing it goes, but when decompiled, the stack frame addressing isn't anywhere near as clean to read.

For example, where the MASM code references arg1 by [ebp+8], and anywhere in that proc that you see [ebp+8] you know that it is addressing arg1, Borland doesn't bother with using ebp and just adresses arg1 as [esp+4]

..... until the first time a push instruction is used then 4 is added to esp, so now arg1 is at [esp+8] ... push again its at [esp+0Ch]... then pop something and its back to [esp+8] etc.

So the location of the first argument to a proc is esp + (number of pushes - number of pops +1) x 4 from the proc's entry point.

    EDIT: The exception is, of course, values that are pushed onto the stack as
    arguments to another proc that is being called from within the proc being
    examined, these will be balanced at the exit of the proc being called.


Modern deompilers like IDA (btw use that link to the free version 5 - they don't advertise that, and all the other versions cost mega$) take a fair bit of the hard work out of this for you by getting creative about how they present their disassembly, but due to the way WC2 uses embedded proc call addresses for optimisation you will still see red flags all over the place warning about stack imbalances because there's simply no way for the decompiler to decipher where many of the procs are being called from, so no way for them to be able to interpret what the state of the stack is before the call.
Axe Thrower Posts: 303 Karma: +20/-0 ****

Lambchops

  • Axe Thrower
  • ****
  • *
  • Posts: 303
    • View Profile
Re: Nerd's Corner
« Reply #12 on: December 20, 2016, 02:04:27 AM »
RE: How WC2 decides which hall a peon exiting a gold mine will return to.


OK don't have time to write a really good explaination but, in a nutshell, it's calculating distance from the top left corner of the mine.

hall is 4x4
mine is 3x3

hall size is taken into account
mine size is not

units are located where their top-left corner is (you can play with the map editor to see this)

Probably they have re-used code from the unit attack AI to find the closest enemy (with annoy factor).





Its the same thing as the old tower bug, where towers firing up and left would hit other towers that wouldnt fire back, because the size of the destination is being taken into account, but not the size of the source.

In this case you can think of the mine as shooting peons at the hall ;)

As mentioned the distance is what I call "square" distance - the maximum X or Y distance from the top left corner of the mine to the closest point on the hall.

so with that map....




because of the extra 2 squares in the width of the mine, the distance becomes 5 not 3, which is greater than the distance between the halls (which is 4) so in effect they are the same distance, in which case the peon will go for the hall with the lower Y co-ordinate (as they were both on the .PUD, if u built them it would go for the one you built first.

 =D

EDIT:
... this is why you dont see this bug on the other side of the map where the halls are to the left of the mine, because then the distance is only 3, so its less then the Y offset....

 (btw: the calculations on the pic above should have -1 on them, = 5 not 6 , but u get the idea)
Axe Thrower Posts: 303 Karma: +20/-0 ****

Lambchops

  • Axe Thrower
  • ****
  • *
  • Posts: 303
    • View Profile
Re: Nerd's Corner
« Reply #13 on: December 20, 2016, 02:14:49 AM »
8-bit palette based graphics
--------------------------------------
Its like this:

For a given bitmap of this type you get to pick your favorite 256 colors from about 1.6 million.

The picture is then made up of those 256 colors. This selection of colors is the "palette".
Then each pixel in the bitmap is one byte which corresponds to the number of one of the colors in the palette.

An 8 bit(1 byte) number can have values 0-255, so an 8 bit image can have at most 256 different colors. However,  when using a palette each part of the color (red,green,blue... or RGB) has a value between 0 and 255, so there are 256^3 (=16777216) colors to choose from, but only 256 of them can be displayed at any one time.

so :
255,0,0         is bright red  (red=255,green=0,blue=0)
20,0,20         is a darkish purple
255,255,255 is bright white
44,44,44       is a darkish grey
66,66,66       is a lighter grey
0,0,0             is black
etc....

A simplified example of how this type of graphics if stored in a file:

HEADER: "I'm a bitmap, im 8-bit, I'm 640 pixels wide by 480 pixels high and i use 256 different colors" (could be <256 but not usually)

PALETTE: list of 256 x 3bytes :  -->  R,G,B for color 0 --> R,G,B for color 1 --> R,G,B for color 2.... etc. to 255 (may be stored in a different order i.e. B,G,R etc)

DATA: 1 byte for each pixel so 640 x 480 =  307,200 bytes*,  - i.e.  color 4, color 162 , color 99 .... or wvr
(*see RLE compression below)

Depending on the file type it will be arranged in slightly different ways: i.e. a BMP is laid out roughly as above, although theres a lot more in the header, which isn't very relevent here, a PNG file has each section compressed using good old PKZip (aka winzip, zlib, 'zip'), PCX is similar to BMP but stores the palette after the pixel data.... and so on.... but regardless of the file, in the end it will be loaded into memory as and array of pixel data  represented by 8-bit indicies into a 256 entry 24 bit color table because that is how that generation of graphics hardware made the pixels on our old CRT monitors light up.

Both BMP and PCX can use a simple form of compression called "run length encoding" or RLE.

RLE simply accounts for long sequences of a single value. For instance if you taks a SS at game start, usually most of your screen will be black, because you havn't scouted anywhere.  In this SS there's maybe half of each pixel line is just a great big line of zeros (pointing to the first palette entry which just happens to be 0,0,0). RLE accounts for this type of thing so instead of actually writing 47 zeros in a row it just makes a little note that says "47 zeros" or "16 '12's" or whatever. The actual way this is written will vary depending on the type of file, however obviously it can be noted using far less than 47 or even 16 bytes (typically 2 or 3).

If it weren't for RLE every SS would be _exactly_ the same size.
Axe Thrower Posts: 303 Karma: +20/-0 ****

Lambchops

  • Axe Thrower
  • ****
  • *
  • Posts: 303
    • View Profile
Re: Nerd's Corner
« Reply #14 on: December 20, 2016, 02:16:04 AM »
<<< HOW TO FIND THAT SS YOU'RE LOOKING FOR >>>

But now we know why they're different sizes you can actually see this by sorting a bunch of SS by size. They end up sorted by the complexity of the image.

Apart from in-game SS, the smallest files are of Human Victory screens because that is the simplest background image, if you look closely at it, you'll see it has not many colors and large blocks of a single color in the clouds and in the castle wall at the lower left. You'll also find that the smallest of these files are from 8 player games, because the stats bars for units, buildings etc. are single blocks of color, so the more players in the game the more stats bars cover the image and the more compressable the image is. The largest of these files will be from 1v1 games.

The Orc defeat screen comes next at the image is a bit more complex, in actual fact you'll probably find 8 player orc defeat screens creeping in around the same size at 6 player human victory screens. It depends on the actual stats of the players and the length of the player names. A little while after this you'll start getting channel screens. Even though there are lots of chunks of black in the text windows, the blue marbled background and bronze edging are actually fairly intricate so these add up to a fair amount of complexity.

Around the end of your channel screens you'll get your 8 player Orc Victory screens with pre-game lounge screens at around 4 player orc victory size and finally, iroicly the most intricate background is on the Human Defeat screen although mostly because of the massively dithered sky / hills. The end of your 1v1 orc victories will be somewhere around the size of a 4 player human defeat.

Your in-game SS will be scattered throughout. A lot of them will be around the size of the lounge screen, however generally SS from earlier in the game where little scouting has been done especially where black unscouted areas are displayed in the main screen will be smaller. In a large collection of late game SS, those on smaller maps would tend to be statistically smaller than those on 128x128 maps, due to a lower complexity mini-map still being displayed using the same number of screen pixels, however this effect is less important that what is on the main display.

In my current folder the very smallest is from the start of a 2v1 chop game right after I halled, most of the screen and mini-map is black and the file is only 152K. I have a 1v1 Human defeat that is 302K (309,079 bytes) which is larger than the theoretical max size for an uncompressed 640x480 PCX with palette+128byte header which is 308,096 bytes, however there are no uncompressed PCX files and the overhead to support the RLE notation for an image that is not effectively RLE compressable will sometimes actually increase the size of the file as in this case..... yet another reason why dithering is crap.

Another thing you might notice, is that black&white SS from when the game is paused are no smaller that color SS. This might seem strange at first, however there is actually almost no difference between the paused screen/SS and the color one. They are both full 256 color screens, the only difference is that when the game is paused the war2 exe swaps the PALETTE for one that has R,G and B values that are all even for each entry, and therefore all shades of grey. When the game continues the full color palette is replaced, however the pixel data for the display remains unchanged.

But anyway, when you want to find that 1v1 victory screen from last month for bragging rights, sort by size, and - if you were orc, you'll find your 1v1 victories are at the large end of your orc victory screens.

(HINT: They're almost always around the 279KB mark, 1v1 Human wins are about 183KB )

GG
=D

(edit) Supplimental info:

When im talking about "orc defeat" screen, im talking about the one with the picture of an orc on it, which comes up when you are human and lose.... cos the orc is smashing your suff...

...also should mention that for in game SS there will also be differences in the file size between different tilesets i.e. forest!=winter, although this depends what's on screen at the time.