Author Topic: Color palettes in channel, tilesets, screenshots, etc.  (Read 12461 times)

Offline Lambchops

  • Ogre Mage
  • ********
  • Posts: 1541
    • View Profile
Re: Color palettes in channel, tilesets, screenshots, etc.
« Reply #15 on: June 03, 2016, 11:13:57 PM »
nice work...i was in 24 bit bmp maybe why i couldnt get the .pal...u into making grps? u seem very keen in the paint stuff.

Happy to help if I can, but I must confess I'm not much of an artist. My interest is far more nerdy than that lol, I understand the internals of graphics containers as a by-product of programming, it helps to know if you want to display pics. I just did a quick prog to pull the palette info out of a pcx and make a bmp.

For the nerds:
grp is an interesting one, its an in-house blizz format. I think its main purpose was to have the graphics prepared for those 1995 486s to blast all those super-flash new graphics onto the screen :) its pretty cool, from a programming angle: each line of pixels is pre packed with a variable offset and length, so the processor only has to go to the exact spot it says to and then squirt out the exact pixels that are there.... very efficiant. Isn't that what normally happens? yes and no, often when you used to have a 'sprite' situation like this where you have to draw a little picture (say a grunt) over a background (grass or whatever) you might have a pic of a grunt which is actually rectangular, with a color (often color 0) designating transparancy. Then the processor has to decided whether or not to draw each pixel in the whole rectangle. grp has just the pixels it needs, and all the possible info that can be calculated before time. Its cool......... Of course these days that sore of stuff is done in nanoseconds by multicore GPUs, but it was neat work back then. Before modern graphics cards CPUs could spend 60-90% of their time just pushing pixels around.
its gooder to hax hard and NEVER get caught!

Offline Lambchops

  • Ogre Mage
  • ********
  • Posts: 1541
    • View Profile
Re: Color palettes in channel, tilesets, screenshots, etc.
« Reply #16 on: September 22, 2016, 02:05:38 AM »
      The return of  The Gigantic Gob    ( queue Scooby-Doo scary music now )



WARNING: Great big nerdy post ahead... continue at you own peril...


Why is there a gigantic gob of....

So, I've been having a look at the extra colors that are added to the channel palette for the banner ads with regards to using them for channel icons.

Some of the stuff I have been talking about with regards to this I have just been assuming based on the results I see being rendered in glorious 8-bit color on my monitor. Understanding how graphics hardware works, there can be no doubt about what is happening (things weren't too complicated in '96), however not having access to the server I could never actually try any of it out. After a recent conversation with Mousey (thanks for your help with that =) I now realise that some of the steps that I was assuming were happening server-side are actually being performed automatically by the client, I suspect in battle.snp or maybe storm.dll. Most notably the color matching.

if you want to take a picture from a bitmap with one palette and display it using another palette, you have to do color matching......Get a SS from the channel. This is an 8-bit palette base pcx file. Convert it into a .BMP file, but make sure it stays 8-bit
...etc

This is stilll a good idea because you will be able to see exactly what your icon will end up looking like, and you won't wate your time designing with colors that just aren't there when it eventually hits the screen, however it turns out that when the icons leave the server they aren't 8-bit palette images at all, they are sent as full 24-bit tga files. So the color-matching must be happening client-side.

The same process takes place, but unfortunatey it's being done automatically somewhere in one of the code modules loaded by the exe. This is nice programming from our friends at Blizz - saving extra steps for themselves later on when developing new stuff as bnet progressed, but sadly for us, it takes away the extra control we might have had by doing it manually and then just sending the finished product to the client.

Mousey tested this out, and despite the fact that the channel screen gets a whole gob of new colors from the banner ads, it seemed the icon color matching refused to use those entries. This puzzled me for a bit, as it would seem to be masive overkill to write in checks that deliberately rejected a certain range of palette entries from the color matching 'just in case' someone at blizz got a bit confused designing icons sometime. So why do it?

Well, maybe one of the programmers was just really anal-retentive and did it anyway...OR... as it dawned on me a bit later (duh) theres a much more likely reason why this is happening: --> As the icons are being served as 24-bit images, then it seems they're mixed down to the 8-bit palette when the client is constructing the channel screen from its various components, then it makes sense that a similar thing is probably happening with the banner ads, only the banners get a much more personalised color conversion, which allows them to import up to 64 new entries into the palette.... and again this would be done automatically by the client.

So... what it seems to me would be happening is: when you log in and the client constructs the channel screen, it starts off with the 'standard' palette that is hard-coded into the exe (or for the channel more likely one of the lib modules) which includes the infamous Gigantic Gob of 64 blank entries. Next the icons are imported and color matched to the existing palette, then finally the banner is imported and the extra palette entries are added. The icons are never matched to the extra palette entries, because at the time the icons are imported, those new entries still havn't been added.

So where does this leave us for extra icon colors? Well unfortunately it seems it would have to be some sort of war2combat client mod, no way around that. Maybe if we're lucky we could get away with just a simple war2patcher mod or perhaps something a bit more substantial would be required. On the face of it, you could easily just overwrite The Gob with a bunch of new entries at its source, then they would already exist when the icons are color-matched and hey-presto... more colors. That's all well and good until the banner loads, at which point it overwrites those entries with it's own and the colors go bleh.

So we can cheat and use the palette from the banner, so the correct entries from the banner are already there so when it overwrites them it does so with exactly the same values and everything works. Cool... well thats if you never want to change the banner, because as soon as you do, the palette changes and we're back to bleh again :/...

Lambchops scratches his head.... thinks hard (ouch)... then scrambles for the mouse and pulls up a SS with a colorful banner ...


Cool. So i just did a quick crop of just the banner part of a SS, then pasted it back onto a reduced size 8-bit bitmap saved that, then loaded it with Irfanview and it says: "63 unique colors." Nice, so it looks like the banners only use their extra 64 colors (I probably cropped one off at an edge somewhere) and they don't do any color-matching to the standard palette entries. This means that if the people waving the banners are happy settling on having only one palette, we can write that in to the client they can still change the banners without messing up the icons.... Maybe.

Yeah there's always a maybe. The other problem that might occur is that by rights, according to the official specs ... certainly for .bmp format.. and possibly for others as well... palette entries are supposed to be ordered according to their frequency of use... i.e. if your picture contains a heap of a certain shade of green, then that color should be placed at the start of the palette, followed by the color that has the next highest amount of pixels using it, right down to the color that is only used for a handfull of dots at the end. The theory being that this reduces cpu usage as the rendering routine doesn't have to search as far through the palette to locate the colors its more frequently using.

In practice it's such a tiny amount of data anyway that it really doesn't matter in the slightest, and many libraries don't bother doing it............If you still don't know what im referring to, 'it' is the process of, when saving or constructing a new palettised image, going through each unique color that occurs in the image one by one and counting up the number of pixels that use each of those colors, then resorting the palette entries according to this associated frequency data .......So if Captian Anal was programming that bit, there's a chance that even if a new banner contains all the same colors as the old one, they might possibly end up in a completely different order..... and we're back to bleh again. But I dont think this is a big chance, its far quicker and easier to just copy the imported palette straight over The Gob. Anyway it's something that's relatively easy to test.

But REALLY what we want to do is have the client just load the banner ad first, then load the icons after it, and everything would be fine. At first thought that didnt seem too hard - these processes would undoubtably be written as seperate subroutines, then we could just put a debug watch on the memory addresses where The Gob lives and track down the instructions that write to that location, then its relatively simple to trace it back to the call that enters that sub. The icon matching might be a little bit trickier to find, but once we have those, just exchanging their call targets so that they are executed they other way round is easy-peasy.

Seemed like a nice idea for about 1 minute before the obvious smacked me in the face (story of my life). Problem is that this stuff is part of a conversation between the client and the server, and its very unlikely that the client receives all of the info from the server and piles it up in a great big heap and then starts processing it... that would be very inefficiant and make it a resource hog... most likely each section of data is received from the server and processed on-the-fly ...first it gets icons.bni which is then processed into whatever format the client ends up using it in at which point the client is finished with it, then the next bit - maybe the banner, or the list of players in the channel, or whatever - is received and it overwrites the received copy of icons.bni which is still sitting in the input buffer. So if we wanted to change the order that these are processed in we would have to either:

a) Change the order that the server sends the info to the client  or  b) allocate another buffer and make a backup copy of icons.bni then replace it in the input buffer later on before we call the color matching subroutine.

Method a) would, at worst, require modding not only the client but also the server (yuck.. bugs... headaches... crashes... stuff that), although depending on the actual structure of the client- server conversation, it could be as simple as just getting the client to request the various bits in a different order, and the server might happily comply without a care in the world. If we we're really lucky the network requests from the client might be housed in the same subroutine as their relative graphics importing code and we're back to the simple switcheroo again. But thats a lot of maybes, and a lot of tracing, debugging and swearing at screens full of disassembly to even start testing the theory.

Method b) is technically a bigger mod as it would be actually adding new code to the client, but allocating a buffer and copying a bit of data back and forth is pretty basic stuff and would probably be quite painless to insert, however to get there you still have to do a whole pile of debugging to find the right places to put it.

.... and then there's the cheating method. Just an idea, but one way to avoid the whole client- server transaction and image import routine head-fuck is just to sneak a little bit of code in right at the end that simply saves the 64 entries that make up The Gob to a disk file then have the war2patcher or something write those in when the module is first loaded. The only drawback would be that when the banner is changed, the first time you log in the icon colors would be bleh, but it would only be a matter of exiting out and restarting , and everything would be perfect until the next banner change.

To continue thinking about this I need a bit of info on how the banners are produced. What format are they in when they server sends them to the client? The .bni writer doesnt seem to have anything to handle banners so its some other related format? Are they 24-bit when they leaves the server or already paletised?.... anybody?
its gooder to hax hard and NEVER get caught!

Offline shesycompany

  • Death Knight
  • *********
  • Posts: 3587
  • retired, be in music section
    • View Profile
Re: Color palettes in channel, tilesets, screenshots, etc.
« Reply #17 on: September 22, 2016, 03:17:19 AM »
i caught you lamb like a pokemon!!!!!!!i got your scooby doo lol whole season
« Last Edit: September 22, 2016, 04:23:36 AM by easycompany »

Offline Lambchops

  • Ogre Mage
  • ********
  • Posts: 1541
    • View Profile
Re: Color palettes in channel, tilesets, screenshots, etc.
« Reply #18 on: September 22, 2016, 05:33:05 AM »
i caught you lamb like a pokemon!!!!!!!


I almost get that ....but I hate turning on location services - its the ultimate invasion of google data mining - so I don't do the pokemon thing.

I did see bits of the cartoon once, it was kinda like astro-boy with bright orange wind-up pet guinnie-pigs that shoot lightning bolts or something. Freakin Japanese are nuts sometimes.

But yeah, have to admit I did watch the first Scooby episode lol....  :D

..... and i would have gotten away with it - if it weren't for you meddling kids!


Hey! Here's a real cartoon for ya  --->




Witness my GIGANTIC Awesomness!



... and BTW why cant I work out how to imbed YT on here?..... is that just cos I'm stupid, or do I need some sort of privilage?
its gooder to hax hard and NEVER get caught!

Offline mousEtopher

  • Administrator
  • Ogre Mage
  • *****
  • Posts: 2065
    • View Profile
    • War2.me
Re: Color palettes in channel, tilesets, screenshots, etc.
« Reply #19 on: September 22, 2016, 07:27:08 AM »
Hot damn, lamby, what a post ! Thank you for your investigation & such a thorough and entertaining explanation of the situation :D I feel better now that I understand the purpose of The Gob and know it's not just pointless lazy waste.

To continue thinking about this I need a bit of info on how the banners are produced. What format are they in when they server sends them to the client? The .bni writer doesnt seem to have anything to handle banners so its some other related format? Are they 24-bit when they leaves the server or already paletised?.... anybody?

yay, I get to contribute to this conversation that's way above my head! They're .smk "Smacker" files:

https://en.wikipedia.org/wiki/Smacker_video
http://www.radgametools.com/smkmain.htm

The conversion software is on that second site, iirc there's two versions and you need the older one to convert the banners, and I think it only runs in x86 XP. (Not sure if that version's on the site actually.) I'll assemble any useful materials I can find and send.

astro-boy

Careful lamby, you're showing your age! My dad watched both those shows... xD

... and BTW why cant I work out how to imbed YT on here?..... is that just cos I'm stupid, or do I need some sort of privilage?

Just paste the full url, it won't show in the preview but will work when you publish.
« Last Edit: September 22, 2016, 07:29:44 AM by mousEtopher »
squeak!

Offline shesycompany

  • Death Knight
  • *********
  • Posts: 3587
  • retired, be in music section
    • View Profile
Re: Color palettes in channel, tilesets, screenshots, etc.
« Reply #20 on: September 22, 2016, 09:29:34 AM »
youtube sucks i suggest hopefully finding it on a ftp or other means,naw the pokemon thing was capturing a modder lol ....let me read what you was saying since i can now see straight.

wow great youtube dude that put that up....they usually screw them up like my scooby looked :/ go to keepvid.com if you want to save them.

yeah the vids are smacker..in sc1 the unit portraits are smacker :O why it looks so 3d.
yeah smaker is like 2000 win98 only to create.but u can watch.

the images,from looking in my paint shop they are 8bit.(my grp's)

wanna play with tiles they are .raw but out of order. :(

i take that back i can write to them using .raw but i still cant read them,the header is missing wonder what happen if you put a header back on it....perplexing
« Last Edit: September 22, 2016, 12:09:26 PM by easycompany »

Offline Lambchops

  • Ogre Mage
  • ********
  • Posts: 1541
    • View Profile
Re: Color palettes in channel, tilesets, screenshots, etc.
« Reply #21 on: September 30, 2016, 10:25:08 PM »
........from looking in my paint shop they are 8bit.(my grp's)

wanna play with tiles they are .raw but out of order. :(

i take that back i can write to them using .raw but i still cant read them,the header is missing wonder what happen if you put a header back on it....perplexing

Hey mate. Sorry, missed this post, thanks for the info - will have a look at this stuff again soon, very interested but flat out right now.

Careful lamby, you're showing your age! My dad watched both those shows... xD

 :'(  I know right? I swear to God im 15 and looking forward to being all mature when I hit 16... lol


« Last Edit: September 30, 2016, 10:28:29 PM by Lambchops »
its gooder to hax hard and NEVER get caught!

Offline shesycompany

  • Death Knight
  • *********
  • Posts: 3587
  • retired, be in music section
    • View Profile
Re: Color palettes in channel, tilesets, screenshots, etc.
« Reply #22 on: October 01, 2016, 02:42:32 AM »
Hey mate. Sorry, missed this post, thanks for the info - will have a look at this stuff again soon, very interested but flat out right now.

lol about the same