Goldeneye map hacking.

[MrHTFord]

Hi,

I've been hacking goldeneye on and off for about 6 months now and am pleased to present some results.

I've reversed the compression code and the model format.

Here are some renders taken from Maya of the silo level; I've not got all of the textures sorted out yet, but am working on it.

If anyone wants to collaborate on re-compressing maps back into Goldeneye (or hacking Perfect Dark), I could be tempted.

FYI, the maps are all saved out in Collada format.

[SubDrag]

Originally Posted by MrHTFord

I've reversed the compression code and the model format.

Are you f-ing serious? Where have you been all this time! Which format did you reverse? 1172 type %b 101?

I'm quite interested...
We reversed a lot of the model format, including textures and stuff, but the compression is elusive other than the one type I know.

[MrHTFord]

Hi SubDrag,

I remember asking you for specific help on the compression format in January and you telling me only that it was a variant of LZB. Perhaps you thought I was a time waster?

Anyway, I went off and reverse engineered the MIPS code by hand... took ages, but I got there.

I reversed all of the decompression code, so yes, it handles all the block types and I can decompress every 0x1172 block from Goldeneye. My reverse engineering method was a very literal MIPS asm->C code conversion, so I've not made any attempt to understand the actual format. I'd like to understand it better so I can recompress the data.

Also, whilst I've got your attention, I'm having a bit of trouble with a few of the textures. Though I'm thinking now that they aren't actually textures. Specifically, the textures that begin with "07 84", one is at address 0x9ad551 (~128 bytes in size), another at address 0x92fa47 (~1109 bytes in size).

Anyway, I've been reading on another thread about attempting to create a level editor. I recommend not trying to do this and instead using standard tools (Maya/XSI/3dstudioMax/Blender/Milkshape) and standard formats instead. In fact, if you used something like the Collada format, people could edit the levels in their favourite package and just export them to that format.

Cheers...

[SubDrag]

Update on level editor:
I tried to make one in directX, I got it displaying each of the level rooms, but never could get the textures displaying properly. It did rotate and all that jazz, but I realized it would be too difficult and I gave up.

Instead, I've been working on a window-based setup editor. It's actually pretty far in development, actually the hold-up is knowledge of their crazy pseudo-code for the level-specific actions/stuff. It would go hand in hand with a model importer, and we could really remake levels (that was actually our plan, to import, and then export back into the game). Zoinkity and I have made a lot of progress detailing the ROM, gimme a msg. And I don't think your models had images there? They use presets into the ROM.

07XX is another form of compression involving greyscale images. Uses some kind of table, then offsets into the table, or something along those lines. There are actually more compression types than 07 and 1172 even.

Do you have AIM/ICQ? I've got an unreleased pictorial detailing of every image in the ROM. And want to congratulate you on your work, I spent a long time trying, had no success trying to reverse it to algorithm, though I prob should've tried your approach.

[MrHTFord]

I've not got AIM (AOL - uck) or ICQ, I've got MSN if that helps though. I can look into my code and give you the correct method for handling the %b101 blocks, there is a Run Length Encoding scheme used for one of the block types, so perhaps that's what tripped you up.

How did you figure out the compression method? Did you do it by examining the outputs against the inputs?

My models do have images on them, I've attached another one so you can see it more clearly. Basically, the model format requires processing GBI command lists (as the graphics plugins for the emulators do)

Got to go for now. It's too sunny outside!

Speak later.

[SubDrag]

Almost all of them combine the RLE with LZ, that's in the type I know, one of the types in uncompressed really, the type I have is specialized LZB, and then there's type 101 and 110 which are essentially the same except 110 can modify its whole bit scheme. Be very curious for it. The whole compression reversing for the one type was some RE-ing code for help, but it had to involve a lot of brute force too. What trips me up is how the table details in the beginning work, how they set all the sets of bit sequeneces.

Impressive model stuff. I do have MSN messenger, though I have to turn it on and don't like it really, but it's ok. You're missing the greyscale images that are RGB-ified.

[MrHTFord]

Hi,

I'm back again, Brits can only stand the sun for so long!

I think the best thing to do would be to get together on MSN then and discuss things properly there. I'm willing to help you with the bit you're missing, and it sounds like you'll be able to help me short circuit doing more reverse-engineering on more tedious texture code.

I've had a quick look at my verbatim decompilation of the n64 decompression code and here's an overview:-

There's only 3 types of compressed block, determined by the first 2 bits. If you are reading 3 bits to get the compression type, that's a bit too far. One of the types is a literal copy, the other two are both complicated RLE/LZB codings, the last bit pattern (0x3) is a terminator.

Why do you want to create a level editing tool? If you want to go the full way, then you'll want to implement full on polygon editing, and without meaning to be offensive, there'll always be much better tools on the market (and indeed free tools) that do the job, and interchangeable formats that can be used to allow artists with skills on different packages to make new Goldeneye levels.

Anyway, lets arrange a time to discuss it all; there's a lot to go over! I'm in the UK btw.

[SubDrag]

Added you to MSN, you haven't been on.

Hrmm, I always thought it was 3-bits for the designator, you sure about that? Yeah literal copy I know, one is the LZ/RLE fixed bit sequences, the other involves a changing bit sequence pattern thing (one I don't know). However, I believe the third bit determines if the table can be redone or something. Could be wrong.

I think you're confusing things. The setup editor is all the in-level stuff, like where stuff spawns, what it is, what paths it is on, objectives, etc. The other part is the walls and levels, that you're clearly heading for, but keep in mind you don't have clipping, have to extrapolate it as well.

The plan was always to have walls edited by an external program and converted, I was just hoping to be able to display/edit setup spots.

[MrHTFord]

I'm not on MSN at the moment. I'm off out now anyway. Need to arrange for in the week or perhaps tomorrow.

[spoondiddly]

Also need portals, but rather easy to create. That's in background data along with the offsets to room mapping/point tables and their positions within level. Totally uncompressed.

Clipping is easy enough to generate from mapping data. It uses a different format (virtually raw triangle, but with a fun header), but basically just constructs polys from surfaces below a certain angle. Anything like ladders were likely added later at great expense (these rarely are 1-1, oddly enough) Citadel's exterior walls and floor heights were done using mapping, although interior walls and those ramps had to be raised and pushed out to avoid annoying exploits.

The older the stage, the less likely they were to remove extroneous clipping. You'll see unused flat surfaces all over in GE, such as the rims of containers, overhead platforms, and other useless things.

Yeah, the setup editor would work a thousand times better with some actual reference. 2xxx presets, guard paths, camera routines, etc all are much easier to set when you can see what you're doing.

[spoondiddly]

oh! hah! Now I know what's wrong with that silo room model. Only the first of the room's mapping is displayed here. The warning stripe overlays, walkways, lights + glass, etc are in the second mapping block for this room.

[MrHTFord]

Hi Spoondiddly,

I'm handling the 2nd GBI block correctly, it's just Z-buffer fighting that's causing the signs to vanish. I've attached a wireframe from Maya so you can see this.

I've already got the portal data, just not outputting it yet. Are you saying that I should be rendering some of the clipping data (in the _stan files)? That might explain why I'm missing the walkways around the Silo missile bays. Any chance of detailing the format of stan files for me?

What are 2xxx presets?

SubDrag: the n64 asm code pulls 2 bits from the input stream and jumps to a different decompression function for each of the 3 possible block types, I'm 100% on this. If the bit pattern is %b10, then the first thing that the handling function does is grabs the next five bits and then the next five bits after that.

With regard to placements for things in levels and guard walk paths, this could probably be done in Maya too.

[spoondiddly]

Ahh, there they are.
Where the heck did you get ahold of the GDI commands? I had to do that crap by hand tediously, and never did get all the commands or options...

Well, you really shouldn't be rendering clipping actually, since it is completely seperate from model data. Possibly as another layer (does Maya use layers, or were they groups? can't remember - been ages since seeing it). It shouldn't have any impact on model data at all. Walkways use the second mapping, and the images for floors and railings have a transparent index value (might be partially transparent to boot). That might screw it up, concidering they don't display right.
Silo is odd really, since most of the rooms have no floors/ceilings. The hatches above the silos are objects, so don't expect them there, although the rails and floors use the transparency-ridden images. Can't remember where it indicates which index value was the transparent one, but most likely not really necessary.

You obviously have stage models down effectively. Have you looked at the object format yet? It uses pointer arrays to link the mapping info with the various point tables. Each table's handle follows the points themselves.

Let me get all the clipping stuff together in one place. Its a very easy format but does tie into setup heavily. The first three bytes must match the values within the setup or most likely the stage will crash. Unlike rooms that use their own centerpoints and are positioned around the stage, clipping uses the global points seen in portals. Don't expect some rooms to have any at all.

Are you certain that the byte following the compression type doesn't indicate table switching within the compression? That could be indicated by a special byte sequence during decompression, but it does seem odd that only those with table switching have a third byte set to 1. Have you attempted building a compressor yet for the three types?

[MrHTFord]

I got the GBI commands from open source glN64 plugin. Google for glN64_v0.4.1_Source.zip. I then re-implemented this code (or as much as I've needed to so far) in Python. To get just the polygons, you need to implement the F3D_VTX command (which copies verts from RAM into the graphics chip's cache) and F3D_TRI4 and F3D_TRI1. If I implement more of it, I should be able to get animated models and other game objects converted too.

What I posted about 2 bits, then (if %b10), 5 bits and 5 bits is definitely correct. As I say, I rev-engineered it in the most literal way, making no attempt to actually understand what it was doing. My initial goal was just to get the data out of GE, but I would be interested in re-compressing so that we can modify the ROM and make our own versions of the game. One of the 3 types is a simple copy operation, so that one will be easy...

What exactly is in the _stan files? Is it just collision/clipping data? Does it contain object placements and guard paths too?

SubDrag : I'll be on MSN all day today if you want to try to hook up.

[spoondiddly]

Well that certainly is easier. I had built a rather ugly c program to convert the uncompressed files into dxf and obj. (well, guess compressed too since only image calls are different) Neither output format supports rgba, so blah. Figuring it out by hand is tedious at best, but it was kinda cool looking around. Unfortunately never could try compressing one, since the one compressor that could handle more than 1000 bytes of data wouldn't allow the all-mighty cop2 they all start with. That doc is a lifesaver really, though now probably will never use it.

Attached the clipping stuff. That was probably a crappy explaination though. It is needed for object placements to determine where they are in the stage. From what I can tell the rooms are really more a nifty backdrop to it.

Object placements, object calls, guard paths, objectives - well, virtualy everything that makes the stage is in a setup file. I think those are the same rom addresses as PAL and the chart doesn't deviate until the text stuff(will check on that). They are the ones named Usetup(lvl)Z and Ump_setup(lvl)Z. From internal notes it seems clipping and setup were originally compiled together and split on cartridge, but that really isn't an issue.

There's something odd with that table though, since the briefing text is not Ubrief(lvl)Z but stored with the in-stage text banks L(lvl)(region). Heck, there's two copies of it, one in lowercase for viewing in the watch menus.

I made some notes on PD model format, but not sure which disk they're on... The point entries are only 0xC large, with reference values to rgba and spectral effect (?) combos following the point table. It looks like those are in 16 bit format, but never really tested it. They also store a header on every object which is just a copy of the handle used in GE.


Ah, yes. The "uncompressed" compression type. Bloody useless though, since it takes a blunt object to convince GE to load anything in the wrong place, much less to keep loading past the next address. Getting unmoddified decompressed binaries is rather easy without a decompressor, though. GE is rather effecient about it, and a simple breakpoint does wonders.