Register - Login
Views: 86639763
Main - Memberlist - Active users - Calendar - Wiki - IRC Chat - Online users
Ranks - Rules/FAQ - JCS - Stats - Latest Posts - Color Chart - Smilies
10-24-17 01:01:12 AM

Jul - Game Research/Hacking/Modding - Bustin' Shockwave New poll - New thread - New reply
Next newer thread | Next older thread
brian151
Banned
The administration (as well as a few users) have decided that you're creepy/weird enough that it's time to activate the sploded clause.
Laters.


Level: NaN


Posts: 20/-249
EXP: NaN
For next: 0

Since: 08-09-16
From: USA

Since last post: 82 days
Last activity: 55 days

Posted on 11-14-16 03:42:24 AM Link | Quote
So, as I recently announced/published, I've been trying to clone the old LEGO Spybotics game. In the midst of this, I did some research along the lines of completely decompiling or reverse-egineering the format. I've done this because I want a non-adobe tool that can read/[maybe modify] shockwave movies. perhaps play them (EXTREMELY unlikely, won't be my project to succeed, if it should happen)

So, I discovered a huge cache of information concerning the bytecode in particular, in addition to learning why that "XFIR" header appears on all the shockwave movies I presently have. It seems odd this was there, I never understood it. Well, Macromedia had used a special variant of the Resource Interchange File Format, not an in-house one.

From there, I started diving deeper. For the last week or so, I've been combing through one of the files of interest (the diropened spybotics: nightfall incident) in a hex editor, trying to peace together how in the hell it's structured. Recently, while searching for the actual chunks that SWA contain audio (found the files/data) , and bitmap cast members (neither header nor data), I discovered the way the mmap chunk works (mostly) It's essentially an offset table for EVERYTHING in the file (even itself, and the other main header chunks!) It also demystified why I was seeing so many of the supposed "interesting" chunks, that weren't at ALL their proper length or structure. These were all entries in the mapping chunk. With that knowledge in hand, I can find basically ANYTHING in the file (manually, RIFF viewers can't read jack, they say file is invalid)

I used this information to identify a bit more, including the chunks containing text (and/or text fields) and flash .swf files... I managed to extract three chunks related to the compiled bytecode (that diropener can recover...lol) , but I'll get back to that, later.

More recently (last two days) I created an XFIR-compatable JS RIFF parser, and have started building a tool to basically decompile the [uncompressed/unprotected] director/shockwave movie files. Currently, it can parse the mmap chunk to get a list of chunk types in a given .dir I have used this to indentify what I think are the palettes, and early today, I did manage to make headway in figuring-out how casts and cast members all fit together. (as it stands, I could make a tool to dump basically the whole file, and even disassemble the bytecode, but not yet link it all together)

Now, let's talk about the bytecode. I have never messed much with any programming language that uses a stack, or implements other low-level memory management (except a few tries at assembly, which I have a LOOOOOOONG way to go, should I even continue) the bytecode is used for a stack-based system. I could disassemble it to a human-readable form, but too fully decompile it down to javascript/lingo (both are actually valid for shockwave) , I'd need to understand how to turn stack-based code to something like
function foo(bar) {
return ("foo.bar : " + bar);
}
I don't get how to do THAT at all...
One of my goals is to successfully de-compile the scripts, so does ANYONE here understand how to use a stack-oriented language (and transcribe it to something with more formally declared functions/objects)?
(also, IDK a REASONABLE name for the opcodes, so that'd be helpful, too)

goals:
parse a .dir/.dcr/.dxr and get literally EVERYTHING out of it, in the way of assets*
decompile the bytecode to at the very least, have some understanding how the devs did something*
perhaps convert the movie to a custom (and totally open format)
fully (or almost fully) document the shockwave movie format*
open the door to preserving/porting or ripping from one hell of a lot of old games*
map-out a given movie, showing how it was structured by its devs
possibly an open-source player/editor

*primary concerns, ideal to complete entirely

I seriously need help with this, though, IDK how far I can get before calling it quits, or hitting something I just cannot solve.

current Target games (that I can remember right now):
Spybotics: The Nightfall Incident (goal: complete port to JAVA or HTML5, complete: most currently do-able ripping, certain key parts of research into how code part might work [without aid of any original code])
WorldBuilder 1 & 2 (goal: re-create mapping system, maybe port, complete: tile rips)
Junkbot and Junkbot Undercover (goal: complete rip, complete: [not done by me] presumably 'hard-rips' of everything or close to it, on TSR)

Sorry this is such a LONG post, and I also apologize if this is entirely the wrong section.






brian151
Banned
The administration (as well as a few users) have decided that you're creepy/weird enough that it's time to activate the sploded clause.
Laters.


Level: NaN


Posts: 110/-249
EXP: NaN
For next: 0

Since: 08-09-16
From: USA

Since last post: 82 days
Last activity: 55 days

Posted on 02-22-17 05:05:46 AM Link | Quote
Alright, so... I have somewhat renewed my interest and attempts to crack this file format, or even lay the ground work for fully decompiling games made with it. (Especially with the push to get rid of plugins, and most especially any owned by Adobe)

Two problems I encountered:
https://github.com/Brian151/OpenShockwave/blob/master/examples/SPYBOT_NIGHTFALL-LctX_0x658D66.txt

This doesn't match some of the previous documentation (not made by me) ever so slightly. It also seems to be irrelevant in general, but I don't see why Director would compile scripts or script collections in that are completely empty. Seems kinda wasteful to me. (unless it's the result of a compiled script being corrupted/deleting by dir-opening the .dcr, but I somewhat doubt that)

Other problem with this file is I have yet to determine the ID of any of the referenced name tables and scripts, problematic, to say the least.

Now, in the broader scope of things, second problem:
https://github.com/Brian151/OpenShockwave/blob/master/docu/OPs.txt

I tried to write fairly short tokens/IDs for each of the opcodes... TRIED... Some I can't seem to write in any shorthand methods, and others seem to rely on previous opcodes to determine their specific context, as the documentation says they pull a value from the stack to determine what value to get or set. (for example, a date).

My reasonings for this mainly are:
1. In the event I can't create a complete decompiler (extremely likely...) , there's still some output of what the compiled bytecode is assumed to be doing.
2. It makes debugging/developing a decompiler somewhat easier, since it's easier to read these opcode names (with names extracted from the name tables, or values from the constant records) than a hexadecimal or decimal dump of the actual bytes.
3. Probably will be used in a 'raw data' view of source code.
4. Keeping the names short, since there's no point in being super verbose if I don't explicitly need to. Actually could make it easier to read in some cases

I suppose to some extent, one could call me an idiot. I honestly do not fully understand what I am doing here. This is all based on previous findings that until last year I did not know even existed, and much trial-and-error. I've stated many times I do not intimately understand binary formats, or low-level computing. I also don't understand stack-based programming particularly well, either. I once tried to read and understand the documentation for ActionScript3 bytecode, and failed pretty horribly at it.

However, Lingo is far less advanced than Flash Actionscript, and based on the documentation provided by others, I mostly understand what the opcodes would say to do. (except in areas where no one seems to be sure) My biggest challenge is put all the rest of the pieces of the puzzle together so that I can extract SOME form of human-readable data from the scripts. However, I'm not really sure how all the bytecode-related sections actually go together, and I'm really not sure how to transform stack-based bytecode into something more like JavaScript (or in this case, Lingo...although both are actually valid in the source code of Director/Shockwave movies)

Anyways, (constructive) feedback, or even help would be greatly appreciated. I know that this CAN be done. But IDK if it ever WILL be done. It'd be cool if my research either realizes this, or pushes that goal further towards completion at the very least. Finally having an understanding of Shockwave opens the door to so many possibilities concerning the hacking, archiving, and restoration of a lot of older (at least, compared to what we have today) games. I for one want to see it happen. I'm glad to lend a hand to it.

brian151
Banned
The administration (as well as a few users) have decided that you're creepy/weird enough that it's time to activate the sploded clause.
Laters.


Level: NaN


Posts: 125/-249
EXP: NaN
For next: 0

Since: 08-09-16
From: USA

Since last post: 82 days
Last activity: 55 days

Posted on 03-14-17 05:57:38 AM Link | Quote
https://github.com/Brian151/OpenShockwave/commit/453ac78ce1419161d193c0845a8be0c048cc0203

Honestly, I've been doing a lot with this...
but it'd be pointless to constantly post comments about the latest commits, so to summarize latest progress (not in exact order cuz...my memory):

1. parsing length and offsets from mmap section
2. linking LctX , Lnam, Lscr
3. understanding CLUT
4. slowly but surely piecing together the CASt sections... (they're kinda annoying, seriously)
5. various documentation on the formats
6. There is a bytecode disassembler in-progress, but it was developed independent of my parser, so it doesn't use the names table yet
7. Due to modification of machine code instructions, a newer version of diropener has been made, it doesn't always work, though.
8. the list/preference format can be converted to JSON

Between the fact I'm actually really starting to get a hang of what I should be looking for in the files, and that I picked-up someone to help with this rather expansive project, a lot is happening. I would be very shocked at this point if my efforts in the end are not complete, or whatever might happen to cancel it, that someone else doesn't soon completely crack the Director/Shockwave format. Truth be told, I'm surprised it didn't happen a lot sooner. At least two of the formats (DCR/CCT,DIR/CST) are anything but 'secure' or even obscure... (aside from compressed ones using Varints, not exactly a standard data type)

Anyways, I'm excited. Most of the days I apply myself to working on this, I manage to find something, and/or write a parser for it. My collaborator also usually has something to report.
brian151
Banned
The administration (as well as a few users) have decided that you're creepy/weird enough that it's time to activate the sploded clause.
Laters.


Level: NaN


Posts: 218/-249
EXP: NaN
For next: 0

Since: 08-09-16
From: USA

Since last post: 82 days
Last activity: 55 days

Posted on 07-10-17 05:28:43 AM Link | Quote
Wow, I've seriously neglected this thread!

https://github.com/Brian151/OpenShockwave/commit/3d275d6c8786cb24546692408bb3aa94fa93c185

also... Back from a hiatus on this project

https://github.com/Brian151/OpenShockwave/issues/9
https://github.com/Brian151/OpenShockwave/issues/5

In a nutshell:
Figuring-out the cast list format
Bitmap data is slowly being deciphered, albeit, almost totally not by me
And the SWA format is at least understood enough to separate the playback data from the MP3 stream (in theory, this also is not my observation)

So exciting!

As for posting to this thread, ill still try not to like, spam, but I also don't want to totally neglect it.
I will try to remember to post significant updates...

Now, I should go to bed, I'm up a couple hours past largely because I decided to just get this new stuff DONE.

brian151
Banned
The administration (as well as a few users) have decided that you're creepy/weird enough that it's time to activate the sploded clause.
Laters.


Level: NaN


Posts: 234/-249
EXP: NaN
For next: 0

Since: 08-09-16
From: USA

Since last post: 82 days
Last activity: 55 days

Posted on 07-23-17 01:57:27 AM Link | Quote
So, a lot has been going on...

https://github.com/Brian151/OpenShockwave/commit/f2eba788251cad574995d11608323b7bf31ad720

also:

https://github.com/Brian151/OpenShockwave/tree/master/tools/lscrtoscript

Yes, we're now officially able to decompile Lingo, partially!

There's a lot more in the works or pending.
We will most likely be re-naming the whole project to EarthQuake, and significantly re-structuring everything. I currently am cleaning-up the documentation and example directories, right now.
We have adopted a naming convention for applications/libraries, and it's to use earthquake-related puns! As of yet, it's not been officially deployed, this will happen when the repo gets completely overhauled and re-named.

Sorry if I'm not making myself terribly clear here, I know I'm really not covering anything in detail, and that's simply because a LOT has been happening, and it's take too much time to list all of that right now, as progress isn't documented nearly as well as all other aspects of this project.

divingkataetheweirdo

Lantern Ghost
TCRF Super Editor
Level: 52


Posts: 739/765
EXP: 1014519
For next: 69321

Since: 07-09-11


Since last post: 1 day
Last activity: 2 hours

Posted on 07-23-17 02:07:10 AM Link | Quote
Impressive work. Considering Shockwave is ancient and none of the games run on anything newer than XP, this is some relieving news. Could probably crack open Total Distortion now (and yes, that game did use Shockwave).
brian151
Banned
The administration (as well as a few users) have decided that you're creepy/weird enough that it's time to activate the sploded clause.
Laters.


Level: NaN


Posts: 235/-249
EXP: NaN
For next: 0

Since: 08-09-16
From: USA

Since last post: 82 days
Last activity: 55 days

Posted on 07-23-17 07:11:36 AM Link | Quote
Originally posted by divingkataetheweirdo
Impressive work. Considering Shockwave is ancient and none of the games run on anything newer than XP, this is some relieving news. Could probably crack open Total Distortion now (and yes, that game did use Shockwave).


Eh, I didn't have a whole lot to do with the decompiler, mainly just naming opcodes, and certain parts of the documentation.

But, thanks!

I need to sleep like, now... but I will be trying to mess with this PFR font format thing 'tomorrow'
Finally finished manually verifying the header on one of my example files!
(I screwed-up like two or three times...)

Not sure I've heard of that game...lol
Indeed my primary motivation for this is making such efforts possible, as I said before, I have a small list of games I wanna crack-open, myself.
If you can provide any sample files, it'd be helpful. Especially in regards to the decompiler!
Next newer thread | Next older thread
Jul - Game Research/Hacking/Modding - Bustin' Shockwave New poll - New thread - New reply




Rusted Logic

Acmlmboard - commit 2f1bc75 [2017-08-27]
©2000-2017 Acmlm, Xkeeper, Inuyasha, et al.

26 database queries.
Query execution time: 0.162254 seconds
Script execution time: 0.010890 seconds
Total render time: 0.173144 seconds