Register - Login
Views: 99408822
 Main - Memberlist - Active users - Calendar - Wiki - IRC Chat - Online users
Ranks - Rules/FAQ - Stats - Latest Posts - Color Chart - Smilies		 04-24-22 09:12:46 PM
Jul - General Game/ROM Hacking - Bustin' Shockwave New poll - New thread - New reply
Next newer thread | Next older thread
brian151
Banned
The administration (as well as a few users) have decided that you're creepy/weird enough that it's time to activate the sploded clause.
Laters.


Level: NaN


Posts: 20/-249
EXP: NaN
For next: 0

Since: 08-09-16

From: USA

Since last post: 4.7 years
Last activity: 4.7 years
Posted on 11-14-16 06:42:24 AM Link | Quote

WIP



So, as I recently announced/published, I've been trying to clone the old LEGO Spybotics game. In the midst of this, I did some research along the lines of completely decompiling or reverse-egineering the format. I've done this because I want a non-adobe tool that can read/[maybe modify] shockwave movies. perhaps play them (EXTREMELY unlikely, won't be my project to succeed, if it should happen)

So, I discovered a huge cache of information concerning the bytecode in particular, in addition to learning why that "XFIR" header appears on all the shockwave movies I presently have. It seems odd this was there, I never understood it. Well, Macromedia had used a special variant of the Resource Interchange File Format, not an in-house one.

From there, I started diving deeper. For the last week or so, I've been combing through one of the files of interest (the diropened spybotics: nightfall incident) in a hex editor, trying to peace together how in the hell it's structured. Recently, while searching for the actual chunks that SWA contain audio (found the files/data) , and bitmap cast members (neither header nor data), I discovered the way the mmap chunk works (mostly) It's essentially an offset table for EVERYTHING in the file (even itself, and the other main header chunks!) It also demystified why I was seeing so many of the supposed "interesting" chunks, that weren't at ALL their proper length or structure. These were all entries in the mapping chunk. With that knowledge in hand, I can find basically ANYTHING in the file (manually, RIFF viewers can't read jack, they say file is invalid)

I used this information to identify a bit more, including the chunks containing text (and/or text fields) and flash .swf files... I managed to extract three chunks related to the compiled bytecode (that diropener can recover...lol) , but I'll get back to that, later.

More recently (last two days) I created an XFIR-compatable JS RIFF parser, and have started building a tool to basically decompile the [uncompressed/unprotected] director/shockwave movie files. Currently, it can parse the mmap chunk to get a list of chunk types in a given .dir I have used this to indentify what I think are the palettes, and early today, I did manage to make headway in figuring-out how casts and cast members all fit together. (as it stands, I could make a tool to dump basically the whole file, and even disassemble the bytecode, but not yet link it all together)

Now, let's talk about the bytecode. I have never messed much with any programming language that uses a stack, or implements other low-level memory management (except a few tries at assembly, which I have a LOOOOOOONG way to go, should I even continue) the bytecode is used for a stack-based system. I could disassemble it to a human-readable form, but too fully decompile it down to javascript/lingo (both are actually valid for shockwave) , I'd need to understand how to turn stack-based code to something like
function foo(bar) {
return ("foo.bar : " + bar);
}
I don't get how to do THAT at all...
One of my goals is to successfully de-compile the scripts, so does ANYONE here understand how to use a stack-oriented language (and transcribe it to something with more formally declared functions/objects)?
(also, IDK a REASONABLE name for the opcodes, so that'd be helpful, too)

goals:
parse a .dir/.dcr/.dxr and get literally EVERYTHING out of it, in the way of assets*
decompile the bytecode to at the very least, have some understanding how the devs did something*
perhaps convert the movie to a custom (and totally open format)
fully (or almost fully) document the shockwave movie format*
open the door to preserving/porting or ripping from one hell of a lot of old games*
map-out a given movie, showing how it was structured by its devs
possibly an open-source player/editor

*primary concerns, ideal to complete entirely

I seriously need help with this, though, IDK how far I can get before calling it quits, or hitting something I just cannot solve.

current Target games (that I can remember right now):
Spybotics: The Nightfall Incident (goal: complete port to JAVA or HTML5, complete: most currently do-able ripping, certain key parts of research into how code part might work [without aid of any original code])
WorldBuilder 1 & 2 (goal: re-create mapping system, maybe port, complete: tile rips)
Junkbot and Junkbot Undercover (goal: complete rip, complete: [not done by me] presumably 'hard-rips' of everything or close to it, on TSR)

Sorry this is such a LONG post, and I also apologize if this is entirely the wrong section.
brian151
Banned
The administration (as well as a few users) have decided that you're creepy/weird enough that it's time to activate the sploded clause.
Laters.


Level: NaN


Posts: 110/-249
EXP: NaN
For next: 0

Since: 08-09-16

From: USA

Since last post: 4.7 years
Last activity: 4.7 years
Posted on 02-22-17 08:05:46 AM Link | Quote

WIP



Alright, so... I have somewhat renewed my interest and attempts to crack this file format, or even lay the ground work for fully decompiling games made with it. (Especially with the push to get rid of plugins, and most especially any owned by Adobe)

Two problems I encountered:
https://github.com/Brian151/OpenShockwave/blob/master/examples/SPYBOT_NIGHTFALL-LctX_0x658D66.txt

This doesn't match some of the previous documentation (not made by me) ever so slightly. It also seems to be irrelevant in general, but I don't see why Director would compile scripts or script collections in that are completely empty. Seems kinda wasteful to me. (unless it's the result of a compiled script being corrupted/deleting by dir-opening the .dcr, but I somewhat doubt that)

Other problem with this file is I have yet to determine the ID of any of the referenced name tables and scripts, problematic, to say the least.

Now, in the broader scope of things, second problem:
https://github.com/Brian151/OpenShockwave/blob/master/docu/OPs.txt

I tried to write fairly short tokens/IDs for each of the opcodes... TRIED... Some I can't seem to write in any shorthand methods, and others seem to rely on previous opcodes to determine their specific context, as the documentation says they pull a value from the stack to determine what value to get or set. (for example, a date).

My reasonings for this mainly are:
1. In the event I can't create a complete decompiler (extremely likely...) , there's still some output of what the compiled bytecode is assumed to be doing.
2. It makes debugging/developing a decompiler somewhat easier, since it's easier to read these opcode names (with names extracted from the name tables, or values from the constant records) than a hexadecimal or decimal dump of the actual bytes.
3. Probably will be used in a 'raw data' view of source code.
4. Keeping the names short, since there's no point in being super verbose if I don't explicitly need to. Actually could make it easier to read in some cases

I suppose to some extent, one could call me an idiot. I honestly do not fully understand what I am doing here. This is all based on previous findings that until last year I did not know even existed, and much trial-and-error. I've stated many times I do not intimately understand binary formats, or low-level computing. I also don't understand stack-based programming particularly well, either. I once tried to read and understand the documentation for ActionScript3 bytecode, and failed pretty horribly at it.

However, Lingo is far less advanced than Flash Actionscript, and based on the documentation provided by others, I mostly understand what the opcodes would say to do. (except in areas where no one seems to be sure) My biggest challenge is put all the rest of the pieces of the puzzle together so that I can extract SOME form of human-readable data from the scripts. However, I'm not really sure how all the bytecode-related sections actually go together, and I'm really not sure how to transform stack-based bytecode into something more like JavaScript (or in this case, Lingo...although both are actually valid in the source code of Director/Shockwave movies)

Anyways, (constructive) feedback, or even help would be greatly appreciated. I know that this CAN be done. But IDK if it ever WILL be done. It'd be cool if my research either realizes this, or pushes that goal further towards completion at the very least. Finally having an understanding of Shockwave opens the door to so many possibilities concerning the hacking, archiving, and restoration of a lot of older (at least, compared to what we have today) games. I for one want to see it happen. I'm glad to lend a hand to it.
brian151
Banned
The administration (as well as a few users) have decided that you're creepy/weird enough that it's time to activate the sploded clause.
Laters.


Level: NaN


Posts: 125/-249
EXP: NaN
For next: 0

Since: 08-09-16

From: USA

Since last post: 4.7 years
Last activity: 4.7 years
Posted on 03-14-17 08:57:38 AM Link | Quote

WIP



https://github.com/Brian151/OpenShockwave/commit/453ac78ce1419161d193c0845a8be0c048cc0203

Honestly, I've been doing a lot with this...
but it'd be pointless to constantly post comments about the latest commits, so to summarize latest progress (not in exact order cuz...my memory):

1. parsing length and offsets from mmap section
2. linking LctX , Lnam, Lscr
3. understanding CLUT
4. slowly but surely piecing together the CASt sections... (they're kinda annoying, seriously)
5. various documentation on the formats
6. There is a bytecode disassembler in-progress, but it was developed independent of my parser, so it doesn't use the names table yet
7. Due to modification of machine code instructions, a newer version of diropener has been made, it doesn't always work, though.
8. the list/preference format can be converted to JSON

Between the fact I'm actually really starting to get a hang of what I should be looking for in the files, and that I picked-up someone to help with this rather expansive project, a lot is happening. I would be very shocked at this point if my efforts in the end are not complete, or whatever might happen to cancel it, that someone else doesn't soon completely crack the Director/Shockwave format. Truth be told, I'm surprised it didn't happen a lot sooner. At least two of the formats (DCR/CCT,DIR/CST) are anything but 'secure' or even obscure... (aside from compressed ones using Varints, not exactly a standard data type)

Anyways, I'm excited. Most of the days I apply myself to working on this, I manage to find something, and/or write a parser for it. My collaborator also usually has something to report.
brian151
Banned
The administration (as well as a few users) have decided that you're creepy/weird enough that it's time to activate the sploded clause.
Laters.


Level: NaN


Posts: 218/-249
EXP: NaN
For next: 0

Since: 08-09-16

From: USA

Since last post: 4.7 years
Last activity: 4.7 years
Posted on 07-10-17 08:28:43 AM Link | Quote


Wow, I've seriously neglected this thread!

https://github.com/Brian151/OpenShockwave/commit/3d275d6c8786cb24546692408bb3aa94fa93c185

also... Back from a hiatus on this project

https://github.com/Brian151/OpenShockwave/issues/9
https://github.com/Brian151/OpenShockwave/issues/5

In a nutshell:
Figuring-out the cast list format
Bitmap data is slowly being deciphered, albeit, almost totally not by me
And the SWA format is at least understood enough to separate the playback data from the MP3 stream (in theory, this also is not my observation)

So exciting!

As for posting to this thread, ill still try not to like, spam, but I also don't want to totally neglect it.
I will try to remember to post significant updates...

Now, I should go to bed, I'm up a couple hours past largely because I decided to just get this new stuff DONE.



____________________

awesome title later...
brian151
Banned
The administration (as well as a few users) have decided that you're creepy/weird enough that it's time to activate the sploded clause.
Laters.


Level: NaN


Posts: 234/-249
EXP: NaN
For next: 0

Since: 08-09-16

From: USA

Since last post: 4.7 years
Last activity: 4.7 years
Posted on 07-23-17 04:57:27 AM Link | Quote


So, a lot has been going on...

https://github.com/Brian151/OpenShockwave/commit/f2eba788251cad574995d11608323b7bf31ad720

also:

https://github.com/Brian151/OpenShockwave/tree/master/tools/lscrtoscript

Yes, we're now officially able to decompile Lingo, partially!

There's a lot more in the works or pending.
We will most likely be re-naming the whole project to EarthQuake, and significantly re-structuring everything. I currently am cleaning-up the documentation and example directories, right now.
We have adopted a naming convention for applications/libraries, and it's to use earthquake-related puns! As of yet, it's not been officially deployed, this will happen when the repo gets completely overhauled and re-named.

Sorry if I'm not making myself terribly clear here, I know I'm really not covering anything in detail, and that's simply because a LOT has been happening, and it's take too much time to list all of that right now, as progress isn't documented nearly as well as all other aspects of this project.



____________________

awesome title later...
divingkataetheweirdo

Bandit
TCRF Super Editor
Level: 57


Posts: 739/822
EXP: 1479805
For next: 6123

Since: 07-09-11


Since last post: 1.6 years
Last activity: 253 days
Posted on 07-23-17 05:07:10 AM Link | Quote
Impressive work. Considering Shockwave is ancient and none of the games run on anything newer than XP, this is some relieving news. Could probably crack open Total Distortion now (and yes, that game did use Shockwave).

____________________
brian151
Banned
The administration (as well as a few users) have decided that you're creepy/weird enough that it's time to activate the sploded clause.
Laters.


Level: NaN


Posts: 235/-249
EXP: NaN
For next: 0

Since: 08-09-16

From: USA

Since last post: 4.7 years
Last activity: 4.7 years
Posted on 07-23-17 10:11:36 AM Link | Quote


Originally posted by divingkataetheweirdo
Impressive work. Considering Shockwave is ancient and none of the games run on anything newer than XP, this is some relieving news. Could probably crack open Total Distortion now (and yes, that game did use Shockwave).


Eh, I didn't have a whole lot to do with the decompiler, mainly just naming opcodes, and certain parts of the documentation.

But, thanks!

I need to sleep like, now... but I will be trying to mess with this PFR font format thing 'tomorrow'
Finally finished manually verifying the header on one of my example files!
(I screwed-up like two or three times...)

Not sure I've heard of that game...lol
Indeed my primary motivation for this is making such efforts possible, as I said before, I have a small list of games I wanna crack-open, myself.
If you can provide any sample files, it'd be helpful. Especially in regards to the decompiler!

____________________

awesome title later...
Next newer thread | Next older thread
Jul - General Game/ROM Hacking - Bustin' Shockwave New poll - New thread - New reply


Rusted Logic
Acmlmboard - commit 47be4dc [2021-08-23]
©2000-2022 Acmlm, Xkeeper, Kaito Sinclaire, et al.

26 database queries.
Query execution time:  0.076373 seconds
Script execution time:  0.019094 seconds
Total render time:  0.095467 seconds


TidyHTML vomit below
line 1 column 1 - Warning: missing <!DOCTYPE> declaration
line 119 column 11 - Warning: <form> isn't allowed in <table> elements
line 118 column 10 - Info: <table> previously mentioned
line 120 column 11 - Warning: missing <tr>
line 120 column 119 - Warning: missing </font> before </td>
line 124 column 16 - Warning: plain text isn't allowed in <tr> elements
line 120 column 11 - Info: <tr> previously mentioned
line 125 column 68 - Warning: missing </nobr> before </td>
line 141 column 68 - Warning: missing </nobr> before <tr>
line 147 column 35 - Warning: missing <tr>
line 147 column 50 - Warning: missing </font> before </td>
line 148 column 37 - Warning: unescaped & or unknown entity "&id"
line 147 column 190 - Warning: missing </font> before </table>
line 149 column 35 - Warning: missing <tr>
line 149 column 50 - Warning: missing </font> before </td>
line 149 column 91 - Warning: missing </font> before </table>
line 156 column 9 - Warning: <div> isn't allowed in <table> elements
line 152 column 17 - Info: <table> previously mentioned
line 158 column 9 - Warning: missing <tr>
line 176 column 13 - Warning: missing <tr>
line 177 column 102 - Warning: unescaped & or unknown entity "&postid"
line 179 column 74 - Warning: <style> isn't allowed in <td> elements
line 179 column 9 - Info: <td> previously mentioned
line 228 column 9 - Warning: <div> isn't allowed in <table> elements
line 152 column 17 - Info: <table> previously mentioned
line 230 column 9 - Warning: missing <tr>
line 248 column 13 - Warning: missing <tr>
line 249 column 102 - Warning: unescaped & or unknown entity "&postid"
line 251 column 74 - Warning: <style> isn't allowed in <td> elements
line 251 column 9 - Info: <td> previously mentioned
line 283 column 9 - Warning: <div> isn't allowed in <table> elements
line 152 column 17 - Info: <table> previously mentioned
line 285 column 9 - Warning: missing <tr>
line 303 column 13 - Warning: missing <tr>
line 304 column 102 - Warning: unescaped & or unknown entity "&postid"
line 306 column 74 - Warning: <style> isn't allowed in <td> elements
line 306 column 9 - Info: <td> previously mentioned
line 328 column 9 - Warning: <div> isn't allowed in <table> elements
line 152 column 17 - Info: <table> previously mentioned
line 330 column 9 - Warning: missing <tr>
line 348 column 13 - Warning: missing <tr>
line 349 column 102 - Warning: unescaped & or unknown entity "&postid"
line 351 column 74 - Warning: <style> isn't allowed in <td> elements
line 351 column 9 - Info: <td> previously mentioned
line 351 column 1089 - Warning: missing </div>
line 377 column 9 - Warning: <div> isn't allowed in <table> elements
line 152 column 17 - Info: <table> previously mentioned
line 379 column 9 - Warning: missing <tr>
line 397 column 13 - Warning: missing <tr>
line 398 column 102 - Warning: unescaped & or unknown entity "&postid"
line 400 column 74 - Warning: <style> isn't allowed in <td> elements
line 400 column 9 - Info: <td> previously mentioned
line 400 column 1089 - Warning: missing </div>
line 421 column 9 - Warning: <div> isn't allowed in <table> elements
line 152 column 17 - Info: <table> previously mentioned
line 423 column 9 - Warning: missing <tr>
line 441 column 13 - Warning: missing <tr>
line 442 column 102 - Warning: unescaped & or unknown entity "&postid"
line 447 column 9 - Warning: <div> isn't allowed in <table> elements
line 152 column 17 - Info: <table> previously mentioned
line 449 column 9 - Warning: missing <tr>
line 467 column 13 - Warning: missing <tr>
line 468 column 102 - Warning: unescaped & or unknown entity "&postid"
line 470 column 74 - Warning: <style> isn't allowed in <td> elements
line 470 column 9 - Info: <td> previously mentioned
line 470 column 1089 - Warning: missing </div>
line 487 column 17 - Warning: missing <tr>
line 487 column 17 - Warning: discarding unexpected <table>
line 490 column 35 - Warning: missing <tr>
line 490 column 50 - Warning: missing </font> before </td>
line 490 column 91 - Warning: missing </font> before </table>
line 492 column 35 - Warning: missing <tr>
line 492 column 50 - Warning: missing </font> before </td>
line 493 column 37 - Warning: unescaped & or unknown entity "&id"
line 492 column 190 - Warning: missing </font> before </table>
line 494 column 17 - Warning: discarding unexpected </textarea>
line 494 column 28 - Warning: discarding unexpected </form>
line 494 column 35 - Warning: discarding unexpected </embed>
line 494 column 43 - Warning: discarding unexpected </noembed>
line 494 column 53 - Warning: discarding unexpected </noscript>
line 494 column 64 - Warning: discarding unexpected </noembed>
line 494 column 74 - Warning: discarding unexpected </embed>
line 494 column 82 - Warning: discarding unexpected </table>
line 494 column 90 - Warning: discarding unexpected </table>
line 496 column 9 - Warning: missing </font> before <table>
line 508 column 25 - Warning: discarding unexpected </font>
line 517 column 37 - Warning: discarding unexpected </font>
line 495 column 1 - Warning: missing </center>
line 120 column 63 - Warning: <img> lacks "alt" attribute
line 125 column 19 - Warning: <td> attribute "width" has invalid value "120px"
line 125 column 93 - Warning: <img> lacks "alt" attribute
line 141 column 19 - Warning: <td> attribute "width" has invalid value "120px"
line 141 column 98 - Warning: <img> lacks "alt" attribute
line 148 column 44 - Warning: <img> proprietary attribute value "absmiddle"
line 148 column 142 - Warning: <img> proprietary attribute value "absmiddle"
line 148 column 246 - Warning: <img> proprietary attribute value "absmiddle"
line 161 column 23 - Warning: <img> lacks "alt" attribute
line 161 column 64 - Warning: <img> lacks "alt" attribute
line 172 column 15 - Warning: <img> lacks "alt" attribute
line 233 column 23 - Warning: <img> lacks "alt" attribute
line 233 column 64 - Warning: <img> lacks "alt" attribute
line 244 column 15 - Warning: <img> lacks "alt" attribute
line 288 column 23 - Warning: <img> lacks "alt" attribute
line 288 column 64 - Warning: <img> lacks "alt" attribute
line 299 column 15 - Warning: <img> lacks "alt" attribute
line 333 column 23 - Warning: <img> lacks "alt" attribute
line 333 column 64 - Warning: <img> lacks "alt" attribute
line 344 column 15 - Warning: <img> lacks "alt" attribute
line 374 column 2115 - Warning: <ul> anchor "ihadbtnz" already defined
line 374 column 2323 - Warning: <li> anchor "ihadbtn" already defined
line 382 column 23 - Warning: <img> lacks "alt" attribute
line 382 column 64 - Warning: <img> lacks "alt" attribute
line 393 column 15 - Warning: <img> lacks "alt" attribute
line 400 column 1089 - Warning: <div> anchor "ihadpost" already defined
line 418 column 2220 - Warning: <h3> anchor "ihadhead" already defined
line 418 column 2265 - Warning: <div> anchor "ihadbtnz" already defined
line 418 column 2284 - Warning: <ul> anchor "ihadbtnz" already defined
line 418 column 2302 - Warning: <li> anchor "ihadbtn" already defined
line 418 column 2492 - Warning: <li> anchor "ihadbtn" already defined
line 425 column 11 - Warning: <img> lacks "alt" attribute
line 426 column 22 - Warning: <img> lacks "alt" attribute
line 426 column 63 - Warning: <img> lacks "alt" attribute
line 426 column 112 - Warning: <img> lacks "alt" attribute
line 426 column 161 - Warning: <img> lacks "alt" attribute
line 427 column 11 - Warning: <img> lacks "alt" attribute
line 437 column 15 - Warning: <img> lacks "alt" attribute
line 444 column 324 - Warning: <img> lacks "alt" attribute
line 452 column 23 - Warning: <img> lacks "alt" attribute
line 452 column 64 - Warning: <img> lacks "alt" attribute
line 463 column 15 - Warning: <img> lacks "alt" attribute
line 470 column 1089 - Warning: <div> anchor "ihadpost" already defined
line 484 column 2148 - Warning: <h3> anchor "ihadhead" already defined
line 484 column 2193 - Warning: <div> anchor "ihadbtnz" already defined
line 484 column 2212 - Warning: <ul> anchor "ihadbtnz" already defined
line 484 column 2230 - Warning: <li> anchor "ihadbtn" already defined
line 484 column 2420 - Warning: <li> anchor "ihadbtn" already defined
line 493 column 44 - Warning: <img> proprietary attribute value "absmiddle"
line 493 column 142 - Warning: <img> proprietary attribute value "absmiddle"
line 493 column 246 - Warning: <img> proprietary attribute value "absmiddle"
line 502 column 25 - Warning: <img> lacks "alt" attribute
line 507 column 267 - Warning: <img> lacks "alt" attribute
line 149 column 50 - Warning: trimming empty <font>
line 487 column 17 - Warning: trimming empty <tr>
line 490 column 50 - Warning: trimming empty <font>
line 125 column 68 - Warning: <nobr> is not approved by W3C
line 141 column 68 - Warning: <nobr> is not approved by W3C
line 177 column 27 - Warning: <nobr> is not approved by W3C
line 249 column 27 - Warning: <nobr> is not approved by W3C
line 304 column 27 - Warning: <nobr> is not approved by W3C
line 349 column 27 - Warning: <nobr> is not approved by W3C
line 398 column 27 - Warning: <nobr> is not approved by W3C
line 442 column 27 - Warning: <nobr> is not approved by W3C
line 468 column 27 - Warning: <nobr> is not approved by W3C
Info: Document content looks like HTML5
Info: No system identifier in emitted doctype
Tidy found 138 warnings and 0 errors!


The alt attribute should be used to give a short description
of an image; longer descriptions should be given with the
longdesc attribute which takes a URL linked to the description.
These measures are needed for people using non-graphical browsers.

For further advice on how to make your pages accessible
see http://www.w3.org/WAI/GL.
You are recommended to use CSS to specify the font and
properties such as its size and color. This will reduce
the size of HTML files and make them easier to maintain
compared with using <FONT> elements.

You are recommended to use CSS to control line wrapping.
Use "white-space: nowrap" to inhibit wrapping in place
of inserting <NOBR>...</NOBR> into the markup.

About HTML Tidy: https://github.com/htacg/tidy-html5
Bug reports and comments: https://github.com/htacg/tidy-html5/issues
Official mailing list: https://lists.w3.org/Archives/Public/public-htacg/
Latest HTML specification: http://dev.w3.org/html5/spec-author-view/
Validate your HTML documents: http://validator.w3.org/nu/
Lobby your company to join the W3C: http://www.w3.org/Consortium

Do you speak a language other than English, or a different variant of
English? Consider helping us to localize HTML Tidy. For details please see
https://github.com/htacg/tidy-html5/blob/master/README/LOCALIZE.md