Register - Login
Views: 87348905
Main - Memberlist - Active users - Calendar - Wiki - IRC Chat - Online users
Ranks - Rules/FAQ - JCS - Stats - Latest Posts - Color Chart - Smilies
11-19-17 12:00:55 PM

Jul - Game Research/Hacking/Modding - Etrian Odyssey hacking (EO1 & 4) New poll - New thread - New reply
Pages: 1 2 3 4Next newer thread | Next older thread
Ehm
Member
Level: 46


Posts: 461/532
EXP: 681068
For next: 30706

Since: 06-13-09
From: Canada

Since last post: 173 days
Last activity: 131 days

Posted on 10-27-14 02:04:23 PM (last edited by m64m at 10-27-14 04:00:15 PM) Link | Quote
Would definitely be cool to see how similar the file formats are. I'll have to take a look at this.

EDIT: Sound and movie clips are easily accessible, but everything else is in a custom archive named "MORI1R.HPB", at 323MB. There's an accompanying "MORI1R.HPI" table of contents file. It looks like they're still using common formats, I just can't get to them.


DICTIONARY/DICTIONARYENEMYSKILLDATA.TBL
DUNGEON/D02_F06.BCMDL
DUNGEON/OBJ07/D07_FLAGDOOR01.BAM
EFFECT/ENEMY/EG_1557C.EPL
SND/MORI1R/STREAM/BGM_TOWN_LARDA.DSPADPCM.BCSTM
VOICE/ART_SYS/ART_SYS_292.BCWAV
STEX/BATTLE/CHARADATA/IG_BAT_CHA01_01.STEX
STEX/PACKDATA/IG_KEY_HIRAGANA_STEX.BIN
TESTDATA/TEST/DUNGEON.MBM
TEXTABLE/BOOK.TTD TEXTABLE/CAMP.TTD
TRANSRATE/ICON48.CTPK


I probably missed a few formats, but this is just to show which common ones are back. Think you can work with the .HPB archive?
xdaniel
660
Level: 51


Posts: 564/669
EXP: 989812
For next: 24126

Since: 12-04-08
From: Germany

Since last post: 10 days
Last activity: 4 hours

Posted on 10-28-14 12:59:56 PM Link | Quote
Hm, I'm pretty sure I've got enough support for the .HPB/.HPI pair in Tharsis now, that it can at least correctly name and extract most files from the .HPB...

...but it can't decompress them, because I have no idea what kind of compression format they used: magic number is ACMP, followed by the compressed data size (uint), then two unknowns (both uint, first one always(?) being 0x20), then presumably the uncompressed data size (uint again), then three values I assume might be endianness markers or somesuch (all uint, each one being 0x67452301), then the compressed data itself.

I could try throwing some decompression code from Yggdrasil at them, which supports some of the common DS formats, but I'm not sure if that'll help much...

Also, as a side note, for some odd reason the .HPI also refers to files stored outside of the .HPB, such as the BCSTM streams, complete with bogus offsets and filesizes of zero. Why that's the case, I have no idea.
Ehm
Member
Level: 46


Posts: 463/532
EXP: 681068
For next: 30706

Since: 06-13-09
From: Canada

Since last post: 173 days
Last activity: 131 days

Posted on 10-29-14 08:19:43 PM Link | Quote
Wish there was something I could do to help, but this is way too out of my league.
Cuber456

Lantern Ghost
Don't mind me. Just passing through.
Level: 51


Posts: 565/754
EXP: 948857
For next: 65081

Since: 02-19-12
From: Everywhere at once.

Since last post: 9 days
Last activity: 10 hours

Posted on 10-29-14 09:22:56 PM Link | Quote
Originally posted by xdaniel
...but it can't decompress them, because I have no idea what kind of compression format they used
Compression has to ruin everything. It's bullshit.
Ehm
Member
Level: 46


Posts: 464/532
EXP: 681068
For next: 30706

Since: 06-13-09
From: Canada

Since last post: 173 days
Last activity: 131 days

Posted on 10-31-14 11:18:26 PM (last edited by m64m at 11-04-14 09:50:53 PM) Link | Quote
Somewhat related, Persona Q (which seems to use the EO engine) is also packed together in a single archive. It's everything this time though - 1.77GB in a single .CPK file.

Here is what I believe to be the archive's table of contents, or at least part of it. http://pastebin.com/sSE1kimW
Most of the common EO formats we all know and love are back.

I've only briefly viewed the archive in a hex editor, but I managed to find this little gem already:

C:/xrd713/psq/design/tutorial/t046.tga


If you're interested in taking a look at this, let me know.

------------

EDIT: The .CPK file can be extracted using this QuickBMS script: http://aluigi.altervista.org/papers/bms/cpk.bms
(Thanks Einstein95 for the script!)

It looks like the file structure is very similar to Etrian Odyssey, and a lot of the formats have returned. I'll post later if I find anything using Tharsis.

------------
EDIT 2: Majority of the game uses .bam in place of .stex, since the characters are full 3D models. Most of the .stex files in Persona Q are for visual effects and they're similar, if not identical to the ones found in Etrian Odyssey 4. Tharsis was able to convert all of them except for one, where it gave a "failed" message. I haven't checked any of the text files yet.

------------
EDIT 3: I decided to take a look at Persona 3 Portable. It also uses .CPK archives, and it features a bunch of common Atlus formats. Graphics are stored in two ways: .BIN archives or .TMX format images. The .TMX images are partially compatible with Tharsis:


------------
EDIT 4: I was using the wrong version of Tharsis (0.2) for the conversion above. Here's what that image looks like with the latest release:
Askedforthis
Random nobody
Level: 3


Posts: 1/2
EXP: 94
For next: 34

Since: 11-05-14


Since last post: 3.0 years
Last activity: 3.0 years

Posted on 11-05-14 07:33:13 PM Link | Quote
Extremely interesting thread. I just now came over it after decrypting EO Untold (both the Japanese and English release) in hopes of getting it undubbed. Unfortunately that's impossible without unpacking the HPB file (and probably re-packing it as well) as there is clearly a "voice" folder inside of it by looking at the hex text. I wonder why they'd want to compress this in such a special way, they didn't with the Japanese Persona Q release.

I'm counting on you xdaniel to unpack it! Seems like you guys are the only ones with the knowledge and the skills to pull this off.
xdaniel
660
Level: 51


Posts: 565/669
EXP: 989812
For next: 24126

Since: 12-04-08
From: Germany

Since last post: 10 days
Last activity: 4 hours

Posted on 11-06-14 05:35:15 AM Link | Quote
m64m: I'd be interested in some of those Persona 3 .TMX files. Didn't really pay much attention to that format before, because of how it isn't used very much in EO4, but seeing that portrait from Persona I'd like to try and fix the conversion.

Askedforthis: If it's mainly the voice files you're after, those actually appear to be uncompressed inside the .HPB archive and seem to be normal BCWAVs... although 3DSExplorer doesn't seem to convert them correctly, as all I can hear is slience. Anyway, you'll probably still need to rebuild the .HPB and .HPI, tho, as I'd imagine it's pretty unlikely for the Japanese and English voice files to have the same length.

I'll have a look into rebuilding the .HPB/.HPI files, probably over the next couple of days. Not promising any fast results tho, seeing how there's still a bunch of data in there that I don't understand yet

---

Also, EO4 and EOU aside, I had gotten some older R4i Advance flash cartridge for the DS at a flea market last weekend. That means I was finally able to check if the modifications Yggdrasil does to EO1's ARM9 binary - that is, mainly changing filenames - actually work on a real DS... and yes, my little custom "Yggdrasil Sword" works quite well on hardware!
einstein95
Member
Level: 33


Posts: 159/305
EXP: 218552
For next: 10627

Since: 04-11-13


Since last post: 62 days
Last activity: 1 day

Posted on 11-07-14 06:27:07 AM Link | Quote
Originally posted by xdaniel
Askedforthis: If it's mainly the voice files you're after, those actually appear to be uncompressed inside the .HPB archive and seem to be normal BCWAVs... although 3DSExplorer doesn't seem to convert them correctly, as all I can hear is slience.


Use vgmstream to convert bcwavs, 3DSExplorer's seems to just be a dummy function.
Askedforthis
Random nobody
Level: 3


Posts: 2/2
EXP: 94
For next: 34

Since: 11-05-14


Since last post: 3.0 years
Last activity: 3.0 years

Posted on 11-08-14 07:30:54 PM Link | Quote
Originally posted by xdaniel

Askedforthis: If it's mainly the voice files you're after, those actually appear to be uncompressed inside the .HPB archive and seem to be normal BCWAVs... although 3DSExplorer doesn't seem to convert them correctly, as all I can hear is slience. Anyway, you'll probably still need to rebuild the .HPB and .HPI, tho, as I'd imagine it's pretty unlikely for the Japanese and English voice files to have the same length.

I'll have a look into rebuilding the .HPB/.HPI files, probably over the next couple of days. Not promising any fast results tho, seeing how there's still a bunch of data in there that I don't understand yet



That sounds fantastic that they are possible to get out and maybe replace as well. As long as the voices can be extracted, played and put back into the HPB it would most likely work.

Thanks a lot for taking a look at it. Take your time, there's no hurry!
Ehm
Member
Level: 46


Posts: 466/532
EXP: 681068
For next: 30706

Since: 06-13-09
From: Canada

Since last post: 173 days
Last activity: 131 days

Posted on 11-09-14 07:23:03 PM Link | Quote
Looks like Tharsis is almost natively compatible with Shin Megami Tensei IV's graphics.

A lot of them are stored in .stex format, but there are a good amount as .cmp files. The .cmp files are LZ11 compressed .stex files. Here's an example of one I decompressed and ran through Tharsis:


There were a few .tmx files in a test folder. I haven't done an in-depth search of the RomFS yet, so I don't know if there are any others. Our old friend from P4 is back, looking a little worse for the wear:

xdaniel
660
Level: 51


Posts: 566/669
EXP: 989812
For next: 24126

Since: 12-04-08
From: Germany

Since last post: 10 days
Last activity: 4 hours

Posted on 11-10-14 07:02:00 AM (last edited by xdaniel at 11-10-14 12:52:30 PM) Link | Quote


Tharsis 0.4 is here and the changes are the following:

* Preliminary support for extracting .HPB/.HPI archives (Etrian Odyssey Untold)
-- Does not support decompressing files contained within the archive; format unknown
* Much improved .TMX image support (see image above)

The problem with the .TMX files was, basically, that I didn't convert the palette correctly. I originally assumed it's just straight up RGBA, 8 bits per channel, one color per index, and while in reality it's close to that, it isn't exactly... The color format is RGBA8888, but 1) the palette is sort-of tiled, with the first 32 entries in the palette being the colors for indices 0-7, then 16-23, then 8-15, and then 24-31, and 2) the alpha value has to be scaled somehow.

However, figuring that out is not to my credit at all, but it's rather all thanks to this thread on the PCSX2 forums as well as this one on Xentax. I ended up googling Persona and TMX, and as it turns out, this format has been used by Atlus in a bunch of games, mostly Persona, as far back as Persona 3 on the PS2 - and it's been cracked quite a while ago.

But regardless, I implemented what I had learned in Tharsis, and lo and behold, TMXs started converting properly!

As for repacking .HPB/.HPI archives, I haven't really worked on that yet, but it's on the proverbial to-do list; the compressed STEXs I'll be looking into soon as well.

Edit! Put both Tharsis and the ETC1 library on GitHub:

https://github.com/xdanieldzd/Tharsis/
https://github.com/xdanieldzd/ETC1Lib/
Ehm
Member
Level: 46


Posts: 467/532
EXP: 681068
For next: 30706

Since: 06-13-09
From: Canada

Since last post: 173 days
Last activity: 131 days

Posted on 11-10-14 04:37:13 PM (last edited by m64m at 11-10-14 04:39:02 PM) Link | Quote
Another excellent update!

Very cool to see the TMX format fixed! There's one test file I'm not sure of though. It shows up in Etrian Odyssey 4, Untold, and Persona Q. Below are two rips, the first from Tharsis 0.3 and the second from Tharsis 0.4. Do you think the second rip is fully accurate? I'm not sure if the specs are supposed to be part of the image.



Other than that, TMX is pretty much perfect. One file in Persona 3 Portable (umd0.cpk\title\logo.tmx) had some extra data before the TMX header, so Tharsis crashed. Removing the extra data let it convert flawlessly.



For Untold, not all of the files inside the HPB archive are compressed. All BCWAV files (except for MORI1R.HPB\TRANSRATE\MORI4BANNERSND.BCWAV) are uncompressed and can be converted with vgmstream, as Einstein95 said earlier.
xdaniel
660
Level: 51


Posts: 567/669
EXP: 989812
For next: 24126

Since: 12-04-08
From: Germany

Since last post: 10 days
Last activity: 4 hours

Posted on 11-10-14 06:04:45 PM Link | Quote
I noticed that problem as well, and I have a feeling that file might in fact be slightly corrupted...

The second 32-bit value in a TMX, i.e. from offset 0x04 to 0x08, holds the total file size, in this file's case (the one from EO4, to be exact) 0x10440 bytes - however, the file is actually 0x1052A bytes long, thus 0xEA bytes too long. For one, there's something going on after offset 0x210, which is right inside the palette data and where the color values stop "lining up" properly. Compare ex. to be_0001.tmx from EO4's \romfs\effect\tex, where the unused palette entries are all set to black with an alpha of 128...

...hey, wait a second...

*looks over the messed up palette data*

...ha-haaa, someone's dun goofed at Atlus. Go ahead and use a hex editor to replace every occurance of 0x0D0A inside the file with 0x0A. Notice how the actual filesize now matches the one given in the TMX header? Put that file through Tharsis now.



0x0D and 0x0A, carriage return and line feed... Windows and Unix newlines, resulting in messed up graphics... Where have I seen something like this before?

As for the P3P Atlus logo file, mind sending that one over? Even though you say it's only one file with extra data, I'd like to have Tharsis not outright crash when encountering something unusual like that.

Finally, as for the HPB archive, I had noticed that in the post einstein95 quoted, although I haven't tried vgmstream yet
Ehm
Member
Level: 46


Posts: 468/532
EXP: 681068
For next: 30706

Since: 06-13-09
From: Canada

Since last post: 173 days
Last activity: 131 days

Posted on 11-10-14 06:13:47 PM Link | Quote
Interesting. Well, if I happen to see something like this again, I'll let you know

PM sent for that Atlus logo.
Xkeeper






Posted on 11-10-14 06:29:59 PM Link | Quote
Originally posted by xdaniel
I noticed that problem as well, and I have a feeling that file might in fact be slightly corrupted...

The second 32-bit value in a TMX, i.e. from offset 0x04 to 0x08, holds the total file size, in this file's case (the one from EO4, to be exact) 0x10440 bytes - however, the file is actually 0x1052A bytes long, thus 0xEA bytes too long. For one, there's something going on after offset 0x210, which is right inside the palette data and where the color values stop "lining up" properly. Compare ex. to be_0001.tmx from EO4's \romfs\effect\tex, where the unused palette entries are all set to black with an alpha of 128...

...hey, wait a second...

*looks over the messed up palette data*

...ha-haaa, someone's dun goofed at Atlus. Go ahead and use a hex editor to replace every occurance of 0x0D0A inside the file with 0x0A. Notice how the actual filesize now matches the one given in the TMX header? Put that file through Tharsis now.



0x0D and 0x0A, carriage return and line feed... Windows and Unix newlines, resulting in messed up graphics... Where have I seen something like this before?

As for the P3P Atlus logo file, mind sending that one over? Even though you say it's only one file with extra data, I'd like to have Tharsis not outright crash when encountering something unusual like that.

Finally, as for the HPB archive, I had noticed that in the post einstein95 quoted, although I haven't tried vgmstream yet


I wonder if this is a sign of them using FTP or Git or something -- both are fairly common tools that have the well-known capability to mangle newlines like that.

Still pretty weird to see, though
Ehm
Member
Level: 46


Posts: 469/532
EXP: 681068
For next: 30706

Since: 06-13-09
From: Canada

Since last post: 173 days
Last activity: 131 days

Posted on 11-11-14 10:58:12 PM Link | Quote
I'm in the process of doing a full graphics and text dump for Shin Megami Tensei IV. I've come across about 40-45 .stex images that fail to convert with Tharsis, and there will probably be more. Should I send you a few of them for analysis?
xdaniel
660
Level: 51


Posts: 568/669
EXP: 989812
For next: 24126

Since: 12-04-08
From: Germany

Since last post: 10 days
Last activity: 4 hours

Posted on 11-11-14 11:54:06 PM (last edited by xdaniel at 11-13-14 11:59:06 AM) Link | Quote
Sure; the more (= test cases), the merrier (= better the program). Or something like that.

I do know STEX support is ex. missing some image formats, but as I don't have STEXs that use those formats, I couldn't really try to implement them just yet. Would be a hunch as to why those 40-45 you've got there might not convert, but it could also be something else entirely... Yeah, just send a few of them over, I'll see what I can do.

Edit, yep, it was missing image/pixel formats in Tharsis:

2D_CharID170_03.stex:


button_help.stex:


menu_icon001.stex:


Will commit the code changes to GitHub in a bit and post a new build tomorrow or over the weekend, depending on if I get another feature in, the LZ11-compressed STEXs.
xdaniel
660
Level: 51


Posts: 569/669
EXP: 989812
For next: 24126

Since: 12-04-08
From: Germany

Since last post: 10 days
Last activity: 4 hours

Posted on 11-16-14 08:25:26 PM Link | Quote
Sorry for the quick double-post, just wanted to note that I just posted Tharsis 0.5 to GitHub. It's the usual release build, only without source, as that's now on GitHub anyway.
Ehm
Member
Level: 46


Posts: 472/532
EXP: 681068
For next: 30706

Since: 06-13-09
From: Canada

Since last post: 173 days
Last activity: 131 days

Posted on 11-19-14 10:59:34 PM Link | Quote
Awesome! There were still a very small amount of .stex images that didn't convert, but I think they may be corrupted.
xdaniel
660
Level: 51


Posts: 570/669
EXP: 989812
For next: 24126

Since: 12-04-08
From: Germany

Since last post: 10 days
Last activity: 4 hours

Posted on 11-19-14 11:14:10 PM Link | Quote
Well, there's still a few unsupported/unverified pixel formats, so it's certainly possible that the STEXs left use them. If you could send those over, too, I could take a look and - if it really is the pixel formats again - probably add support for them. But regardless of that, I should also look into making Tharsis' error handling a bit more descriptive, instead of just telling the user that something "failed"
Pages: 1 2 3 4Next newer thread | Next older thread
Jul - Game Research/Hacking/Modding - Etrian Odyssey hacking (EO1 & 4) New poll - New thread - New reply




Rusted Logic

Acmlmboard - commit 2f1bc75 [2017-08-27]
©2000-2017 Acmlm, Xkeeper, Inuyasha, et al.

29 database queries, 8 query cache hits.
Query execution time: 0.152811 seconds
Script execution time: 0.020337 seconds
Total render time: 0.173148 seconds