Jul -- The Cutting Room Floor: Can anyone look at these LZSS-compressed files?

Vague Rant
User
Level: 8

Posts: 1/10
EXP: 1867
For next: 320

Since: 10-05-12

Since last post: 8.7 years
Last activity: 4.2 years

Posted on 10-05-12 06:40:41 PM

Link | Quote

I'm looking into Babysitting Party (Imagine Party Babyz in the US), a party/minigame collection on Wii, but most of the game's files are compressed with some LZSS variant I haven't found any tools to extract. They all appear to begin with the hex sequence 21436587, and a batch file left on the disc shows that they were compressed with a tool called clzss:

For /F %%i in ('dir /A:A /O:S /S /B babyz\*.*') do call k:\bin\clzss %%i


xcopy K:\game_data\wii_data\music\*.* C:\RVL_SDK\dvddata\babyz\music /s /e


xcopy K:\game_data\wii_data\menu\legals\*.thp C:\RVL_SDK\dvddata\babyz\menu\legals\ /s /e


xcopy K:\game_data\wii_data\menu\*.tpl C:\RVL_SDK\dvddata\babyz\menu\ /s /e

Here's a sample, including a .c, .h and a .bmp:
http://www.smallfiles.org/download/2608/lzss.zip.html

If anyone knows of a tool that can extract these, that'd be great, otherwise, thanks for reading.

zoinkity

Level: 13

Posts: 3/27
EXP: 8319
For next: 1948

Since: 09-07-12

Since last post: 4.6 years
Last activity: 4.5 years

Posted on 10-05-12 10:16:23 PM

Link | Quote

It's traditional LZSS with a header on it:
0x0 4 ID: 0x21436587
0x4 4 compressed size (little endian)
0x8 4 decompressed size (little endian)
0xC var compressed data

You can use the original LZSS.exe by Haruhiko Okumura. Just clip off the first 0x8 bytes, since his expects only the decompressed size and data.

(The tabs got flattened by this message board's parser, but you'll see well enough when you get them decompressed yourself.)

The .h file:

// Items : Dummies, Trigzones, Splines, Cameras
// GENERATED FILE, MODIFY AT YOUR OWN RISK.

#ifndef ZED_ITEMS
#define ZED_ITEMS
typedef struct { float x,y,z; } vector;
typedef struct { float x,y,z, r; } sphere;
typedef struct { float Xx,Yx,Zx, x, Xy,Yy,Zy, y, Xz,Yz,Zz, z; } matrix;

typedef struct { vector pos, tan; float curve_pos; } control_point;
typedef struct { control_point* cpoints; float nbcp; } spline;
#endif

// DUMMIES

enum TRON_DUMMY
{
TRON_DUM_MNG_TRON,
TRON_DUM_BABY1,
TRON_DUM_BABY2,
TRON_DUM_BABY3,
TRON_DUM_BABY4,
TRON_DUM_HAUTGAUCHE,
TRON_DUM_BASDROITE,
TRON_NB_DUMMIES
};

extern matrix tron_dum[TRON_NB_DUMMIES];

// CAMERAS

enum TRON_CAMERA
{
TRON_CAM_TRON,
TRON_NB_CAMERAS
};

extern matrix tron_cam[TRON_NB_CAMERAS];

The .c file:

// Items : Dummies, Trigzones, Splines, Cameras
// GENERATED FILE, MODIFY AT YOUR OWN RISK.

// Matrices are row major.
// XYZ -> -XZY. HAVE FUN.

#include ""

// DUMMIES

matrix tron_dum[TRON_NB_DUMMIES] =
{
{ // TRON_DUM_MNG_TRON
1, 0, 0, 0.440365,
0, 1, 0, 0.0383208,
0, 0, 1, 3.00757
},
{ // TRON_DUM_BABY1
1, 0, 0, 2.62759,
0, 1, 0, -0.0113605,
0, 0, 1, 4.77104
},
{ // TRON_DUM_BABY2
1, 0, 0, -1.71902,
0, 1, 0, 0,
0, 0, 1, 4.77078
},
{ // TRON_DUM_BABY3
-0.99945, 0, 0.0331614, -1.73066,
0, 1, 0, 0,
-0.0331614, 0, -0.99945, 1.24962
},
{ // TRON_DUM_BABY4
-0.999999, 0, 0.0011544, 2.57853,
0, 1, 0, 0,
-0.0011544, 0, -0.999999, 1.30426
},
{ // TRON_DUM_HAUTGAUCHE
1, 0, 0, 3.31478,
0, 1, 0, -0.0214868,
0, 0, 1, 5.33592
},
{ // TRON_DUM_BASDROITE
1, 0, 0, -2.43594,
0, 1, 0, 0,
0, 0, 1, 0.681589
},
};

// CAMERAS

matrix tron_cam[TRON_NB_CAMERAS] =
{
{ // TRON_CAM_TRON
-1, -2.03576e-008, 1.85891e-008, 0.490089,
0, 0.674304, 0.738454, 4.47602,
-2.75679e-008, 0.738454, -0.674304, -0.646258
},
};

The third is a bitmap file, which is sadly too large to post here.

Joe
Common spammer
🍬
Level: 111

Posts: 2679/3392
EXP: 14488840
For next: 379520

Since: 08-02-07

From: Pororoca

Since last post: 3 days
Last activity: 1 hour

Posted on 10-05-12 10:37:16 PM

Link | Quote

Originally posted by zoinkity
The third is a bitmap file, which is sadly too large to post here.

Upload it somewhere (e.g. mediafire), PM the download link to me, and I'll post it.

____________________

GuyPerfect
Catgirl
Level: 68

Posts: 789/1096
EXP: 2663411
For next: 65389

Since: 07-23-07

Since last post: 1.6 years
Last activity: 210 days

Posted on 10-05-12 10:40:41 PM (last edited by GuyPerfect at 10-05-12 10:41:43 PM)

Link | Quote

EDIT:

Aw, man! You guys ninja'd me all up in the head!

Original post is below.

I swear, every time I crack a compression algorithm, it works on some principle I've never heard of before. Today's magical methodology? The circular buffer.

Here's the contents of the .h file:

// Items : Dummies, Trigzones, Splines, Cameras

// GENERATED FILE, MODIFY AT YOUR OWN RISK.


#ifndef ZED_ITEMS

#define ZED_ITEMS

typedef struct { float x,y,z; } vector;

typedef struct { float x,y,z, r; } sphere;

typedef struct { float Xx,Yx,Zx, x, Xy,Yy,Zy, y, Xz,Yz,Zz, z; } matrix;


typedef struct { vector pos, tan; float curve_pos; } control_point;

typedef struct { control_point* cpoints; float nbcp; } spline;

#endif


// DUMMIES


enum TRON_DUMMY

{

       TRON_DUM_MNG_TRON,

  TRON_DUM_BABY1,

     TRON_DUM_BABY2,

     TRON_DUM_BABY3,

     TRON_DUM_BABY4,

     TRON_DUM_HAUTGAUCHE,

        TRON_DUM_BASDROITE,

 TRON_NB_DUMMIES

};


extern matrix tron_dum[TRON_NB_DUMMIES];



// CAMERAS


enum TRON_CAMERA

{

    TRON_CAM_TRON,

      TRON_NB_CAMERAS

};


extern matrix tron_cam[TRON_NB_CAMERAS];

And of the .c file:

// Items : Dummies, Trigzones, Splines, Cameras

// GENERATED FILE, MODIFY AT YOUR OWN RISK.


// Matrices are row major.

// XYZ -> -XZY. HAVE FUN.


#include ""



// DUMMIES


matrix tron_dum[TRON_NB_DUMMIES] =

{

     {       // TRON_DUM_MNG_TRON

          1, 0, 0, 0.440365,

          0, 1, 0, 0.0383208,

         0, 0, 1, 3.00757

  },

  {       // TRON_DUM_BABY1

     1, 0, 0, 2.62759,

   0, 1, 0, -0.0113605,

        0, 0, 1, 4.77104

  },

  {       // TRON_DUM_BABY2

     1, 0, 0, -1.71902,

          0, 1, 0, 0,

         0, 0, 1, 4.77078

  },

  {       // TRON_DUM_BABY3

     -0.99945, 0, 0.0331614, -1.73066,

   0, 1, 0, 0,

         -0.0331614, 0, -0.99945, 1.24962

  },

  {       // TRON_DUM_BABY4

     -0.999999, 0, 0.0011544, 2.57853,

   0, 1, 0, 0,

         -0.0011544, 0, -0.999999, 1.30426

 },

  {       // TRON_DUM_HAUTGAUCHE

        1, 0, 0, 3.31478,

   0, 1, 0, -0.0214868,

        0, 0, 1, 5.33592

  },

  {       // TRON_DUM_BASDROITE

         1, 0, 0, -2.43594,

          0, 1, 0, 0,

         0, 0, 1, 0.681589

 },

};



// CAMERAS


matrix tron_cam[TRON_NB_CAMERAS] =

{

       {       // TRON_CAM_TRON

      -1, -2.03576e-008, 1.85891e-008, 0.490089,

          0, 0.674304, 0.738454, 4.47602,

     -2.75679e-008, 0.738454, -0.674304, -0.646258

     },

};

And biberon_anim.BMP:

The file format of these compressed files is pretty simple:

Fields are read little-endian


File Structure

==============================================================

MagicNumber     UInt32    4    Must be 0x87654321

PackedSize      UInt32    4    Size of compressed data in file

UnpackedSize    UInt32    4    Size of decompressed data

PackedData      Byte[]    *    PackedSize bytes of data

==============================================================


PackedData is a stream of packed LZSS chunks.


Each chunk begins with a single byte where each bit, starting with the lowest,

represents one data unit to decode.


  1 - If the flag is set, then copy one byte from the input to the output.


  0 - If the flag is cleared, then copy two bytes to process an LZSS string.


The decoder has a 4096-byte circular buffer that begins at byte 0xFEE. When

bytes are written to the output via either unit type, they are also written to

the circlar buffer. After a byte at location 0xFFF is written, the next byte

will be at 0x000, then at 0x001, and so-on.


The "distance" value of the distance/length pair actually specifies a location

within the circular buffer. It is not a typical distance value seen in ordinary

LZ specifications.


When processing a flag with a value of 0, two bytes are loaded.


  * The first byte is the lower 8 bits of the "distance" value.


  * The second byte contains the highest 4 bits of "distance", and the length.


Bits 4-7 of the second byte, which are the higher four bits, actually specify

bits 8-11 of the "distance" value, meaning the next four bits above 255.

Consider both bytes, read in order, and displayed as their component bits:


  aaaaaaaa bbbbcccc


For such a pair of bytes, the "distance" value is composited as bbbbaaaaaaaa.

The length value, then, becomes only cccc.


Lengths are biased by 3, so add 3 to the processed value to retrieve the actual

length.


When processing the "distance"/length pair, start in the circular buffer at the

location specified by "distance", and from there copy length bytes into both

the output as well as the circular buffer itself from the last-written

position.

While investigating the format, I discovered what I believe to be the original source files used in the very "clzss" utility used to compress the files in the first place. They look to be a public domain of a particular LZSS implementation, structured as a C++ class definition:

• LZSS.h
• LZSS.cpp

These source files were especially helpful, because they follow exactly the format used by the Babysitting Party resources, at least in terms of the packed LZSS streams.

After some personal investigation and dozens of minutes programming, I've produced a C source file that compiles directly to a utility that will decompress these Babysitting Party files.

I'm calling the program "debabyz" over here, but that's not set in stone. It's a command line utility with the following usage:

debabyz <infile> <outfile>
debabyz.c:

#include <stdio.h>

#include <stdlib.h>


// Loads a file into a buffer with optional size limitations

int LoadFile(char *filename, unsigned char **data, int minsize, int maxsize) {

    FILE *file = NULL;

    int len, err;


    // Error checking

    if (data != NULL) *data = NULL;

    if (filename == NULL || data == NULL) goto catch;


    // Open the file

    file = fopen(filename, "rb");

    if (file == NULL) goto catch;


    // Get the file's size, throwing an error if it's too big or too small

    fseek(file, 0, SEEK_END);

    len = ftell(file);

    if (len < minsize || (maxsize && len > maxsize) )

        goto catch;


    // Load the file data

    *data = malloc(len);

    if (*data == NULL) goto catch;

    fseek(file, 0, SEEK_SET);

    err = fread(*data, 1, len, file);

    if (err != len) goto catch;


    // Close the file and return its length

    fclose(file);

    return len;


catch: // Error handling. Don't judge; you need goto for try/catch in C.


    // Cleanup local resources

    if (file  != NULL)

        fclose(file);

    if (data != NULL && *data != NULL)

        { free(*data); *data = NULL; }


    // Return an error code

    return -1;

}


// Inline function for extracting a 32-bit integer from a byte buffer

#define GetInt32(data, offset) (       \

    ( (int) data[offset + 3] << 24 ) | \

    ( (int) data[offset + 2] << 16 ) | \

    ( (int) data[offset + 1] <<  8 ) | \

      (int) data[offset]               \

)


// Program entry point

int main(int argc, char **argv) {

    int fLen, magic, size_p, size_u, offset_i, offset_o, offset_c;

    int flags, distance, length, c;

    unsigned char *fData, *oData, circle[0x1000];

    FILE *file;


    // Check the input parameters

    if (argc != 3) {

        printf("Usage: %s <infile> <outfile>\n", argv[0]);

        return 1;

    }


    // Load the file

    fLen = LoadFile(argv[1], &fData, 12, 0); // Must be at least 12 bytes

    if (fLen == -1) {

        printf("ERROR: There was a problem loading %s\n", argv[1]);

        return 2;

    }


    // Read the file header

    magic  = GetInt32(fData, 0);

    size_p = GetInt32(fData, 4);

    size_u = GetInt32(fData, 8);

    if (magic != 0x87654321 || size_p > fLen - 12) {

        printf("ERROR: The file does not appear to be supported.\n");

        free(fData); // Delete the file buffer

        return 3;

    }


    // Prepare an output buffer

    oData = malloc(size_u);

    if (oData == NULL) {

        printf("ERROR: Not enough memory for output buffer.\n");

        free(fData); // Delete the file buffer

        return 4;

    }


    // Decode the file

    offset_i = 12;    // Offset within the input buffer

    offset_o = 0;     // Offset within the output buffer

    offset_c = 0xFEE; // Starting point in the circular buffer

    flags    = 0;     // Processing flags for raw/distance+length values


    // Loop through all input bytes

    for ( ; ; ) {


        // Read a new flags byte if necessary

        if (!(flags & 0x100)) {

            // Prevent a buffer overrun

            if (offset_i >= fLen) break;


            // I learned this trick from the reference implementation.

            // Flags will be > 255 if we have processed fewer than 8 bits.

            flags = 0xFF00 | (int) fData[offset_i++];

        }


        // Decode a raw byte

        if (flags & 1) {


            // Prevent a buffer overrun

            if (offset_i >= fLen || offset_o >= size_u)

                break;


            // Copy the next byte from input to output

            oData[offset_o]    = fData[offset_i++];

            circle[offset_c++] = oData[offset_o++];

            offset_c &= 0xFFF; // Clamp to buffer range

        }


        // Decode a "distance"/length pair

        else {


            // Prevent a buffer overrun

            if (offset_i + 1 >= fLen) break;


            // Process "distance" and length values

            // Distance is actually the offset in the circular buffer

            // Bits 4-7 of length are actually bits 8-11 of distance

            distance  = fData[offset_i++];

            length    = fData[offset_i++];

            distance |= ( (length & 0xF0) << 4);

            length    =   (length & 0x0F)  + 3;


            // Prevent a buffer overflow

            if (offset_o + length > size_u) break;


            // Copy bytes from the circular buffer to the output

            for (c = 0 ; c < length; c++) {

                oData[offset_o]    = circle[(distance + c) & 0xFFF];

                circle[offset_c++] = oData[offset_o++];

                offset_c &= 0xFFF; // Clamp to buffer range

            }

        }


        // Move to next flag

        flags >>= 1;


    } // for loop


    // Save the decoded file data

    file = fopen(argv[2], "wb");

    if (file != NULL) {

        fwrite(oData, 1, size_u, file);

        fclose(file);

    } else printf("ERROR: Could not open output file %s\n", argv[2]);


    // Indicate if a decoding error may have occurred

    if (offset_o != size_u)

        printf("WARNING: The data may have contained errors.\n");


    // Cleanup and exit

    free(fData);

    free(oData);

    return 0;

}

Vague Rant, if you are able to compile that and get it to work on your own, perfect. If not, post a reply here in this thread and I'll see about getting you an EXE version of it.

Ninji

Birdo
Why did my user title say I'm a toaster anyway
Level: 88

Posts: 1916/2014
EXP: 6632941
For next: 17723

Since: 07-26-07

Pronouns: he/him or they/them
From: Glasgow, Scotland

Since last post: 104 days
Last activity: 17 days

Posted on 10-05-12 10:46:27 PM (last edited by Treeki at 10-05-12 10:47:23 PM)

Link | Quote

I shall join the club! A little late, but technically, I did this first (a couple of hours ago) :p I posted it in #tcrf, though, not on the board.

Here's my Python implementation. I started out figuring it myself from the data based on what I knew about LZ, but then had to resort to disassembling the game code to figure out how the distance copies worked.

import sys, struct


filename = sys.argv[1]

decomp_filename = filename+'_decomp'


in_file = open(filename, 'rb')

out_file = open(decomp_filename, 'wb')


header = in_file.read(12)

magic, comp_size, decomp_size = struct.unpack('<III', header)


#print('Magic: 0x%08x; Compressed size: 0x%x(%d) bytes;

#Decompressed size: 0x%x(%d) bytes' % (magic,comp_size,

#comp_size,decomp_size,decomp_size))


u8 = struct.Struct('<B')


rolling_buffer = []

init_char = ' '

for i in xrange(0x1010):

       rolling_buffer.append(init_char)

rolling_index = 0xFEE


read_count = 0

while read_count < comp_size:

 flags = u8.unpack(in_file.read(1))[0]

       read_count += 1


 for i in xrange(8):

         if (flags & 1) == 1:

                        byte = in_file.read(1)

                      out_file.write(byte)

                        rolling_buffer[rolling_index] = byte

                        rolling_index = (rolling_index + 1) & 0xFFF

                 read_count += 1

             else:

                       cparam1,cparam2 = struct.unpack('<BB', in_file.read(2))

                  read_count += 2

                     #print('Compressed block @ %x (%x) : %x,%x' %

                       #(read_count, read_count+12, cparam1, cparam2))


                 # I hope this is right, need to double-check the code that reads it

                 copy_count = (cparam2 & 0xF) + 3


                        # final copy

                        copy_offset = cparam1 | ((cparam2 & 0xF0) << 4)

                       for j in xrange(copy_count):

                                byte = rolling_buffer[copy_offset & 0xFFF]

                          copy_offset += 1

                            rolling_buffer[rolling_index] = byte

                                rolling_index = (rolling_index + 1) & 0xFFF

                         out_file.write(byte)


            flags >>= 1

         if read_count >= comp_size:

                 break

And yeah, the decompression assembly does use that 0xFF00 thing too - I didn't do it that way in my code because by the time I got around to disassembling the original, I had already written the flag byte handling code.

Edit: Code edited slightly (Just the print statements) so it doesn't stretch the tables.

____________________

GuyPerfect
Catgirl
Level: 68

Posts: 790/1096
EXP: 2663411
For next: 65389

Since: 07-23-07

Since last post: 1.6 years
Last activity: 210 days

Posted on 10-05-12 10:50:19 PM

Link | Quote

Originally posted by Treeki
And yeah, the decompression assembly does use that 0xFF00 thing too - I didn't do it that way in my code because by the time I got around to disassembling the original, I had already written the flag byte handling code.

So many hackers independently converging on the same thing all at once... It's historic!

Kazinsal

Level: 53

Posts: 428/674
EXP: 1122202
For next: 34917

Since: 01-19-11

Pronouns: he/him
From: Vancouver, Canada

Since last post: 232 days
Last activity: 191 days

Posted on 10-05-12 11:21:26 PM

Link | Quote

Originally posted by GuyPerfect
So many hackers independently converging on the same thing all at once... It's historic!

Nah, it's just Jul.

____________________

Originally posted by Girlydragon
I entered some weird Touhou chat yesterday, and introduced myself with.
"Hello I am a massive *explicit removed*"
The response I got was simple.
"Hi there Aioli the massive *explicit removed*"

Vague Rant
User
Level: 8

Posts: 2/10
EXP: 1867
For next: 320

Since: 10-05-12

Since last post: 8.7 years
Last activity: 4.2 years

Posted on 10-06-12 06:30:02 AM

Link | Quote

Thanks to everyone for the rapid and comprehensive help; I really wasn't expecting it, and especially on a "shovelware-style" Wii game (although the game actually is pretty good: 7.5 review). It's very much appreciated. The disc is drowning in those .c and .h files, they don't seem especially interesting but I've started work on a wiki article and I'll probably throw them up on a subpage for anyone who wants to see them. There's also a couple of log files, some batch files and a fairly large number of bitmaps, most of which are only unused at a stretch; just pre-conversion assets that got missed in the cleanup. Hopefully some of it's worth looking at for others, too, so I'll get back to the wiki. Thanks again!

Register - Login
Views: 99384753	Main - Memberlist - Active users - Calendar - Wiki - IRC Chat - Online users Ranks - Rules/FAQ - Stats - Latest Posts - Color Chart - Smilies	04-24-22 03:24:38 AM

Query execution time:	0.082337 seconds
Script execution time:	0.022245 seconds
Total render time:	0.104582 seconds