fix some edge cases in gci decryption

README.md

@@ -33,7 +33,6 @@ newserv supports several versions of PSO. Specifically:
 This project is primarily for my own nostalgia; I offer no guarantees on how or when this project will advance. With that said, feel free to submit GitHub issues if you find bugs or have feature requests. I'd like to make the server as stable and complete as possible, but I can't promise that I'll respond to issues in a timely manner.
 
 Current known issues / missing features:
-- Partially-encrypted GCI quest files probably don't work. Fix this. Also check that completely unencrypted GCI quest files actually work.
 - Support disconnect hooks to clean up state, like if a client disconnects during quest loading or a trade window execution.
 - Episode 3 battles aren't implemented.
 - PSOBB is not well-tested and likely will disconnect or misbehave when clients try to use unimplemented features.
@@ -72,14 +71,24 @@ Standard quest files should be named like `q###-CATEGORY-VERSION.EXT`, battle qu
 - `###`: quest number (this doesn't really matter; it should just be unique for the PSO version)
 - `CATEGORY`: ret = Retrieval, ext = Extermination, evt = Events, shp = Shops, vr = VR, twr = Tower, gov = Government (BB only), dl = Download (these don't appear during online play), 1p = Solo (BB only)
 - `VERSION`: d1 = Dreamcast v1, dc = Dreamcast v2, pc = PC, gc = GameCube Episodes 1 & 2, gc3 = Episode 3, bb = Blue Burst
-- `EXT`: file extension (bin, dat, bind, datd, bin.gci, dat.gci, bin.dlq, dat.dlq, or qst)
+- `EXT`: file extension (see table below)
 
-There are multiple PSO quest formats out there; newserv supports most of them. Specifically, newserv can use quests in any of the following formats:
-- Compressed bin/dat format: These quests consist of two files with the same base name, a .bin file and a .dat file. (This is the format you'll get if you saved a quest with set-save-files.)
-- Uncompressed bin/dat format: These quests consist of two files with the same base name, a .bind file and a .datd file.
-- Unencrypted GCI format: These quests also consist of a .bin and .dat file, but an encoding is applied on top of them. The filenames should end in .bin.gci and .dat.gci. Note that there also exists an encrypted GCI format, which newserv does not support at runtime, but you can also use newserv to convert these files to bin/dat format and then use them with the server. Run `newserv --help` and see the `--decode-gci` option for more information.
-- Encrypted DLQ format: These quests also consist of a .bin and .dat file, but download quest encryption is applied on top of them. The filenames should end in .bin.dlq and .dat.dlq.
-- QST format: These quests consist of only a .qst file, which contains both the .bin and .dat files within it.
+There are multiple PSO quest formats out there; newserv supports most of them. Specifically:
+
+| Format                    | Extension         | Supported online? | Offline decode option     |
+|---------------------------|-------------------|-------------------|---------------------------|
+| Compressed                | .bin/.dat         | Yes               | None (1)                  |
+| Uncompressed              | .bind/.datd       | Yes               | None (2)                  |
+| Unencrypted GCI           | .bin.gci/.dat.gci | Yes               | --decode-gci=FILENAME     |
+| Encrypted GCI with key    | .bin.gci/.dat.gci | Yes               | --decode-gci=FILENAME     |
+| Encrypted GCI without key | .bin.gci/.dat.gci | No                | --decode-gci=FILENAME (3) |
+| Encrypted DLQ             | .bin.dlq/.dat.dlq | Yes               | --decode-dlq=FILENAME     |
+| QST                       | .qst              | Yes               | --decode-qst=FILENAME     |
+
+*Notes:*
+1. *This is the default format. You can convert these to uncompressed format with [gctools](https://github.com/fuzziqersoftware/gctools)' prsd like this: `prsd -d < FILENAME.bin > FILENAME.bind`*
+2. *As in (1), to compress an uncompressed quest file: `prsd < FILENAME.bind > FILENAME.bin`*
+3. *If you know the encryption seed (serial number), pass it in as a hex string with the `--seed=` option. If you don't know the encryption seed, newserv will find it for you, which will likely take a long time.*
 
 Episode 3 quests consist only of a .bin file - there is no corresponding .dat file. Episode 3 .bin files can be encoded in any of the formats described above, except .qst.
 

src/Quest.cc

@@ -71,7 +71,9 @@ struct ShuffleTables {
 
 struct PSOGCIFileHeader {
   parray<uint8_t, 0x40> gci_header;
-  parray<uint8_t, 0x40> pso_header;
+  ptext<char, 0x1C> game_name; // e.g. "PSO EPISODE I & II"
+  be_uint32_t embedded_seed; // Used in some of Ralf's quest packs
+  ptext<char, 0x20> quest_name;
   parray<uint8_t, 0x2000> banner_and_icon;
   // data_size specifies the number of bytes in the encrypted section, including
   // the encrypted header (below) and all encrypted data after it.
@@ -82,7 +84,9 @@ struct PSOGCIFileHeader {
   be_uint32_t checksum;
 
   bool checksum_correct() const {
-    uint32_t cs = crc32(&this->pso_header, sizeof(this->pso_header));
+    uint32_t cs = crc32(&this->game_name, sizeof(this->game_name));
+    cs = crc32(&this->embedded_seed, sizeof(this->embedded_seed), cs);
+    cs = crc32(&this->quest_name, sizeof(this->quest_name), cs);
     cs = crc32(&this->banner_and_icon, sizeof(this->banner_and_icon), cs);
     cs = crc32(&this->data_size, sizeof(this->data_size), cs);
     cs = crc32("\0\0\0\0", 4, cs);
@@ -110,7 +114,6 @@ string decrypt_gci_data_section(
     shuf.shuffle(decrypted.data(), data_section, size, true);
   }
 
-  decrypted.resize((decrypted.size() + 3) & (~3));
   auto* be_dwords = reinterpret_cast<be_uint32_t*>(decrypted.data());
 
   PSOV2Encryption crypt(seed);
@@ -132,13 +135,24 @@ string decrypt_gci_data_section(
     throw runtime_error("incorrect decrypted data section checksum");
   }
 
+  size_t orig_size = decrypted.size();
+  decrypted.resize((orig_size + 3) & (~3));
   PSOV2Encryption(header->round3_seed).decrypt(
       decrypted.data() + sizeof(PSOGCIFileEncryptedHeader),
       decrypted.size() - sizeof(PSOGCIFileEncryptedHeader));
+  decrypted.resize(orig_size);
 
   string ret = decrypted.substr(sizeof(PSOGCIFileEncryptedHeader));
-  if (prs_decompress_size(ret) != header->decompressed_size) {
-    throw runtime_error("decompressed size does not match size in header");
+
+  // Some GCI files have decompressed_size fields that are 8 bytes smaller than
+  // the actual decompressed size of the data. They seem to work fine, so we
+  // accept both cases as correct.
+  size_t decompressed_size = prs_decompress_size(ret);
+  if ((decompressed_size != header->decompressed_size) &&
+      (decompressed_size != header->decompressed_size - 8)) {
+    throw runtime_error(string_printf(
+        "decompressed size (%zu) does not match size in header (%" PRId32 ")",
+        decompressed_size, header->decompressed_size.load()));
   }
 
   return ret;
@@ -594,6 +608,10 @@ string Quest::decode_gci(
       return decrypt_gci_data_section(
           r.getv(header.data_size), header.data_size, known_seed);
 
+    } else if (header.embedded_seed != 0) {
+      return decrypt_gci_data_section(
+          r.getv(header.data_size), header.data_size, header.embedded_seed);
+
     } else {
       if (find_seed_num_threads < 0) {
         throw runtime_error("GCI file appears to be encrypted");
@@ -607,7 +625,7 @@ string Quest::decode_gci(
 
   } else { // Unencrypted GCI format
     r.skip(sizeof(PSOGCIFileEncryptedHeader));
-    string compressed_data = r.read(r.remaining());
+    string compressed_data = r.readx(header.data_size - sizeof(PSOGCIFileEncryptedHeader));
     size_t decompressed_bytes = prs_decompress_size(compressed_data);
 
     size_t expected_decompressed_bytes = encrypted_header.decompressed_size - 8;