support qst format

CMakeLists.txt

@@ -17,6 +17,8 @@ endif()
 include_directories("/usr/local/include")
 link_directories("/usr/local/lib")
 
+set(CMAKE_BUILD_TYPE Debug)
+
 
 
 # Executable definitions

README.md

@@ -4,22 +4,10 @@ newserv is a game server and proxy for Phantasy Star Online (PSO).
 
 This project includes code that was reverse-engineered by the community in ages long past, and has been included in many projects since then. It also includes some game data from Phantasy Star Online itself; this data was originally created by Sega.
 
-This project is a rewrite of a rewrite of a game server that I wrote many years ago. So far, it works well with PSO GC Episodes 1 & 2, and lobbies (but not games) are implemented on Episode 3. newserv is based on an older project of mine that supported other versions (PC and BB), but I no longer have a way to test those versions, so the implementation here probably doesn't work for them.
+This project is a rewrite of a rewrite of a game server that I wrote many years ago. So far, it works well with PSO GC Episodes 1 & 2, and lobbies (but not games) are implemented on Episode 3. Some basic functionality works on PSO PC, but there are probably still some cases that lead to errors (which will disconnect the client). newserv is based on an older project of mine that supported BB as well, but I no longer have a way to test BB, so the implementation here probably doesn't work for it.
 
 Feel free to submit GitHub issues if you find bugs or have feature requests. I'd like to make the server as stable and complete as possible, but I can't promise that I'll respond to issues in a timely manner.
 
-## History
-
-In ages long past (probably 2004? I honestly can't remember), I wrote a proxy for PSO, which I named khyps. This haphazardly-glued-together mess of Windows GUI code and socket programming provided an interface to insert commands into the connection between PSO and its server, enabling some fun new features. Importantly, it also automatically blocked malformed commands which would have crashed the client, providing a safe way to navigate the wasteland that the official Sega servers had turned into after the Action Replay enable code for the game was released.
-
-khyps soon reached "maturity" and became uninteresting, so in 2005 I began writing a PSO server. This project became known as khyller, evolving into a full-featured environment supporting all versions of the game that I had access to - PC, GC, and BB. But as this evolution occurred, the code became increasingly ugly and hard to work with, littered with debugging filth that I never cleaned up and odd coding patterns that I had picked up over the years.
-
-Sometime in 2006 or 2007, I abandoned khyller and rebuilt the entire thing from scratch, resulting in newserv. But this newserv was not the project you're looking at now; 2007's newserv was substantially cleaner in code than khyller but was still quite ugly, and it lacked a few of the more esoteric features I had originally written (for example, the ability to convert any quest into a download quest). I felt better about working with this code, but it still had some stability problems. It turns out that 2007's newserv's concurrency implementation was simply incorrect - I had derived the concept of a mutex myself (before taking any real computer engineering classes) but implemented it incorrectly. No wonder newserv would randomly crash after running seemingly fine for a few days.
-
-A little-known fact is that no version of khyller or newserv was ever tested with the DreamCast versions of PSO. Both projects claimed to support them, but the DC server implementations were based only on chat conversations (likely now lost to time) with other people in the community who had done research on the DC version.
-
-Sometime in October 2018, I had some random cause to reminisce. I looked back in my old code archives and came across newserv. Somehow inspired, I spent a weekend and a couple more evenings rewriting the entire project again, cleaning up ancient patterns I had used eleven years ago, replacing entire modules with simple STL containers, and eliminating even more support files in favor of configuration autodetection. The code is now suitably modern and it no longer has insidious concurrency bugs because it's no longer concurrent - the server is now entirely event-driven.
-
 ## Future
 
 This project is primarily for my own nostalgia; I offer no guarantees on how or when this project will advance.
@@ -42,6 +30,28 @@ So, you've read all of the above and you want to try it out? Here's what you do:
 - Run `./newserv` in the newserv directory. This will start the game server and run the interactive shell. You may need `sudo` if newserv's built-in DNS server is enabled.
 - Use the interactive shell to add a license. Run `help` in the shell to see how to do this.
 
+### Installing quests
+
+newserv automatically finds quests in the system/quests directory. To install your own quests, or to use quests you've saved using the proxy's set-save-files option, just put them in that directory and name them appropriately.
+
+There are multiple PSO quest formats out there; newserv supports most of them. Specifically, newserv can use quests in any of the following formats:
+- bin/dat format: These quests consist of two files with the same base name, a .bin file and a .dat file.
+- Unencrypted GCI format: These quests also consist of a .bin and .dat file, but an encoding is applied on top of them. The filenames should end in .bin.gci and .dat.gci.
+- Encrypted DLQ format: These quests also consist of a .bin and .dat file, but downlaod quest encryption is applied on top of them. The filenames should end in .bin.dlq and .dat.dlq.
+- QST format: These quests consist of only a .qst file, which contains both the .bin and .dat files within it.
+
+Standard quest file names should be like `q###-CATEGORY-VERSION.EXT`; battle quests should be named like `b###-VERSION.EXT`, and challenge quests should be named like `c###-VERSION.EXT`. The fields in each filename are:
+- `###`: quest number (this doesn't really matter; it should just be unique for the version)
+- `CATEGORY`: ret = Retrieval, ext = Extermination, evt = Events, shp = Shops, vr = VR, twr = Tower, dl = Download (these don't appear during online play), 1p = Solo (BB only)
+- `VERSION`: d1 = DreamCast v1, dc = DreamCast v2, pc = PC, gc = GameCube Episodes 1 & 2, gc3 = Episode 3, bb = Blue Burst
+- `EXT`: file extension (bin, dat, bin.gci, dat.gci, bin.dlq, dat.dlq, or qst)
+
+When newserv indexes the quests during startup, it will warn (but not fail) if any quests are corrupt or in unrecognized formats.
+
+If you've changed the contents of the quests directory, you can re-index the quests without restarting the server by running `reload quests` in the interactive shell.
+
+All quests, including those originally in GCI or DLQ format, are treated as online quests unless their filenames specify the dl category. newserv allows players to download all quests, even those in non-download categories.
+
 ### Using newserv as a proxy
 
 If you want to play online on remote servers rather than running your own server, newserv also includes a PSO proxy. Currently this works with PSO GC and may work with PC, but not with BB.

src/CommandFormats.hh

@@ -6,6 +6,8 @@
 #include <phosg/Encoding.hh>
 
 #include "PSOProtocol.hh"
+#include "Text.hh"
+#include "Player.hh"
 
 
 

src/Quest.cc

@@ -8,6 +8,7 @@
 #include <phosg/Random.hh>
 #include <phosg/Strings.hh>
 
+#include "CommandFormats.hh"
 #include "Compression.hh"
 #include "PSOEncryption.hh"
 #include "Text.hh"
@@ -153,6 +154,9 @@ Quest::Quest(const string& bin_filename)
   } else if (ends_with(bin_filename, ".bin.dlq")) {
     this->file_format = FileFormat::BIN_DAT_DLQ;
     this->file_basename = bin_filename.substr(0, bin_filename.size() - 8);
+  } else if (ends_with(bin_filename, ".qst")) {
+    this->file_format = FileFormat::QST;
+    this->file_basename = bin_filename.substr(0, bin_filename.size() - 4);
   } else if (ends_with(bin_filename, ".bin")) {
     this->file_basename = bin_filename.substr(0, bin_filename.size() - 4);
   } else {
@@ -168,7 +172,9 @@ Quest::Quest(const string& bin_filename)
       basename = bin_filename;
     }
   }
-  basename.resize(basename.size() - ((this->file_format == FileFormat::BIN_DAT) ? 4 : 8));
+  bool has_short_extension = (this->file_format == FileFormat::BIN_DAT) ||
+      (this->file_format == FileFormat::QST);
+  basename.resize(basename.size() - (has_short_extension ? 4 : 8));
 
   // quest filenames are like:
   // b###-VV.bin for battle mode
@@ -346,6 +352,12 @@ shared_ptr<const string> Quest::bin_contents() const {
       case FileFormat::BIN_DAT_DLQ:
         this->bin_contents_ptr.reset(new string(this->decode_dlq(this->file_basename + ".bin.dlq")));
         break;
+      case FileFormat::QST: {
+        auto result = this->decode_qst(this->file_basename + ".qst");
+        this->bin_contents_ptr.reset(new string(move(result.first)));
+        this->dat_contents_ptr.reset(new string(move(result.second)));
+        break;
+      }
       default:
         throw logic_error("invalid quest file format");
     }
@@ -365,6 +377,12 @@ shared_ptr<const string> Quest::dat_contents() const {
       case FileFormat::BIN_DAT_DLQ:
         this->dat_contents_ptr.reset(new string(this->decode_dlq(this->file_basename + ".dat.dlq")));
         break;
+      case FileFormat::QST: {
+        auto result = this->decode_qst(this->file_basename + ".qst");
+        this->bin_contents_ptr.reset(new string(move(result.first)));
+        this->dat_contents_ptr.reset(new string(move(result.second)));
+        break;
+      }
       default:
         throw logic_error("invalid quest file format");
     }
@@ -372,35 +390,6 @@ shared_ptr<const string> Quest::dat_contents() const {
   return this->dat_contents_ptr;
 }
 
-string Quest::decode_dlq(const string& filename) {
-  uint32_t decompressed_size;
-  uint32_t key;
-  string data;
-  {
-    auto f = fopen_unique(filename, "rb");
-    decompressed_size = freadx<le_uint32_t>(f.get());
-    key = freadx<le_uint32_t>(f.get());
-    data = read_all(f.get());
-  }
-
-  PSOPCEncryption encr(key);
-
-  // The compressed data size does not need to be a multiple of 4, but the PC
-  // encryption (which is used for all download quests, even in V3) requires the
-  // data size to be a multiple of 4. We'll just temporarily stick a few bytes
-  // on the end, then throw them away later if needed.
-  size_t original_size = data.size();
-  data.resize((data.size() + 3) & (~3));
-  encr.decrypt(data);
-  data.resize(original_size);
-
-  if (prs_decompress_size(data) != decompressed_size) {
-    throw runtime_error("decompressed size does not match size in header");
-  }
-
-  return data;
-}
-
 string Quest::decode_gci(const string& filename) {
 
   string data = load_file(filename);
@@ -448,6 +437,147 @@ string Quest::decode_gci(const string& filename) {
   return data_to_decompress;
 }
 
+string Quest::decode_dlq(const string& filename) {
+  uint32_t decompressed_size;
+  uint32_t key;
+  string data;
+  {
+    auto f = fopen_unique(filename, "rb");
+    decompressed_size = freadx<le_uint32_t>(f.get());
+    key = freadx<le_uint32_t>(f.get());
+    data = read_all(f.get());
+  }
+
+  PSOPCEncryption encr(key);
+
+  // The compressed data size does not need to be a multiple of 4, but the PC
+  // encryption (which is used for all download quests, even in V3) requires the
+  // data size to be a multiple of 4. We'll just temporarily stick a few bytes
+  // on the end, then throw them away later if needed.
+  size_t original_size = data.size();
+  data.resize((data.size() + 3) & (~3));
+  encr.decrypt(data);
+  data.resize(original_size);
+
+  if (prs_decompress_size(data) != decompressed_size) {
+    throw runtime_error("decompressed size does not match size in header");
+  }
+
+  return data;
+}
+
+template <typename HeaderT, typename OpenFileT>
+static pair<string, string> decode_qst_t(FILE* f) {
+  string qst_data = read_all(f);
+  StringReader r(qst_data);
+
+  string bin_contents;
+  string dat_contents;
+  string internal_bin_filename;
+  string internal_dat_filename;
+  uint32_t bin_file_size = 0;
+  uint32_t dat_file_size = 0;
+  while (!r.eof()) {
+    // Handle BB's implicit 8-byte command alignment
+    static constexpr size_t alignment = sizeof(HeaderT);
+    size_t next_command_offset = (r.where() + (alignment - 1)) & ~(alignment - 1);
+    r.go(next_command_offset);
+    if (r.eof()) {
+      break;
+    }
+
+    const auto& header = r.get<HeaderT>();
+    if (header.command == 0x44) {
+      if (header.size != sizeof(HeaderT) + sizeof(OpenFileT)) {
+        throw runtime_error("qst open file command has incorrect size");
+      }
+      const auto& cmd = r.get<OpenFileT>(f);
+      string internal_filename = cmd.filename;
+
+      if (ends_with(internal_filename, ".bin")) {
+        if (internal_bin_filename.empty()) {
+          internal_bin_filename = internal_filename;
+        } else {
+          throw runtime_error("qst contains multiple bin files");
+        }
+        bin_file_size = cmd.file_size;
+
+      } else if (ends_with(internal_filename, ".dat")) {
+        if (internal_dat_filename.empty()) {
+          internal_dat_filename = internal_filename;
+        } else {
+          throw runtime_error("qst contains multiple dat files");
+        }
+        dat_file_size = cmd.file_size;
+
+      } else {
+        throw runtime_error("qst contains non-bin, non-dat file");
+      }
+
+    } else if (header.command == 0x13) {
+      if (header.size != sizeof(HeaderT) + sizeof(S_WriteFile_13_A7)) {
+        throw runtime_error("qst write file command has incorrect size");
+      }
+      const auto& cmd = r.get<S_WriteFile_13_A7>();
+      string filename = cmd.filename;
+
+      string* dest = nullptr;
+      if (filename == internal_bin_filename) {
+        dest = &bin_contents;
+      } else if (filename == internal_dat_filename) {
+        dest = &dat_contents;
+      } else {
+        throw runtime_error("qst contains write commnd for non-open file");
+      }
+
+      if (cmd.data_size > 0x400) {
+        throw runtime_error("qst contains invalid write command");
+      }
+      if (dest->size() & 0x3FF) {
+        throw runtime_error("qst contains uneven chunks out of order");
+      }
+      if (header.flag != dest->size() / 0x400) {
+        throw runtime_error("qst contains chunks out of order");
+      }
+      dest->append(reinterpret_cast<const char*>(cmd.data), cmd.data_size);
+
+    } else {
+      throw runtime_error("invalid command in qst file");
+    }
+  }
+
+  if (bin_contents.size() != bin_file_size) {
+    throw runtime_error("bin file does not match expected size");
+  }
+  if (dat_contents.size() != dat_file_size) {
+    throw runtime_error("dat file does not match expected size");
+  }
+
+  return make_pair(bin_contents, dat_contents);
+}
+
+pair<string, string> Quest::decode_qst(const string& filename) {
+  auto f = fopen_unique(filename, "rb");
+
+  // qst files start with an open file command, but the format differs depending
+  // on the PSO version that the qst file is for. We can detect the format from
+  // the first 4 bytes in the file:
+  // - BB: 58 00 44 00
+  // - PC: 3C ?? 44 00
+  // - DC/GC: 44 ?? 3C 00
+  uint32_t signature = freadx<be_uint32_t>(f.get());
+  fseek(f.get(), 0, SEEK_SET);
+  if (signature == 0x58004400) {
+    return decode_qst_t<PSOCommandHeaderBB, S_OpenFile_BB_44_A6>(f.get());
+  } else if ((signature & 0xFF00FFFF) == 0x3C004400) {
+    return decode_qst_t<PSOCommandHeaderPC, S_OpenFile_PC_GC_44_A6>(f.get());
+  } else if ((signature & 0xFF00FFFF) == 0x44003C00) {
+    return decode_qst_t<PSOCommandHeaderDCGC, S_OpenFile_PC_GC_44_A6>(f.get());
+  } else {
+    throw runtime_error("invalid qst file format");
+  }
+}
+
 
 
 QuestIndex::QuestIndex(const std::string& directory) : directory(directory) {
@@ -465,7 +595,8 @@ QuestIndex::QuestIndex(const std::string& directory) : directory(directory) {
 
     if (ends_with(filename, ".bin") ||
         ends_with(filename, ".bin.gci") ||
-        ends_with(filename, ".bin.dlq")) {
+        ends_with(filename, ".bin.dlq") ||
+        ends_with(filename, ".qst")) {
       try {
         shared_ptr<Quest> q(new Quest(full_path));
         this->version_id_to_quest.emplace(make_pair(q->version, q->quest_id), q);

src/Quest.hh

@@ -38,12 +38,14 @@ class Quest {
 private:
   static std::string decode_gci(const std::string& filename);
   static std::string decode_dlq(const std::string& filename);
+  static std::pair<std::string, std::string> decode_qst(const std::string& filename);
 
 public:
   enum class FileFormat {
     BIN_DAT = 0,
     BIN_DAT_GCI,
     BIN_DAT_DLQ,
+    QST,
   };
   int64_t quest_id;
   QuestCategory category;