enable item tracking on NTE and 11/2000 and make $item work

README.md

@@ -166,7 +166,7 @@ newserv supports many features unique to Episode 3:
 
 #### Battle records
 
-After playing a battle, you can save the record of the battle with the $saverec command. You can then replay the battle later by using the $playrec command in a lobby - this will create a spectator team and play the recording of the battle as if it were happening in realtime. Note that there is a bug in older versions of Dolphin that seems to be frequently triggered when playing battle records, which causes the emulator to crash with the message `QObject::~QObject: Timers cannot be stopped from another thread`. To avoid this, use the latest version of Dolphin.
+After playing a battle, you can save the record of the battle with the `$saverec` command. You can then replay the battle later by using the `$playrec` command in a lobby - this will create a spectator team and play the recording of the battle as if it were happening in realtime. Note that there is a bug in older versions of Dolphin that seems to be frequently triggered when playing battle records, which causes the emulator to crash with the message `QObject::~QObject: Timers cannot be stopped from another thread`. To avoid this, use the latest version of Dolphin.
 
 #### Tournaments
 
@@ -258,13 +258,13 @@ There are many options available when starting a proxy session. All options are
     * Episode 3 card definitions (saved as .mnr files)
     * Episode 3 media updates (saved as .gvm, .bml, or .bin files)
 
-The remote server will probably try to assign you a Guild Card number that doesn't match the one you have on newserv. On PSO DC, PC and GC, the proxy server rewrites the commands in transit to make it look like the remote server assigned you the same Guild Card number as you have on newserv, but if the remote server has some external integrations (e.g. forum or Discord bots), they will use the Guild Card number that the remote server believes it has assigned to you. The number assigned by the remote server is shown to you when you first connect to the remote server, and you can retrieve it in lobbies or during games with the $li command.
+The remote server will probably try to assign you a Guild Card number that doesn't match the one you have on newserv. On PSO DC, PC and GC, the proxy server rewrites the commands in transit to make it look like the remote server assigned you the same Guild Card number as you have on newserv, but if the remote server has some external integrations (e.g. forum or Discord bots), they will use the Guild Card number that the remote server believes it has assigned to you. The number assigned by the remote server is shown to you when you first connect to the remote server, and you can retrieve it in lobbies or during games with the `$li` command.
 
 Some chat commands (see below) have the same basic function on the proxy server but have different effects or conditions. In addition, there are some server shell commands that affect clients on the proxy (run `help` in the shell to see what they are). If there's only one proxy session open, the shell's proxy commands will affect that session. Otherwise, you'll have to specify which session to affect with the `on` prefix - to send a chat message in LinkedSession:17205AE4, for example, you would run `on 17205AE4 chat ...`.
 
 ### Chat commands
 
-newserv supports a variety of commands players can use by chatting in-game. Any chat message that begins with `$` is treated as a chat command. (If you actually want to send a chat message starting with `$`, type `$$` instead.)
+newserv supports a variety of commands players can use by chatting in-game. Any chat message that begins with `$` is treated as a chat command. (If you actually want to send a chat message starting with `$`, type `$$` instead.) On the DC 11/2000 prototype, `@` is used instead of `$` for all chat commands, since `$` does not appear on the English virtual keyboard.
 
 Some commands only work on the game server and not on the proxy server. The chat commands are:
 

src/ChatCommands.cc

@@ -1870,6 +1870,9 @@ struct SplitCommand {
 // command, and to execute the command and block the chat if it is.
 void on_chat_command(std::shared_ptr<Client> c, const std::string& text) {
   SplitCommand cmd(text);
+  if (!cmd.name.empty() && cmd.name[0] == '@') {
+    cmd.name[0] = '$';
+  }
 
   const ChatCommandDefinition* def = nullptr;
   try {

src/CommandFormats.hh

@@ -3953,7 +3953,11 @@ struct G_DestroyNPC_6x1C {
 // 6x1D: Invalid subcommand
 // 6x1E: Invalid subcommand
 
-// 6x1F: Set player floor
+// 6x1F: Set player floor and request positions
+
+struct G_SetPlayerFloor_DCNTE_6x1F {
+  G_ClientIDHeader header;
+} __packed__;
 
 struct G_SetPlayerFloor_6x1F {
   G_ClientIDHeader header;
@@ -3961,8 +3965,8 @@ struct G_SetPlayerFloor_6x1F {
 } __packed__;
 
 // 6x20: Set position
-// Existing clients send this when a new client joins a lobby/game, so the new
-// client knows where to place them.
+// Existing clients send this in response to a 6x1F command when a new client
+// joins a lobby or game, so the new client knows where to place them.
 
 struct G_SetPosition_6x20 {
   G_ClientIDHeader header;
@@ -4565,7 +4569,13 @@ struct G_UseBossWarp_6x6A {
   le_uint16_t unused = 0;
 } __packed__;
 
-// 6x6B: Sync enemy state (used while loading into game; same header format as 6E)
+// 6x6B: Sync enemy state (used while loading into game)
+
+struct G_SyncGameStateHeader_DCNTE_6x6B_6x6C_6x6D_6x6E {
+  G_ExtendedHeader<G_UnusedHeader> header;
+  le_uint32_t decompressed_size = 0;
+  // BC0-compressed data follows here (see bc0_decompress)
+} __packed__;
 
 struct G_SyncGameStateHeader_6x6B_6x6C_6x6D_6x6E {
   G_ExtendedHeader<G_UnusedHeader> header;
@@ -4576,7 +4586,6 @@ struct G_SyncGameStateHeader_6x6B_6x6C_6x6D_6x6E {
 
 // Decompressed format is a list of these
 struct G_SyncEnemyState_6x6B_Entry_Decompressed {
-  // TODO: Verify this format on DC and PC. It appears correct for GC and BB.
   le_uint32_t flags = 0;
   le_uint16_t last_attacker = 0;
   le_uint16_t remaining_hp = 0;
@@ -4591,7 +4600,6 @@ struct G_SyncEnemyState_6x6B_Entry_Decompressed {
 
 // Decompressed format is a list of these
 struct G_SyncObjectState_6x6C_Entry_Decompressed {
-  // TODO: Verify this format on DC and PC. It appears correct for GC and BB.
   le_uint16_t flags = 0;
   le_uint16_t object_index = 0;
 } __packed__;
@@ -4641,7 +4649,6 @@ struct G_SyncItemState_6x6D_Decompressed {
 // Compressed format is the same as 6x6B.
 
 struct G_SyncFlagState_6x6E_Decompressed {
-  // TODO: Verify this format on DC and PC. It appears correct for GC and BB.
   // The three unknowns here are the sizes (in bytes) of three fields
   // immediately following this structure. It is currently unknown what these
   // fields represent. The three unknown fields always sum to the size field.
@@ -4663,9 +4670,11 @@ struct G_SetQuestFlags_6x6F {
 // 6x70: Sync player disp data and inventory (used while loading into game)
 // Annoyingly, they didn't use the same format as the 65/67/68 commands here,
 // and instead rearranged a bunch of things.
-// The format appears to be the same for all pre-BB PSO versions, although
-// Episode 3 does not send this command at all since the relevant data is sent
-// to the joining player in the 64 command instead.
+// The format appears to be the same for all pre-BB PSO versions except DC NTE,
+// although Episode 3 does not send this command at all since the relevant data
+// is sent to the joining player in the 64 command instead.
+
+// TODO: Document DC NTE format, and check if DC 11/2000 format is the same.
 
 struct G_SyncPlayerDispAndInventory_DC_PC_GC_6x70 {
   // Offsets in this struct are relative to the overall command header
@@ -4682,9 +4691,9 @@ struct G_SyncPlayerDispAndInventory_DC_PC_GC_6x70 {
   /* 0024 */ le_uint32_t angle_y;
   /* 0028 */ le_uint32_t angle_z;
   /* 002C */ le_uint16_t unknown_a3a;
-  /* 002C */ le_uint16_t current_hp;
-  /* 002C */ le_uint16_t bonus_hp_from_materials;
-  /* 002C */ le_uint16_t bonus_tp_from_materials;
+  /* 002E */ le_uint16_t current_hp;
+  /* 0030 */ le_uint16_t bonus_hp_from_materials; // Missing on DC NTE
+  /* 0032 */ le_uint16_t bonus_tp_from_materials; // Missing on DC NTE
   /* 0034 */ parray<parray<le_uint32_t, 3>, 5> unknown_a4;
   /* 0070 */ le_uint32_t language = 0;
   /* 0074 */ le_uint32_t player_tag = 0;

src/Lobby.cc

@@ -479,10 +479,11 @@ void Lobby::add_client(shared_ptr<Client> c, ssize_t required_client_id) {
         this->next_game_item_id = m.reassign_all_item_ids(this->next_game_item_id);
       }
     }
-    // We don't consume item IDs here because the 6F handler will do it for
-    // real; we just want to see what they would be when the join command is
-    // sent
-    this->assign_inventory_and_bank_item_ids(c, false);
+    // On DC NTE and 11/2000, the game assigns item IDs immediately when a
+    // player joins a game, then assigns them again after the 6x6D equivalent is
+    // received. For this reason, we consume item IDs here only if the client is
+    // NTE or 11/2000.
+    this->assign_inventory_and_bank_item_ids(c, is_pre_v1(c->version()));
   }
 
   // If the lobby is recording a battle record, add the player join event
@@ -713,7 +714,7 @@ void Lobby::assign_inventory_and_bank_item_ids(shared_ptr<Client> c, bool consum
   if (!consume_ids) {
     this->next_item_id_for_client[c->lobby_client_id] = start_item_id;
   }
-  if (c->log.info("Assigned inventory item IDs")) {
+  if (c->log.info("Assigned inventory item IDs%s", consume_ids ? "" : " but did not mark IDs as used")) {
     p->print_inventory(stderr, c->version(), c->require_server_state()->item_name_index);
     if (p->bank.num_items) {
       p->bank.assign_ids(0x99000000 + (c->lobby_client_id << 20));

src/ProxyCommands.cc

@@ -1595,13 +1595,14 @@ static HandlerResult C_06(shared_ptr<ProxyServer::LinkedSession> ses, uint16_t,
       return HandlerResult::Type::SUPPRESS;
     }
 
-    bool is_command = (text[0] == '$') ||
-        (text[0] == '\t' && text[1] != 'C' && text[2] == '$');
+    char command_sentinel = (ses->version() == Version::DC_V1_11_2000_PROTOTYPE) ? '@' : '$';
+    bool is_command = (text[0] == command_sentinel) ||
+        (text[0] == '\t' && text[1] != 'C' && text[2] == command_sentinel);
     if (is_command && ses->config.check_flag(Client::Flag::PROXY_CHAT_COMMANDS_ENABLED)) {
       size_t offset = ((text[0] & 0xF0) == 0x40) ? 1 : 0;
-      offset += (text[offset] == '$') ? 0 : 2;
+      offset += (text[offset] == command_sentinel) ? 0 : 2;
       text = text.substr(offset);
-      if (text.size() >= 2 && text[1] == '$') {
+      if (text.size() >= 2 && text[1] == command_sentinel) {
         if (ses->config.check_flag(Client::Flag::PROXY_CHAT_FILTER_ENABLED)) {
           send_chat_message_from_client(ses->server_channel, add_color(text.substr(1)), private_flags);
         } else {

src/ReceiveCommands.cc

@@ -3084,8 +3084,9 @@ static void on_06(shared_ptr<Client> c, uint16_t, uint32_t, string& data) {
     return;
   }
 
-  if (text[0] == '$') {
-    if (text[1] == '$') {
+  char command_sentinel = (c->version() == Version::DC_V1_11_2000_PROTOTYPE) ? '@' : '$';
+  if (text[0] == command_sentinel) {
+    if (text[1] == command_sentinel) {
       text = text.substr(1);
     } else {
       on_chat_command(c, text);

src/SendCommands.cc

@@ -24,6 +24,16 @@ using namespace std;
 
 extern const char* QUEST_BARRIER_DISCONNECT_HOOK_NAME;
 
+inline uint8_t get_pre_v1_subcommand(Version v, uint8_t nte_subcommand, uint8_t proto_subcommand, uint8_t final_subcommand) {
+  if (v == Version::DC_NTE) {
+    return nte_subcommand;
+  } else if (v == Version::DC_V1_11_2000_PROTOTYPE) {
+    return proto_subcommand;
+  } else {
+    return final_subcommand;
+  }
+}
+
 const unordered_set<uint32_t> v2_crypt_initial_client_commands({
     0x00260088, // (17) DCNTE license check
     0x00B0008B, // (02) DCNTE login
@@ -2297,13 +2307,6 @@ void send_ep3_change_music(Channel& ch, uint32_t song) {
   ch.send(0x60, 0x00, cmd);
 }
 
-void send_set_player_visibility(shared_ptr<Lobby> l, shared_ptr<Client> c, bool visible) {
-  uint8_t subcmd = visible ? 0x23 : 0x22;
-  uint16_t client_id = c->lobby_client_id;
-  G_SetPlayerVisibility_6x22_6x23 cmd = {{subcmd, 0x01, client_id}};
-  send_command_t(l, 0x60, 0x00, cmd);
-}
-
 void send_game_item_state(shared_ptr<Client> c) {
   auto l = c->require_lobby();
   auto s = c->require_server_state();
@@ -2340,16 +2343,25 @@ void send_game_item_state(shared_ptr<Client> c) {
 
   string compressed_data = bc0_compress(decompressed_w.str());
 
-  G_SyncGameStateHeader_6x6B_6x6C_6x6D_6x6E compressed_header;
-  compressed_header.header.basic_header.subcommand = 0x6D;
-  compressed_header.header.basic_header.size = 0x00;
-  compressed_header.header.basic_header.unused = 0x0000;
-  compressed_header.header.size = (compressed_data.size() + sizeof(G_SyncGameStateHeader_6x6B_6x6C_6x6D_6x6E) + 3) & (~3);
-  compressed_header.decompressed_size = decompressed_w.size();
-  compressed_header.compressed_size = compressed_data.size();
-
   StringWriter w;
-  w.put(compressed_header);
+  if (is_pre_v1(c->version())) {
+    G_SyncGameStateHeader_DCNTE_6x6B_6x6C_6x6D_6x6E compressed_header;
+    compressed_header.header.basic_header.subcommand = (c->version() == Version::DC_NTE) ? 0x5E : 0x65;
+    compressed_header.header.basic_header.size = 0x00;
+    compressed_header.header.basic_header.unused = 0x0000;
+    compressed_header.header.size = (compressed_data.size() + sizeof(G_SyncGameStateHeader_DCNTE_6x6B_6x6C_6x6D_6x6E) + 3) & (~3);
+    compressed_header.decompressed_size = decompressed_w.size();
+    w.put(compressed_header);
+  } else {
+    G_SyncGameStateHeader_6x6B_6x6C_6x6D_6x6E compressed_header;
+    compressed_header.header.basic_header.subcommand = 0x6D;
+    compressed_header.header.basic_header.size = 0x00;
+    compressed_header.header.basic_header.unused = 0x0000;
+    compressed_header.header.size = (compressed_data.size() + sizeof(G_SyncGameStateHeader_6x6B_6x6C_6x6D_6x6E) + 3) & (~3);
+    compressed_header.decompressed_size = decompressed_w.size();
+    compressed_header.compressed_size = compressed_data.size();
+    w.put(compressed_header);
+  }
   w.write(compressed_data);
   while (w.size() & 3) {
     w.put_u8(0x00);
@@ -2368,8 +2380,9 @@ void send_game_item_state(shared_ptr<Client> c) {
 
 void send_drop_item_to_channel(shared_ptr<ServerState> s, Channel& ch, const ItemData& item,
     bool from_enemy, uint8_t floor, float x, float z, uint16_t entity_id) {
+  uint8_t subcommand = get_pre_v1_subcommand(ch.version, 0x51, 0x58, 0x5F);
   G_DropItem_PC_V3_BB_6x5F cmd = {
-      {{0x5F, 0x0B, 0x0000}, {floor, from_enemy, entity_id, x, z, 0, 0, item}}, 0};
+      {{subcommand, 0x0B, 0x0000}, {floor, from_enemy, entity_id, x, z, 0, 0, item}}, 0};
   cmd.item.item.encode_for_version(ch.version, s->item_parameter_table_for_version(ch.version));
   ch.send(0x60, 0x00, &cmd, sizeof(cmd));
 }
@@ -2387,7 +2400,8 @@ void send_drop_item_to_lobby(shared_ptr<Lobby> l, const ItemData& item,
 
 void send_drop_stacked_item_to_channel(
     shared_ptr<ServerState> s, Channel& ch, const ItemData& item, uint8_t floor, float x, float z) {
-  G_DropStackedItem_PC_V3_BB_6x5D cmd = {{{0x5D, 0x0A, 0x0000}, floor, 0, x, z, item}, 0};
+  uint8_t subcommand = get_pre_v1_subcommand(ch.version, 0x4F, 0x56, 0x5D);
+  G_DropStackedItem_PC_V3_BB_6x5D cmd = {{{subcommand, 0x0A, 0x0000}, floor, 0, x, z, item}, 0};
   cmd.item_data.encode_for_version(ch.version, s->item_parameter_table_for_version(ch.version));
   ch.send(0x60, 0x00, &cmd, sizeof(cmd));
 }
@@ -2403,7 +2417,8 @@ void send_drop_stacked_item_to_lobby(shared_ptr<Lobby> l, const ItemData& item,
 }
 
 void send_pick_up_item_to_client(shared_ptr<Client> c, uint8_t client_id, uint32_t item_id, uint8_t floor) {
-  G_PickUpItem_6x59 cmd = {{0x59, 0x03, client_id}, client_id, floor, item_id};
+  uint8_t subcommand = get_pre_v1_subcommand(c->version(), 0x4B, 0x52, 0x59);
+  G_PickUpItem_6x59 cmd = {{subcommand, 0x03, client_id}, client_id, floor, item_id};
   send_command_t(c, 0x60, 0x00, cmd);
 }
 
@@ -2433,7 +2448,8 @@ void send_create_inventory_item_to_lobby(shared_ptr<Client> c, uint8_t client_id
 void send_destroy_item_to_lobby(shared_ptr<Client> c, uint32_t item_id, uint32_t amount, bool exclude_c) {
   auto l = c->require_lobby();
   uint16_t client_id = c->lobby_client_id;
-  G_DeleteInventoryItem_6x29 cmd = {{0x29, 0x03, client_id}, item_id, amount};
+  uint8_t subcommand = get_pre_v1_subcommand(c->version(), 0x25, 0x27, 0x29);
+  G_DeleteInventoryItem_6x29 cmd = {{subcommand, 0x03, client_id}, item_id, amount};
   if (exclude_c) {
     send_command_excluding_client(l, c, 0x60, 0x00, &cmd, sizeof(cmd));
   } else {
@@ -2442,7 +2458,8 @@ void send_destroy_item_to_lobby(shared_ptr<Client> c, uint32_t item_id, uint32_t
 }
 
 void send_destroy_floor_item_to_client(shared_ptr<Client> c, uint32_t item_id, uint32_t floor) {
-  G_DestroyFloorItem_6x63 cmd = {{0x63, 0x03, 0x0000}, item_id, floor};
+  uint8_t subcommand = get_pre_v1_subcommand(c->version(), 0x55, 0x5C, 0x63);
+  G_DestroyFloorItem_6x63 cmd = {{subcommand, 0x03, 0x0000}, item_id, floor};
   send_command_t(c, 0x60, 0x00, cmd);
 }
 
@@ -2511,8 +2528,9 @@ void send_level_up(shared_ptr<Client> c) {
   } catch (const out_of_range&) {
   }
 
+  uint8_t subcommand = get_pre_v1_subcommand(c->version(), 0x2C, 0x2E, 0x30);
   G_LevelUp_6x30 cmd = {
-      {0x30, sizeof(G_LevelUp_6x30) / 4, c->lobby_client_id},
+      {subcommand, sizeof(G_LevelUp_6x30) / 4, c->lobby_client_id},
       stats.atp + (mag ? ((mag->data1w[3] / 100) * 2) : 0),
       stats.mst + (mag ? ((mag->data1w[5] / 100) * 2) : 0),
       stats.evp,

src/SendCommands.hh

@@ -295,7 +295,6 @@ void send_warp(std::shared_ptr<Client> c, uint32_t floor, bool is_private);
 void send_warp(std::shared_ptr<Lobby> l, uint32_t floor, bool is_private);
 
 void send_ep3_change_music(Channel& ch, uint32_t song);
-void send_set_player_visibility(std::shared_ptr<Client> c, bool visible);
 void send_revive_player(std::shared_ptr<Client> c);
 
 void send_game_item_state(std::shared_ptr<Client> c);

src/Version.hh

@@ -31,6 +31,9 @@ Version enum_for_name<Version>(const char* name);
 inline bool is_patch(Version version) {
   return (version == Version::PC_PATCH) || (version == Version::BB_PATCH);
 }
+inline bool is_pre_v1(Version version) {
+  return (version == Version::DC_NTE) || (version == Version::DC_V1_11_2000_PROTOTYPE);
+}
 inline bool is_v1(Version version) {
   return (version == Version::DC_NTE) || (version == Version::DC_V1_11_2000_PROTOTYPE) || (version == Version::DC_V1);
 }