| 03-30-2010, 08:31 AM | #1 |
EMS What is EMS? An introduction EMS was originally conceptualized as a tool to enhance the capabilities of maps, with the only requirement being that the host need be using it in order to receive the benefits. The map itself would not require any player, even the host, to be using the tool in order to run without the EMS enhancements. EMS would only provide additional features, with no additional restriction on the map. What EMS provides specifically is pretty simple: an "EmsPlugin" interface is exposed to programmers, allowing them to write a DLL meant to be loaded by EMS and utilized by maps. EMS allows plugins to register a callback function to a limited set of Wc3 events provided by EMS. The most prominent registration available occurs in the form of registering plugin code to a Wc3 function call. In short, EMS can serve as a gateway to extend Wc3's API indefinitely with very little effort on a plugin writer's part. Of course, that also means malicious plugins can be written and easily used against hosts using EMS, though this should not be a problem so long as hosts using EMS download their plugins from a trusted source only (or compile plugins from source). There are some limitations on what will be provided "out of the box" with EMS by us. As mentioned previously, EMS's original purpose is to provide extended functionality with only the host using it. EMS currently attains this functionality by providing local calls that will not cause desyncs. For example, a map may request the current date and time. This request would be processed by the host and stored in a variable of the map maker's choosing. Of course, only the host would have access to this data, and using it in nearly any useful way would likely result in a desync. Thus, EMS assumes the map maker will write the syncing methods. How is that? Wc3 does provide some natives to sync data in the game cache. These natives are not typically safe to use (and serve little known purpose with no data unknown to Wc3 being introduced by Wc3). There are two main issues with syncing in this manner that I have noticed. The first is that acting on sync data may result in a desync due to not all players being synced to the same data at that point in time. This problem can be bypassed (as far as I have seen) by writing your sync code while exercising some caution. The second problem is that there is no (known) way in Wc3 to ensure that data from a sync is coming from one player. It is possible to write code in a map such that only one player will be expected to send data (with respect to the map's JASS code), but this is not something that should be relied on. EMS currently provides a "locking" mechanism to syncs. Since sync requests are routed through the host (like nearly any other packet), the host may choose to limit the player source of sync data. Currently, it is possible to lock categories, labels, and category/label combinations to specific player IDs. EMS will take note of the sync request and the player the request is coming from (via the socket the request came in). Currently, players sending a disallowed sync request will be desynced from the host immediately. It should be noted that the locking mechanisms take precedence as follows: category > label > category/label The concept that EMS need only be used by the host is probably not a limitation a plugin writer would have to be concerned about. Maps probably could be written to use EMS and EMS plugins very extensively, even requiring their use, but this was not in our original image of the tool and is not officially supported in this first release. In order to use EMS and EMS plugin functions, map makers typically will be expected to write JASS function stubs that will indirectly invoke the EMS / EMS plugin function. If the function is expected to return a value, the body of the JASS function should contain a default return value signifying an error. This way, if EMS is not enabled, the map can continue on as if nothing is wrong by simply checking for a default return value from the function. An example function stub that invokes the EMS presence check is: JASS:function EMS_IsHooked takes nothing returns boolean return false endfunction By default this would return false. However, if EMS is running, a true value would be received by whatever JASS method is calling it. Thus, a set of function stubs will likely be expected to be provided by a plugin writer, similar to a C++ header file. EMS API Currently, the core EMS API is as follows: JASS:library EMS function EMS_CMS takes nothing returns nothing // Create a map session (called only once by the map) endfunction function EMS_Map_ID takes nothing returns integer // Retrieves the map ID (call EMS_CMS first) return 0 endfunction function EMS_Request_IO takes integer map_id, string path returns integer // Request an IO handle to use in IO plugins return -1 endfunction function EMS_Release_IO takes integer io_id returns boolean // Release an IO handle return false endfunction function EMS_Map_ID_From_SHA1 takes string sha1 returns integer // Retrieve a map ID via a SHA-1 (searches the map database) return 0 endfunction function EMS_SyncTest takes integer in returns nothing // ...? endfunction function EMS_PID_From_PN takes integer in returns integer // Converts a player number to a PID (the latter being a value the host stores interally) return -1 endfunction function EMS_SL_Category takes string cat, integer pn returns integer // Sync locks a gamecache category to a player number return -1 // returns an ID with which one can release the lock endfunction function EMS_SL_Label takes string label, integer pn returns integer // Sync locks a gc label to a player number return -1 endfunction function EMS_SL_Pair takes string cat, string label, integer pn returns integer // Sync locks a gc category/label combination return -1 endfunction function EMS_SL_Release takes integer key returns boolean // Releases a sync lock by ID return false endfunction endlibrary There are additional commands provided by a limited plugin set. util JASS:library EMSutil //util function EMS_IsHooked takes nothing returns boolean return false endfunction function EMS_GetHost takes nothing returns player return null endfunction function EMS_Rand takes integer min, integer max returns integer return -1 endfunction function EMS_EnableDebug takes boolean enable returns nothing return endfunction function EMS_LocalYear takes nothing returns integer return -1 endfunction function EMS_LocalMonth takes nothing returns integer return -1 endfunction function EMS_LocalDayOfWeek takes nothing returns integer return -1 endfunction function EMS_LocalDay takes nothing returns integer return -1 endfunction function EMS_LocalHour takes nothing returns integer return -1 endfunction function EMS_LocalMinute takes nothing returns integer return -1 endfunction function EMS_LocalSecond takes nothing returns integer return -1 endfunction function EMS_GetTickCount takes nothing returns integer return -1 endfunction endlibrary fileio JASS:library EMSfileio //fileio function EMS_OpenFile takes integer ioa returns integer return 0 endfunction function EMS_CloseFile takes integer file_id returns nothing return endfunction function EMS_WriteLine takes integer file_id, string text returns nothing return endfunction function EMS_ReadLine takes integer file_id returns string return "" endfunction endlibrary sqlite JASS:library SQLite function SQLite_Open takes integer ioa returns integer return -1 endfunction function SQLite_Close takes integer in_db returns integer return -1 endfunction function SQLite_Prepare_V2 takes integer in_db, string query returns integer return -1 endfunction function SQLite_Step takes integer in_stmt returns integer return -1 endfunction function SQLite_Reset takes integer in_stmt returns integer return -1 endfunction function SQLite_Column_Integer takes integer in_stmt, integer column returns integer return -1 endfunction function SQLite_Column_Real takes integer in_stmt, integer column returns integer return -1 endfunction function SQLite_Finalize takes integer in_stmt returns integer return -1 endfunction function SQLite_Column_Count takes integer in_stmt returns integer return -1 endfunction function SQLite_Get_Status takes nothing returns integer return -1 endfunction function SQLite_Busy_Timeout takes integer in_db, real in_wait returns integer return -1 endfunction endlibrary Data storage As you might have noticed, there are functions provided for file input/output and even an sqlite function set. Thus, here I will explain briefly the attempts at securing the data EMS stores. When EMS first notices a map (via the map calling EMS_CMS while EMS is running), it will calculate the SHA-1 of the map and use that as a key to find the map in the map database that EMS maintains. If the map is not found, a new entry is created in the map database. Map IDs are assigned to maps in sequential order (via SQL's auto_increment). EMS maintains subdirectories inside of its /data subdirectory, with each map data directory being named by its map id (the first map would be 1, the second would be 2, and so on). It is inside these numbered subdirectories that maps store their data (assuming the map uses fileio or sqlite for now). The general rule is that maps have the ability to read data from any other map (locating the map in question by searching the map database with the SHA-1 of the map to find). However, maps can only write data to their /data subdirectory (thus, maps should not be able to write to other maps' data directories). Whether this behavior is maintained or not depends on plugin writers following this rule. War3Natives EMS plugins have the ability to call Wc3 natives. The call might be like so: Code:
ems->native()->War3Native("DisplayTimedTextToPlayer", ems->native()->War3Native("Player", 0), 0., 0., 10., "STRING TEXT");If the above code is no indication, return values of natives are received properly. Syncing values in EMS maps EMS leaves it to the map to sync its values. Wc3's JASS provides various natives that utilize gamecaches to sync values. In particular, these natives are: JASS:native TriggerSyncStart takes nothing returns nothing native TriggerSyncReady takes nothing returns nothing native SyncStoredInteger takes gamecache cache, string missionKey, string key returns nothing native SyncStoredReal takes gamecache cache, string missionKey, string key returns nothing native SyncStoredBoolean takes gamecache cache, string missionKey, string key returns nothing native SyncStoredUnit takes gamecache cache, string missionKey, string key returns nothing native SyncStoredString takes gamecache cache, string missionKey, string key returns nothing From various testing last year, I concluded that TriggerSyncReady and TriggerSyncStart are both pointless. The latter is not necessary to sync values, and the former does not stop possible desyncs. TriggerSyncReady seems to be equivalent to a TriggerSleepAction. Thus, the previous two natives are not used. SyncStoredUnit and SyncStoredString also are not used in the EMS example map. As such, the only natives of concern for the EMS example map are the real, boolean, and integer sync methods, as well as the methods to store/retrieve from gamecaches. Although if someone cares to write up a map to prove me wrong about TriggerSyncReady, feel free to do so. Likely such things would make sync systems easier to write. It was a long time ago that I did any testing using a simple TriggerSyncStart -> Sync -> TriggerSyncReady scheme, I only recall that my implementation (and someone else's) had desync problems when used. The example map uses a pretty simple sync system. Sync requests (a list of strings) are stored in a queue (also a list). Periodically, the map checks members of the queue if their sync has completed (this is both faster and more safe than using TriggerSyncReady). Since syncs are expected to be received sequentially, the example map checks the oldest sync requests (and a constant "look ahead" number of syncs) to avoid needlessly checking all sync requests. Every player syncs a "received" value, which is sent after all the other expected sync values. The map checks locally that it has a "received" value for every human player currently in the game (players set their "ready" value to 1, sync it, then reset it to -1, allowing the sync to bounce back from the host and reset the value to 1 again - supposedly preventing a player from acting too soon). In theory, the method is not foolproof, but in practice it seems to work even if one player is lagging during an intense sync session. As mentioned previously, EMS does provide some functions to block other players from sending values they should not be sending (such as another player's "received" value) in order to ruin a sync. In other notes, the SyncStoredString native does not work. A sync library could implement this storing four letters per integer, or perhaps using some other method, but as is, the example map does not do so. Possible issues in EMS - Previously, I have noted that EMS might sometimes cause Wc3 to silently crash in various situations involving hosting a game in LAN (most likely in many other situations, but LAN games are predominantly where I tested EMS). Fixing a separate issue, I noticed one of the EMS hooks was not thread safe. The hook is now (supposedly) thread safe, and I have not noticed this crash since then, though I am not sure if EMS will not crash due to other hooks in a similar manner. - Not all War3Natives might function correctly; in the past, we have noted that some may cause Wc3 to crash. EMS development has been stalled for a while since then, and I have not been able to dig up the issue in my (very) limited testing. If you happen to discover a native that crashes, feel free to post about it. - EMS currently injects into Wc3 by searching for a window called "Warcraft III" (by default). We are ignorant of whether other languages of Wc3 have different window titles or not. If it is so, the "setpid" command can be utilized on whatever your war3.exe process happens to be, followed by "start" to inject to that pid - EMS identifies function calls by name; do not use any vJASS features that would mangle a function call if you plan to use that function as an EMS stub - You may have to run EMS.exe as an administrator in order to achieve proper permissions to inject into Wc3; various other problems may happen at injection time (I plan on seeing various complaints about injection if this ever kicks off) Random information - EMS is coded in C++/ASM (inlined) - EMS writes Wc3 memory at six locations (to serve as hooks) - Although EMS is meant to be used on Battle.net, it may not exactly be entirely safe to do so if Blizzard decides to update Warden to search for it (which does not seem to be the case now - well, EMS was not released until now) That means a potential ban if used on Battle.net, just so you know (though since Blizzard seems to target hacks specifically and EMS does not fit any previous criteria, this seems unlikely by the precedent set) - Code::Blocks was the IDE used; if you happen to use Code::Blocks, you can load up the project files for each plugin - EMS (and plugins) compiled with GCC (TDM-2 mingw32) 4.4.1 - The source of EMS itself is not currently released (and whether it will be in the future is unknown); source for the packaged plugins (and some others) are available in the attachment, however Usage of most of EMS's provided functionality is notable in the plugins' source provided in the attachment. How to test EMS An example map showing usage of various EMS functionality is also attached. To test the example map properly: 1) Run EMS.exe 2) In the EMS console, enter the command "start" (without quotations) 3) Start Wc3 (also fine if Wc3 is already running) 4) Start example map and run the commands following below (as chat messages in the map) The provided tests are: - Type in "-test" to test file input/output and some EMS util commands - Type in "on" to test syncing between several players - Type "create", then "-level x" (replace x with an integer from 2 to 10), then "save"; then start the map again and try "load" The map is meant to be tested with other players in the game (LAN or Battle.net) The example map has sloppy code in many places, I apologize in advance. The map also has some sync code to handle queuing syncs and prevent desyncs, though this could be written better as well. The map requires vJASS support to save. EMS.exe features a "help" command. EMS is (was?) a collaborative project by Darimus and I. Feel free to post any questions, concerns, suggestions, and so on. A lot more could be written to better detail various parts of EMS, such as some actual documentation, but I would rather see what the reaction is to this beforehand. 04-06-10 - Example map modified slightly test commands should work with the host as different players sync queue modified to remove sync entries with a source player that is invalid (player is not present in the game or is not human) map now finds the host after five seconds and syncs this value with other players; "on" test command does not work before five seconds have passed 06-23-10 - A chance edit due to issues found when adding stuff to TFTLocal ... Example map modified slightly again; previously the sync limit was 10, this has been changed to 12 (if someone was the eleventh or twelfth player, the map would not recognize that person as host due to the sync limit) EMS modified slightly; method of mapping PIDs to map player numbers was changed to hopefully not fail; a crash when EMS was used with TFTLocal's /dp and /ds commands was apparently fixed It seems that using some commands in the first few seconds of the example map (such as "load") will cause desyncs; waiting until after the host is found seems to resolve that; not caring enough to look into the issue for now 05-21-11 - updated for patch 1.26.0 no other intended changes |
| 04-06-2010, 11:46 AM | #2 |
So this is basically RtC, but better since only the host needs to have it running? Wow. |
| 04-06-2010, 12:49 PM | #3 |
The presence of EMS code in a map does not automatically make other players not using EMS unable to play the map. However, EMS will not magically make functions like GetMouseX and GetMouseY work on everyone's computer with just the host running EMS (this should be obvious, but I think it needs to be emphasized again and again). Another difference is that EMS does not use native function calls. Thus, users can simply use empty functions within their map to serve as EMS calls (as noted in the first post). Ever since RtC announced its change of direction (providing an easy method of extending the Wc3 API), I guess the only real advantage EMS has is that map makers can choose to use a limited feature set such that normal players can play the map without a problem. I would not take my word on anything RtC related as dependable, though, since I have never used RtC and do not follow its development much. Example: RPG maps looking to very quickly implement a stored save/load type deal can just look for the local player using EMS, then save a line with the hero's load code; on a load attempt, it could attempt to read a line. Afterwards, the player could sync his code or hero statistics. EMS used in this manner is completely optional, since players could simply opt to type out their load codes. The more robust method might use sqlite and both the host and user: the host could store data on each user that plays in his games, and each user could store his/her own data. The map could choose the more recent information and load that, in the case that both players are using EMS (or just the host's value, in the case that only the host is using EMS). Although EMS could, in practice, be used just like RtC; that is, every player must be using it for the map to function completely. If a lot of people wanted to use it that way, likely EMS would be extended to provide EMS player checks in the lobby or some such. Though, EMS has not been bug tested much at all (besides by myself). And, I meant for development of EMS to change based on user suggestions after the first release (the basic idea and proof of concept is out, now people can suggest where to go and what to add). As currently is, EMS provides mostly IO, which I guess is different from RtC's current aim. If no one decides to utilize EMS, then continued development is not going to happen. Even if that (no mappers using EMS) were to happen, I may still use EMS for some mostly "in house" things. That remains to be seen. |
| 12-06-2010, 06:22 PM | #4 |
Game cache syncs can be made desync-safe quite easily. You just use the following pattern: Code:
temp = current value in game cache
store new value
call sync on new value
restore old value from tempWhat this does is send a game cache sync action without modifying any local values in game cache. This action goes to the host, who includes it in the next tick, which goes to everyone, at which point all players apply it synchronously (w.r.t game time) (this is the same mechanism that syncs unit orders). My async lag system used this functionality to measure latency from within the game. Players synced their current game time into the cache, which would do the round trip to the host, causing the synced-into-cache time was behind the true game time by an amount proportional to their latency. Not too hard to hide this stuff behind a struct. Apologies for errors in the jass, as I am quite rusty. JASS:globals gamecache gc = InitGameCache("sync.gc") //note: not done properly endglobals struct SyncInt private string key = IntToString(this) public method get takes nothing returns integer return LoadInteger(gc, .key, .key) endmethod ///Immediately sets the value. Only safe if the new value is non-local. ///Note: An unfinished syncSet can still overwrite the value. This method does not prevent that from happening. public method unsafeSet takes integer value returns nothing call SaveInteger(gc, .key, .key, value) endmethod ///Sends a game cache sync action which will synchronously set the value in the future for all players. public method syncSet takes integer value returns nothing local int old = .get() call .unsafeSet(value) call SyncInteger(gc, .key, .key) call .unsafeSet(old) endmethod endstruct But beware, because every value used this way can be easily controlled by any one of the clients. A tool with the ability to insert sync actions could easily ruin your day by setting strength to five billion. |
| 12-06-2010, 07:38 PM | #5 |
Here is a much improved version of SyncInt. In particular, it prevents you from making the stupid mistake of assigning the value only to have it overwritten by a delayed sync. It also provides support for mediocre tamper-resistance by allowing you to lock the value. JASS:library SyncTypes initializer init globals private gamecache gc private constant string FILENAME = "SyncTypes.gc" private constant string INT_KEY = "int" endglobals struct SyncInt private boolean locked private int lockedValue private string key private method store takes integer value returns nothing call StoreInteger(gc, INT_KEY, .key, value) endmethod ///Returns the currently assigned value. public method get takes nothing returns integer if .locked then return .lockedValue endif return GetStoredInteger(gc, INT_KEY, .key) endmethod ///Immediately updates the assigned value and starts blocking any running syncTo requests. public method lockTo takes integer syncedValue returns nothing set .locked = true set .lockedValue = syncedValue endmethod ///Unlocks the assigned value, allowing syncTo requests to work. ///Note: Calling lockTo then unlock does *not* cancel active syncTo actions. public method unlock takes nothing returns nothing call .store(.lockedValue) set .locked = false endmethod ///Starts the process of assigning a value potentially known by only one player to the synced integer. ///The assigned value will update after the sync action reaches the host and bounces back to all players. public method syncTo takes integer localValue returns nothing local integer old = .get() if .locked then call BJDebugMsg("Warning: Called syncTo on a locked SyncInt.") endif call .store(localValue) call SyncInteger(gc, INT_KEY, .key) call .store(old) endmethod ///Creates a SyncInt with the given value and locked state. public static method create takes integer value, boolean initiallyLocked returns SyncInt local SyncInt this = SyncInt.allocate() set .key = I2S(this) call .lockTo(value) if not initiallyLocked then call .unlock() endif return this endmethod endstruct private function init takes nothing returns nothing call FlushGameCache(InitGameCache(FILENAME)) set gc = InitGameCache(FILENAME) endfunction endlibrary |
| 12-06-2010, 09:06 PM | #6 | ||
Quote:
The pattern you mention at the top actually sounds quite like how I went about trying to make them desync-safe. I mentioned: Quote:
In particular, the similarity is with the "received" value which is set to 1, synced, and then set to -1 before set back to 1 by the host's bounce back. The received value is sent after all other data though, with which I assumed all the other data would be synced properly beforehand (sending sync requests in packs I guess). I had a timer periodically polling the received values to determine when a sync is "complete" in order to call a callback (in the example map, which is terribly coded I imagine, but eh). As for your last comment in that post, I did have that in mind, which is why EMS provides some rudimentary sync protection functions (basically allowing the map to specify that a sync is expected to come from only one player). Though this functionality only works if the host is using EMS ... an obstacle I would have had to worry about if people actually used EMS (given that host bots rule Battle.net now). In particular I suppose something would have to be done bot-side, or map makers would have to limit what kind of data they leave to syncing. Well, locking syncs to one player would do nothing to stop someone from modifying their own sent packets, in which case I would think that the latter option (limiting what type of data is sent) is what is plausible. I figure any scheme designed to protect users from the person sending the sync (which relies on the person's integrity to begin with) is doomed to fail given a persistent and knowledgeable enough person. |
| 12-06-2010, 10:06 PM | #7 |
Yeah there's no authentication to syncs at all. You can't even tell which player synced the last value. |
| 12-08-2010, 10:40 PM | #8 |
Is there any way to circumvent the blockage on writing to another map's database? I'm currently struggling with RtC to store data throughout a campaign I'm working on. This tool seems alot more fitting for my project, and I was hoping to make a switch. Storing/Loading data in a central database would be alot more fitting then having to read from different map folders. (If not, how do I find the SHA-1 of a particular map?) |
| 12-09-2010, 02:51 AM | #9 |
I suggest using public key crypto. Maps signed by the same key can share data. |
| 12-09-2010, 04:47 AM | #10 | |
Quote:
At the time being there is no way to do this with the provided plugins, though they could easily be modified to allow you read/write access anywhere on the file system - not something we plan on supporting though. The issue of storing and loading data within a campaign is not an issue I had thought of previously, I will consider looking into it sometime. As for why the map read/write system is the way it is now, it seemed like a decent idea to start with. If maps were able to write to other maps' data directories, quite a lot of problems can occur (in the context of independent maps anyway - in a campaign this certainly should be possible). Strilanc's suggestion sounds like a good idea to look into, will read more on it. I am not familiar with public key cryptography, but I imagine the act of signing a map is a much better idea than what we have now, heh, provided I am understanding it right. Well, a lot of stuff was meant to change in EMS after people started suggesting things anyway. |
| 12-09-2010, 11:49 AM | #11 |
I understand, I think I'll be able to work around it using the SHA-1 to get the map's id's. I take it that read-only on file io also means read-only for sqllite tables? |
| 12-09-2010, 12:52 PM | #12 | |
Quote:
Probably I will follow Strilanc's suggestion. Not sure when it would be implemented if at all, considering EMS's popularity at this point, but it would certainly make this whole thing easier for you I imagine. Since I am on break now I probably will look more into this soon. |
| 12-11-2010, 10:36 AM | #13 | |
Quote:
That would indeed make this alot easier for me. I think I'm able to manage it using the SHA-1 of maps though, with some (many) complications. I haven't done any programming other then JASS for a long time, and due to time constraints I can only hope you'll find time to improve on this wonderful tool! |
| 03-01-2011, 05:53 PM | #14 |
I don't think that this would work with host bots, and games are predominantly hosted that way now. |
