HomeUser Control Panel (unavailable in archive)ForumsTutorialsArt GalleryResourcesMaps

EMS

03-30-2010, 08:31 AM#1
Krysho
EMS

What is EMS? An introduction
EMS was originally conceptualized as a tool to enhance the capabilities of maps, with the only requirement being that the host need be using it in order to receive the benefits. The map itself would not require any player, even the host, to be using the tool in order to run without the EMS enhancements. EMS would only provide additional features, with no additional restriction on the map.

What EMS provides specifically is pretty simple: an "EmsPlugin" interface is exposed to programmers, allowing them to write a DLL meant to be loaded by EMS and utilized by maps. EMS allows plugins to register a callback function to a limited set of Wc3 events provided by EMS. The most prominent registration available occurs in the form of registering plugin code to a Wc3 function call.

In short, EMS can serve as a gateway to extend Wc3's API indefinitely with very little effort on a plugin writer's part.

Of course, that also means malicious plugins can be written and easily used against hosts using EMS, though this should not be a problem so long as hosts using EMS download their plugins from a trusted source only (or compile plugins from source).

There are some limitations on what will be provided "out of the box" with EMS by us. As mentioned previously, EMS's original purpose is to provide extended functionality with only the host using it.

EMS currently attains this functionality by providing local calls that will not cause desyncs. For example, a map may request the current date and time. This request would be processed by the host and stored in a variable of the map maker's choosing. Of course, only the host would have access to this data, and using it in nearly any useful way would likely result in a desync. Thus, EMS assumes the map maker will write the syncing methods.

How is that? Wc3 does provide some natives to sync data in the game cache. These natives are not typically safe to use (and serve little known purpose with no data unknown to Wc3 being introduced by Wc3). There are two main issues with syncing in this manner that I have noticed.

The first is that acting on sync data may result in a desync due to not all players being synced to the same data at that point in time. This problem can be bypassed (as far as I have seen) by writing your sync code while exercising some caution.

The second problem is that there is no (known) way in Wc3 to ensure that data from a sync is coming from one player. It is possible to write code in a map such that only one player will be expected to send data (with respect to the map's JASS code), but this is not something that should be relied on. EMS currently provides a "locking" mechanism to syncs. Since sync requests are routed through the host (like nearly any other packet), the host may choose to limit the player source of sync data. Currently, it is possible to lock categories, labels, and category/label combinations to specific player IDs. EMS will take note of the sync request and the player the request is coming from (via the socket the request came in). Currently, players sending a disallowed sync request will be desynced from the host immediately. It should be noted that the locking mechanisms take precedence as follows: category > label > category/label

The concept that EMS need only be used by the host is probably not a limitation a plugin writer would have to be concerned about. Maps probably could be written to use EMS and EMS plugins very extensively, even requiring their use, but this was not in our original image of the tool and is not officially supported in this first release.

In order to use EMS and EMS plugin functions, map makers typically will be expected to write JASS function stubs that will indirectly invoke the EMS / EMS plugin function. If the function is expected to return a value, the body of the JASS function should contain a default return value signifying an error. This way, if EMS is not enabled, the map can continue on as if nothing is wrong by simply checking for a default return value from the function.

An example function stub that invokes the EMS presence check is:
Collapse JASS:
function EMS_IsHooked takes nothing returns boolean
    return false
endfunction


By default this would return false. However, if EMS is running, a true value would be received by whatever JASS method is calling it.

Thus, a set of function stubs will likely be expected to be provided by a plugin writer, similar to a C++ header file.

EMS API
Currently, the core EMS API is as follows:
Expand JASS:

There are additional commands provided by a limited plugin set.

util
Expand JASS:

fileio
Expand JASS:

sqlite
Expand JASS:

Data storage
As you might have noticed, there are functions provided for file input/output and even an sqlite function set. Thus, here I will explain briefly the attempts at securing the data EMS stores.

When EMS first notices a map (via the map calling EMS_CMS while EMS is running), it will calculate the SHA-1 of the map and use that as a key to find the map in the map database that EMS maintains. If the map is not found, a new entry is created in the map database. Map IDs are assigned to maps in sequential order (via SQL's auto_increment). EMS maintains subdirectories inside of its /data subdirectory, with each map data directory being named by its map id (the first map would be 1, the second would be 2, and so on). It is inside these numbered subdirectories that maps store their data (assuming the map uses fileio or sqlite for now).

The general rule is that maps have the ability to read data from any other map (locating the map in question by searching the map database with the SHA-1 of the map to find). However, maps can only write data to their /data subdirectory (thus, maps should not be able to write to other maps' data directories). Whether this behavior is maintained or not depends on plugin writers following this rule.

War3Natives
EMS plugins have the ability to call Wc3 natives. The call might be like so:
Code:
ems->native()->War3Native("DisplayTimedTextToPlayer", ems->native()->War3Native("Player", 0), 0., 0., 10., "STRING TEXT");
in order to display "STRING TEXT" to the local player (if he happens to be player 1).

If the above code is no indication, return values of natives are received properly.

Syncing values in EMS maps
EMS leaves it to the map to sync its values. Wc3's JASS provides various natives that utilize gamecaches to sync values. In particular, these natives are:
Collapse JASS:
native TriggerSyncStart     takes nothing returns nothing
native TriggerSyncReady     takes nothing returns nothing
native SyncStoredInteger        takes gamecache cache, string missionKey, string key returns nothing
native SyncStoredReal           takes gamecache cache, string missionKey, string key returns nothing
native SyncStoredBoolean        takes gamecache cache, string missionKey, string key returns nothing
native SyncStoredUnit           takes gamecache cache, string missionKey, string key returns nothing
native SyncStoredString         takes gamecache cache, string missionKey, string key returns nothing

From various testing last year, I concluded that TriggerSyncReady and TriggerSyncStart are both pointless. The latter is not necessary to sync values, and the former does not stop possible desyncs. TriggerSyncReady seems to be equivalent to a TriggerSleepAction. Thus, the previous two natives are not used. SyncStoredUnit and SyncStoredString also are not used in the EMS example map. As such, the only natives of concern for the EMS example map are the real, boolean, and integer sync methods, as well as the methods to store/retrieve from gamecaches.

Although if someone cares to write up a map to prove me wrong about TriggerSyncReady, feel free to do so. Likely such things would make sync systems easier to write. It was a long time ago that I did any testing using a simple TriggerSyncStart -> Sync -> TriggerSyncReady scheme, I only recall that my implementation (and someone else's) had desync problems when used.

The example map uses a pretty simple sync system. Sync requests (a list of strings) are stored in a queue (also a list). Periodically, the map checks members of the queue if their sync has completed (this is both faster and more safe than using TriggerSyncReady). Since syncs are expected to be received sequentially, the example map checks the oldest sync requests (and a constant "look ahead" number of syncs) to avoid needlessly checking all sync requests.

Every player syncs a "received" value, which is sent after all the other expected sync values. The map checks locally that it has a "received" value for every human player currently in the game (players set their "ready" value to 1, sync it, then reset it to -1, allowing the sync to bounce back from the host and reset the value to 1 again - supposedly preventing a player from acting too soon). In theory, the method is not foolproof, but in practice it seems to work even if one player is lagging during an intense sync session.

As mentioned previously, EMS does provide some functions to block other players from sending values they should not be sending (such as another player's "received" value) in order to ruin a sync.

In other notes, the SyncStoredString native does not work. A sync library could implement this storing four letters per integer, or perhaps using some other method, but as is, the example map does not do so.

Possible issues in EMS
- Previously, I have noted that EMS might sometimes cause Wc3 to silently crash in various situations involving hosting a game in LAN (most likely in many other situations, but LAN games are predominantly where I tested EMS). Fixing a separate issue, I noticed one of the EMS hooks was not thread safe. The hook is now (supposedly) thread safe, and I have not noticed this crash since then, though I am not sure if EMS will not crash due to other hooks in a similar manner.
- Not all War3Natives might function correctly; in the past, we have noted that some may cause Wc3 to crash. EMS development has been stalled for a while since then, and I have not been able to dig up the issue in my (very) limited testing. If you happen to discover a native that crashes, feel free to post about it.
- EMS currently injects into Wc3 by searching for a window called "Warcraft III" (by default). We are ignorant of whether other languages of Wc3 have different window titles or not. If it is so, the "setpid" command can be utilized on whatever your war3.exe process happens to be, followed by "start" to inject to that pid
- EMS identifies function calls by name; do not use any vJASS features that would mangle a function call if you plan to use that function as an EMS stub
- You may have to run EMS.exe as an administrator in order to achieve proper permissions to inject into Wc3; various other problems may happen at injection time (I plan on seeing various complaints about injection if this ever kicks off)

Random information
- EMS is coded in C++/ASM (inlined)
- EMS writes Wc3 memory at six locations (to serve as hooks)
- Although EMS is meant to be used on Battle.net, it may not exactly be entirely safe to do so if Blizzard decides to update Warden to search for it (which does not seem to be the case now - well, EMS was not released until now)
That means a potential ban if used on Battle.net, just so you know
(though since Blizzard seems to target hacks specifically and EMS does not fit any previous criteria, this seems unlikely by the precedent set)
- Code::Blocks was the IDE used; if you happen to use Code::Blocks, you can load up the project files for each plugin
- EMS (and plugins) compiled with GCC (TDM-2 mingw32) 4.4.1
- The source of EMS itself is not currently released (and whether it will be in the future is unknown); source for the packaged plugins (and some others) are available in the attachment, however

Usage of most of EMS's provided functionality is notable in the plugins' source provided in the attachment.

How to test EMS

An example map showing usage of various EMS functionality is also attached.

To test the example map properly:
1) Run EMS.exe
2) In the EMS console, enter the command "start" (without quotations)
3) Start Wc3 (also fine if Wc3 is already running)
4) Start example map and run the commands following below (as chat messages in the map)

The provided tests are:
- Type in "-test" to test file input/output and some EMS util commands
- Type in "on" to test syncing between several players
- Type "create", then "-level x" (replace x with an integer from 2 to 10), then "save"; then start the map again and try "load"

The map is meant to be tested with other players in the game (LAN or Battle.net)

The example map has sloppy code in many places, I apologize in advance. The map also has some sync code to handle queuing syncs and prevent desyncs, though this could be written better as well.

The map requires vJASS support to save.

EMS.exe features a "help" command.

EMS is (was?) a collaborative project by Darimus and I.


Feel free to post any questions, concerns, suggestions, and so on.
A lot more could be written to better detail various parts of EMS, such as some actual documentation, but I would rather see what the reaction is to this beforehand.

04-06-10 - Example map modified slightly
test commands should work with the host as different players
sync queue modified to remove sync entries with a source player that is invalid (player is not present in the game or is not human)
map now finds the host after five seconds and syncs this value with other players; "on" test command does not work before five seconds have passed

06-23-10 - A chance edit due to issues found when adding stuff to TFTLocal ...
Example map modified slightly again; previously the sync limit was 10, this has been changed to 12 (if someone was the eleventh or twelfth player, the map would not recognize that person as host due to the sync limit)

EMS modified slightly; method of mapping PIDs to map player numbers was changed to hopefully not fail; a crash when EMS was used with TFTLocal's /dp and /ds commands was apparently fixed

It seems that using some commands in the first few seconds of the example map (such as "load") will cause desyncs; waiting until after the host is found seems to resolve that; not caring enough to look into the issue for now

05-21-11 - updated for patch 1.26.0
no other intended changes
Attached Files
File type: w3xMapSync2-11-3.w3x (50.3 KB)
File type: 7zems-11.0710.7z (518.0 KB)
04-06-2010, 11:46 AM#2
Element of Water
So this is basically RtC, but better since only the host needs to have it running? Wow.
04-06-2010, 12:49 PM#3
Krysho
The presence of EMS code in a map does not automatically make other players not using EMS unable to play the map. However, EMS will not magically make functions like GetMouseX and GetMouseY work on everyone's computer with just the host running EMS (this should be obvious, but I think it needs to be emphasized again and again).

Another difference is that EMS does not use native function calls. Thus, users can simply use empty functions within their map to serve as EMS calls (as noted in the first post).

Ever since RtC announced its change of direction (providing an easy method of extending the Wc3 API), I guess the only real advantage EMS has is that map makers can choose to use a limited feature set such that normal players can play the map without a problem. I would not take my word on anything RtC related as dependable, though, since I have never used RtC and do not follow its development much.

Example: RPG maps looking to very quickly implement a stored save/load type deal can just look for the local player using EMS, then save a line with the hero's load code; on a load attempt, it could attempt to read a line. Afterwards, the player could sync his code or hero statistics. EMS used in this manner is completely optional, since players could simply opt to type out their load codes. The more robust method might use sqlite and both the host and user: the host could store data on each user that plays in his games, and each user could store his/her own data. The map could choose the more recent information and load that, in the case that both players are using EMS (or just the host's value, in the case that only the host is using EMS).

Although EMS could, in practice, be used just like RtC; that is, every player must be using it for the map to function completely. If a lot of people wanted to use it that way, likely EMS would be extended to provide EMS player checks in the lobby or some such.

Though, EMS has not been bug tested much at all (besides by myself). And, I meant for development of EMS to change based on user suggestions after the first release (the basic idea and proof of concept is out, now people can suggest where to go and what to add). As currently is, EMS provides mostly IO, which I guess is different from RtC's current aim.

If no one decides to utilize EMS, then continued development is not going to happen. Even if that (no mappers using EMS) were to happen, I may still use EMS for some mostly "in house" things. That remains to be seen.
12-06-2010, 06:22 PM#4
Strilanc
Game cache syncs can be made desync-safe quite easily. You just use the following pattern:

Code:
    temp = current value in game cache
    store new value
    call sync on new value
    restore old value from temp

What this does is send a game cache sync action without modifying any local values in game cache. This action goes to the host, who includes it in the next tick, which goes to everyone, at which point all players apply it synchronously (w.r.t game time) (this is the same mechanism that syncs unit orders).

My async lag system used this functionality to measure latency from within the game. Players synced their current game time into the cache, which would do the round trip to the host, causing the synced-into-cache time was behind the true game time by an amount proportional to their latency.

Not too hard to hide this stuff behind a struct. Apologies for errors in the jass, as I am quite rusty.

Collapse JASS:
globals
    gamecache gc = InitGameCache("sync.gc") //note: not done properly
endglobals
struct SyncInt
    private string key = IntToString(this)
    public method get takes nothing returns integer
        return LoadInteger(gc, .key, .key)
    endmethod
    ///Immediately sets the value. Only safe if the new value is non-local.
    ///Note: An unfinished syncSet can still overwrite the value. This method does not prevent that from happening.
    public method unsafeSet takes integer value returns nothing
        call SaveInteger(gc, .key, .key, value)
    endmethod
    ///Sends a game cache sync action which will synchronously set the value in the future for all players.
    public method syncSet takes integer value returns nothing
        local int old = .get()
        call .unsafeSet(value)
        call SyncInteger(gc, .key, .key)
        call .unsafeSet(old)
    endmethod
endstruct

But beware, because every value used this way can be easily controlled by any one of the clients. A tool with the ability to insert sync actions could easily ruin your day by setting strength to five billion.
12-06-2010, 07:38 PM#5
Strilanc
Here is a much improved version of SyncInt. In particular, it prevents you from making the stupid mistake of assigning the value only to have it overwritten by a delayed sync. It also provides support for mediocre tamper-resistance by allowing you to lock the value.

Collapse JASS:
library SyncTypes initializer init
    globals
        private gamecache gc
        private constant string FILENAME = "SyncTypes.gc"
        private constant string INT_KEY = "int"
    endglobals
    
    struct SyncInt
        private boolean locked
        private int lockedValue
        private string key

        private method store takes integer value returns nothing
            call StoreInteger(gc, INT_KEY, .key, value)
        endmethod

        ///Returns the currently assigned value.
        public method get takes nothing returns integer
            if .locked then
                 return .lockedValue
            endif
            return GetStoredInteger(gc, INT_KEY, .key)
        endmethod

        ///Immediately updates the assigned value and starts blocking any running syncTo requests.
        public method lockTo takes integer syncedValue returns nothing
            set .locked = true
            set .lockedValue = syncedValue
        endmethod

        ///Unlocks the assigned value, allowing syncTo requests to work.
        ///Note: Calling lockTo then unlock does *not* cancel active syncTo actions.
        public method unlock takes nothing returns nothing
            call .store(.lockedValue)
            set .locked = false
        endmethod

        ///Starts the process of assigning a value potentially known by only one player to the synced integer.
        ///The assigned value will update after the sync action reaches the host and bounces back to all players.
        public method syncTo takes integer localValue returns nothing
            local integer old = .get()
            if .locked then
                call BJDebugMsg("Warning: Called syncTo on a locked SyncInt.")
            endif
            call .store(localValue)
            call SyncInteger(gc, INT_KEY, .key)
            call .store(old)
        endmethod

        ///Creates a SyncInt with the given value and locked state.
        public static method create takes integer value, boolean initiallyLocked returns SyncInt
            local SyncInt this = SyncInt.allocate()
            set .key = I2S(this)
            call .lockTo(value)
            if not initiallyLocked then
                call .unlock()
            endif
            return this
        endmethod
    endstruct

    private function init takes nothing returns nothing
        call FlushGameCache(InitGameCache(FILENAME))
        set gc = InitGameCache(FILENAME)
    endfunction
endlibrary
12-06-2010, 09:06 PM#6
Krysho
Quote:
Originally Posted by Strilanc
Game cache syncs can be made desync-safe quite easily. You just use the following pattern:

Code:
    temp = current value in game cache
    store new value
    call sync on new value
    restore old value from temp

What this does is send a game cache sync action without modifying any local values in game cache. This action goes to the host, who includes it in the next tick, which goes to everyone, at which point all players apply it synchronously (w.r.t game time) (this is the same mechanism that syncs unit orders).

My async lag system used this functionality to measure latency from within the game. Players synced their current game time into the cache, which would do the round trip to the host, causing the synced-into-cache time was behind the true game time by an amount proportional to their latency.

Not too hard to hide this stuff behind a struct. Apologies for errors in the jass, as I am quite rusty.

Collapse JASS:
globals
    gamecache gc = InitGameCache("sync.gc") //note: not done properly
endglobals
struct SyncInt
    private string key = IntToString(this)
    public method get takes nothing returns integer
        return LoadInteger(gc, .key, .key)
    endmethod
    ///Immediately sets the value. Only safe if the new value is non-local.
    ///Note: An unfinished syncSet can still overwrite the value. This method does not prevent that from happening.
    public method unsafeSet takes integer value returns nothing
        call SaveInteger(gc, .key, .key, value)
    endmethod
    ///Sends a game cache sync action which will synchronously set the value in the future for all players.
    public method syncSet takes integer value returns nothing
        local int old = .get()
        call .unsafeSet(value)
        call SyncInteger(gc, .key, .key)
        call .unsafeSet(old)
    endmethod
endstruct

But beware, because every value used this way can be easily controlled by any one of the clients. A tool with the ability to insert sync actions could easily ruin your day by setting strength to five billion.


The pattern you mention at the top actually sounds quite like how I went about trying to make them desync-safe. I mentioned:
Quote:
Every player syncs a "received" value, which is sent after all the other expected sync values. The map checks locally that it has a "received" value for every human player currently in the game (players set their "ready" value to 1, sync it, then reset it to -1, allowing the sync to bounce back from the host and reset the value to 1 again - supposedly preventing a player from acting too soon). In theory, the method is not foolproof, but in practice it seems to work even if one player is lagging during an intense sync session.

In particular, the similarity is with the "received" value which is set to 1, synced, and then set to -1 before set back to 1 by the host's bounce back. The received value is sent after all other data though, with which I assumed all the other data would be synced properly beforehand (sending sync requests in packs I guess). I had a timer periodically polling the received values to determine when a sync is "complete" in order to call a callback (in the example map, which is terribly coded I imagine, but eh).


As for your last comment in that post, I did have that in mind, which is why EMS provides some rudimentary sync protection functions (basically allowing the map to specify that a sync is expected to come from only one player). Though this functionality only works if the host is using EMS ... an obstacle I would have had to worry about if people actually used EMS (given that host bots rule Battle.net now).

In particular I suppose something would have to be done bot-side, or map makers would have to limit what kind of data they leave to syncing.

Well, locking syncs to one player would do nothing to stop someone from modifying their own sent packets, in which case I would think that the latter option (limiting what type of data is sent) is what is plausible. I figure any scheme designed to protect users from the person sending the sync (which relies on the person's integrity to begin with) is doomed to fail given a persistent and knowledgeable enough person.
12-06-2010, 10:06 PM#7
Strilanc
Yeah there's no authentication to syncs at all. You can't even tell which player synced the last value.
12-08-2010, 10:40 PM#8
Thunder_Eye
Is there any way to circumvent the blockage on writing to another map's database?

I'm currently struggling with RtC to store data throughout a campaign I'm working on. This tool seems alot more fitting for my project, and I was hoping to make a switch.

Storing/Loading data in a central database would be alot more fitting then having to read from different map folders.

(If not, how do I find the SHA-1 of a particular map?)
12-09-2010, 02:51 AM#9
Strilanc
I suggest using public key crypto. Maps signed by the same key can share data.
12-09-2010, 04:47 AM#10
Krysho
Quote:
Originally Posted by Thunder_Eye
Is there any way to circumvent the blockage on writing to another map's database?

I'm currently struggling with RtC to store data throughout a campaign I'm working on. This tool seems alot more fitting for my project, and I was hoping to make a switch.

Storing/Loading data in a central database would be alot more fitting then having to read from different map folders.

At the time being there is no way to do this with the provided plugins, though they could easily be modified to allow you read/write access anywhere on the file system - not something we plan on supporting though.

The issue of storing and loading data within a campaign is not an issue I had thought of previously, I will consider looking into it sometime.

As for why the map read/write system is the way it is now, it seemed like a decent idea to start with. If maps were able to write to other maps' data directories, quite a lot of problems can occur (in the context of independent maps anyway - in a campaign this certainly should be possible).

Strilanc's suggestion sounds like a good idea to look into, will read more on it. I am not familiar with public key cryptography, but I imagine the act of signing a map is a much better idea than what we have now, heh, provided I am understanding it right. Well, a lot of stuff was meant to change in EMS after people started suggesting things anyway.
12-09-2010, 11:49 AM#11
Thunder_Eye
I understand, I think I'll be able to work around it using the SHA-1 to get the map's id's.
I take it that read-only on file io also means read-only for sqllite tables?
12-09-2010, 12:52 PM#12
Krysho
Quote:
Originally Posted by Thunder_Eye
I understand, I think I'll be able to work around it using the SHA-1 to get the map's id's.
I take it that read-only on file io also means read-only for sqllite tables?
There is a limitation imposed by all the IO plugins provided in the official release of EMS. Even getting the map via the SHA-1 would still only allow you to read information from other maps. You could of course make your own IO plugin and compile it.

Probably I will follow Strilanc's suggestion. Not sure when it would be implemented if at all, considering EMS's popularity at this point, but it would certainly make this whole thing easier for you I imagine.

Since I am on break now I probably will look more into this soon.
12-11-2010, 10:36 AM#13
Thunder_Eye
Quote:
Originally Posted by Krysho
... Since I am on break now I probably will look more into this soon.

That would indeed make this alot easier for me.
I think I'm able to manage it using the SHA-1 of maps though, with some (many) complications.

I haven't done any programming other then JASS for a long time, and due to time constraints I can only hope you'll find time to improve on this wonderful tool!
03-01-2011, 05:53 PM#14
Nestharus
I don't think that this would work with host bots, and games are predominantly hosted that way now.