Memory leak causes

Cacodemon: The function is for testing the memory leak problem. It creates and destroys a location, and then sets the handle to null.

Grater: The memory increase might not be directly proportional to the leak. I did a second test (up to 2.4 million locations) and noticed how Warcraft 3's memory usage varied by more than 30 megabytes. This might be resources that are unloaded/loaded etc.

jmoritz: I believe you have to set *all* the references to the handle to null for Warcraft to recycle the handle, which sometimes can be easy as pie and sometimes impossible, as functions who wants to return a handle contained in one of its local variables cannot both set it to null and return it.

everyone:

This function also seems to leak. Go figure:


I believe that one of the reasons why Warcaft would want to avoid having the handles recycled would be that a script which was working on the handle might not have registered that the objects were destroyed. If the handles were to be reused, the script would continue manipulating the newly created objects.

So Warcraft core doesn't destroy handles after function execution?

All that sounds like Warcraft actually uses a kind of reference counting but it is bugged in so far as it does not decrease the count for handles that are kept in local variables at the end of the function.

Setting them to null resolves problem? Thank you, I rewrite my scripts where local handles are not destroyed.

But as for that example again - may test results be incorrect? Test function destroys handle before returns it - I think function returns null handle works faster then same function returns non-null handle.

Cacodemon: I'll take time to reprogram my maps too :). And as some of the leaks seems to be unavoidable, we could only hope for Blizzard to notice this problem. That is, if these observations are correct (which I firmly believe they are.)

The test values aren't very accurate, as I have no way of knowing exactly how much memory has "leaked". If you are interested in testing yourself, I'm attaching my test map.

I don't know anything about the speed of the functions, though.

Unfortunately setting the returned value to NULL after using it doesn't seem to fix the memory leak, in other words, the only time it doesn't leak is when it returns NULL which makes it a useless fix.

It might be possible to get around the problem by using some pseudo return value global variable or maybe even rolling your own stack (now THAT would get an effort award...). Altough I'm not convinced that the handle leakage would actually be a significant problem unless your triggers do some very computationally intensive stuff, just something to be aware of and write your triggers to avoid excessive handle leaking.

But how can we avoid leak in case when calling subroutine? If we assign handle how we can destroy it BEFORE return and not destroying returned value?

Not only does this code look strange, the set testloc = null doesn't make any difference. First time you run this, it leaks 5 megs, but after that nothing.
In this case, set testloc = null makes a huge difference (like Peppar said). Memory usage is now the same as the first test: only an initial leak of around 5 MB. If you remove the set testloc = null, this will leak every time you run it. Strangely enough, the first run always leaks much more (10 times as much).

Most likely handles always get recycled, and the leak is caused by something else. My guess is a different kind of data structure is associated with local variables, in addition to the data associated with a handle. This data only gets destroyed when you set the local var to null.

In other words: the initial leak is not a leak, but simply memory reserved for future use.

So every time you call subroutine, memory leaks:




Any suggestions how to avoid leak in this case?

Maybe there are some problems with local handles, as they don't get destroyed after func is executed.

But what if we replace our local var with a global? Then we'll be using only 1 handle no matter how many times our function executes.

function CreateLocation takes nothing returns location
set udg_loc = Location(0, 0)
call RemoveLocation(udg_loc)
return udg_loc
endfunction

udg_loc, as you guessed, is a global location var.
I think it is worth trying...

BTW, both Location() and PolarProjectionBJ() functions return a location handle - but first is native func, so no locals there, and second uses 2 local vars in it's code...

yep that statement will never execute. The only way to fix this, is use global vars (if my theory is correct). Yes PolarProjectionBJ causes a memory leak every time you use it.

Do be reminded that these leaks are small, and only affect performance if your map uses leaking functions a lot. And by that I mean 10,000 times or more. And even at 10,000 it only leaks a few megabytes...

I thought I might want to tell you that "some" of the memory leaks might not be a problem of WarCraft III at all, but rather of probelm in windows (mainly kernel32.dll) where memory allocation functions are defined. You have to remember that "Destroy, Delete, Remove, etc... functions" don't delete the variables, and thus it still takes up memory, only setting local variables to NULL might causes Wc3 to think that the variable is nolonger being used, and calls the approate function to free that segment of memory.

Also I don't care how good of a programmer you are, but you can't stop the "fragmentation" of memory. Where an array, say 32 bytes long, is deleted then an array is requested memory space of 48 bytes, the memory allocator will then allocate memory at the end, because the blank space is only 32 bytes, and is too short, and can't split varaibles. Thus, the 32 byte segment goes unused and just takes up sapce, untill and array of 32 bytes of less requests memory.

To see what I mean, just try this simple test. Create a string varaible and set it starting value to "x". Now just create a trigger that will add random characters to itself, until it reaches 1024 characters long. Also take note to the memory useage before and after the trigger running, it should increae by at least 2,000,000 bytes (2 MB) instead of the suspected 1KB that the string actually takes up. (Note: I haven't tested this myself, just used knowledge of how Wc3/Windows handles memory allocation, so my prediction may be off.)

So, if your wondering how to stop "memory leakes", or basicly the fragmentation of memory, just try to use as many global variables as you can, and setting strings to the maximum length that would be used, for that string. Also when you delete objects, make sure that no variables are still referencing that object. Hope that clears up any confusion on memory leakage.

True, I don't know much about memory fragmentation, aside from how it works. So I cannot say how big of a role it plays in general. However, I can say it doesn't play any role in this situation at all. Maybe if you had read the entire thread you would've figured this out too.

Why? Because we are requesting blocks of memory that are the exact same size, every time. Beging: 1 local variable, 1 handle, and 1 location object. The location object gets destroyed when you call RemoveLocation, the handle gets recycled, and the local var has some issues, but using set var = null fixes it.

From all that it seems pretty clear that Blizzard forgot that at the end of a functions all local handles can be recycled.
I think that War3 uses multiple redirections when handles are accessed. Handles are stored as integers (probably in a kind of stack in the case of local variables). That integer number represents a place in the handle array (except for null). That array place then contains a pointer to the actual object that has been created with a native. The destroy/remove natives destroy the actual object but the place in the handle array remains, only setting it to null or setting it to another handle removes it.
There might be reference counting involved in that removing the handle once nothing points on it anymore and Blizzard forgot to reduce the count for local variables at the end of their function.
Of course it is strange then that they do not remove the actual objects that are associated with the handle when the place in the handle array is removed. But that might be because a lot of these objects can live on without something in the script pointing to them (special effects, triggers, ...).

So the actual bug that causes the leak is the handling of local variables at the end of their life. They are not counted as being set to null.
If that is true then passing a handle that contains a created object to a function (not a native) without doing anything with it in there might result in that handle not being removed properly, causing a leak. (It may be that that only happens when you set a local handle to that passed handle. That will show if passed handles are handled correctly).

That's about the same I was thinking, except the difference in the amount of memory that is leaked suggests there is more than 1 data structure used when you create a local handle AND make it point to something.

The first time you run a test, it leaks much more than any consequent runs. Also, setting the local handle to null removes any small leaks after the first. If you omit RemoveLocation entirely, the memory leak is even bigger.

I'll do a test like AIAndy suggested later.