I thought this thread was about EE modding so handles being 64 bit in D:OS2 is not very helpful for EE modding ;-)
Yeah I know, but I thought the news was exciting enough to share anyway
Hey, shame on you, YOU are the 'Father of EE modding', Sir, you've already forgotten your own child ? ... ;-)
Tsk, tsk. My statement is actually correct, as you'll see below
... was a joke !
Without your tool I doubt any bigger mod project would ever have been doable, I can say that for XC_Bags for sure. This is why 'Father' ... ;-)
Actually, handles in EE were not even 32 bit, because that would already have been enough to never have to worry about them.
Raze once gave us the information, that the handle generator is actually a simple counter. A patch note once said, they increased the handle limit to 64k.
The _handle_ is actually 32-bit. However, multiple fields are stuffed into this 32-bit handle; an (apparently) 16-bit counter, a type identifier, and some undentified bits (possibly generation count or flags?).
Well, my English ... ;-)
Yes, handles are 32 bit, but handle values are 16 bits, i.e. different types of 16 bit handles stuffed into different 'windows' of a single 32 bit value.
Although I don't know it, I suppose, they use a 16bit short for the handle counter. Extending the possible pool from 32k to 64k might have been a simple change from 'short' to 'unsigned short'.
Yeah, one bit was most likely sacrificed for counters from some other function.
Looking at a save file one time, I came to think that they use several 16 bit short counters for different purposes of handles and 'shift' them to certain windows of a 32bit int, simply by bit-or'ing them with e.g. 0x10000 for items, 0x20000 for characters, 0x30000 for scenery and so forth. Maybe a remainder - as so many things in classic and EE - from earlier Larian games.
The values I saw in the save files were all above 2^16, so it cannot be a pure short that's used to store the actual handle, it's probably a 32bit int.
Yeah, the lower 16-bits (value & 0x0000FFFF) are the counter, upper 16-bits (so, value & 0xFFFF0000 in binary) is the type+misc flags. My observed type prefix list contains:
0x0000 - CacheTemplate
0x0200 - Trigger
0x0040 - Item
0x0140 - Inventory Factory
0x01C0 - Inventory View
0x0440 - Combat Component
0x0040 - Character Creator
0x0080 - Item Creator
0x0180 - Party
Version 1 probably grew bigger than intended which pushed the limit which stemmed from previous games dangerously close to overflow.
There was too little time to rewrite the engine and many other more important priorities.
It's only a stupid implementation of the handle generator that introduced the 64k limit.
On that we can agree. I'd say that a handle limit as low as 32k is a questionable design decision, but maybe with the original scope of the game it made sense? I wouldn't be so quick to blame this on shortsightedness or lazyness before knowing why the handles work the way they do. (I've already experienced this first-hand when someone criticized my code for being bad, when it was actually fulfilling a design requirement they did not know about :P)
I don't blame them for the design, I'm aware that a lot of basics of the engine(s) are from previous games.
Since I've released a mod, I too know what one gets blamed for ;-)
I'd like to note though that each individual type has its separate pool, each with a separate 64k limit (so you can have 64k characters and 64k items, not 64k characters+items).
Raze even confirmed that before ;-)
I only got really paranoid about handle crashes because they *might* be the reason for the crashes-on-zoning I experienced myself in my first two playthrough attempts with the Enhanced Edition.
Only my third attempt, when I had increased stack sizes, introduced 'destruction' crafting recipes etc., was successful.
I am still unsure if it was really the out-of-handles crash I experienced, because the info I have so far does not explain it, unless new items are created 'behind the scenes' when one changes a zone.
I still think that there might be an inventory overflow bug of a too-many-handles-per-region bug, that would much more explain a crash-on-zoning to me.
(Increasing stack sizes and 'consolidation of objects' to very few different highly stackable object types enabled my first successful playthrough.)
And even that would have never been a big problem if there was a 'used/free handles bitmap' that would have enabled handle re-use. The latter could probably never be implemented because none of the API functions ever released their handles, or more precisely, they would have had to report the handle release to some handle bookkeeper which was not the case as it seems.
There is a "bookkeeper" of sorts for every type. Eg. items have an ItemManager, characters have CharacterManager, etc. This is quite visible in the savegame structure too, each "manager" or component gets its own region in the save file, so I believe it could have been implemented.
Most of the time the rationale behind the "counter with no pool" pattern is that handle usage is uncontrolled, so even though an item was destroyed someone somewhere could be holding a handle to that. Eg. imagine the following situation:
1) Script A gets a handle for character 1111, sets a timer to destroy it after eg. 30 seconds.
2) During those 30 secs, the character gets killed, its ID is released to the free pool.
3) A new character is created that claims the now "free" 1111 id.
4) When the script runs, it now destroys the wrong thing.
This kind of handle misuse can be relatively common and the bugs they cause are quite deceptive and hard to detect. They'll also do random and unpredictable things depending on which ID they reuse. Since Osiris code is probably riddled with such possibilities (which is quite plausible, as you don't want to burden your designers with the task of undersanding concurrency at that level -- quite a headache even for programmers!), I believe the ever-increasing-handles to be a plausible solution for the problem; what they failed to account for was the rapid exhaustion of the ID space.
Don't see an unsolvable problem here, it's only a matter of design:
A file in a Unix file system does not get deleted as long as there's a hard link to it, only when the last 'hard link' is removed, the file is actually deleted. (or 'unlink()'ed ;-)
That's the bookkeeper's responsibility.
As I said before, bookkeeping requires reporting and all the handle functions were probably never built in a way to report back the release of a handle.
All might have had to do with too little time to re-do the whole engine when version 1 was already late.
(Remember the 'How Divinity: Original Sin almost bancrupted Larian Studios' article news that was number 1 news on Steam for months.)
Version 2 looks like a big cleanup from what you posted in the other thread. Good for future modders. Maybe they finally do it correct instead of 'as quickly as possible', and generic instead of 'what we currently need'.
(Maybe even the extremely annoying 'Magic Pockets' popups can be turned off but I have doubts. They were the main reason I disabled this whole functionality for shovels and pickaxes in XC_Bags ;-)
... and maybe they even implement a real 'game timer' which they wanted to do a long time ago as one of the comments in an Osiris file suggests ;-)
We shall see. I'm hoping for the best.
Seeing what you posted in the other thread suggests the 'big cleanup'.
After all, they want to continue using the engine in future games too which will not decrease in complexity I suppose. So it's probably primarily for themselves and future modders can profit ....
