11 replies to this topic

[Miloch]

Continued from discussions in the comments thread...

any chance of making the main patch a little more WeiDU-friendly? If for no other reason than most (pretty much all) megamods are WeiDU, thus better to have the chance of detecting each other and recording their installation in the WeiDU.log. I've done a WeiDU conversion for small snippets of your patch but don't understand the bsdiff/bspatch format well enough to translate it without a lot of manual hackery... maybe someone else can.

I'd be happy if someone wrote a proper installer for this. Ideally it should:

.patch the executable (same as now, but check whether the version is correct)
.replace _LOW animations in vanilla game files while checking to not accidentally replace them for creatures using new slots

Further components could include:

.automatically detecting mega mods and replacing their animation references with compatible ones
.going through the vanilla game files to make use of new animations (i.e. placing the new content in the game)

So I used WeiDU command line functions (--cmp-from BGMain_unpatched.exe --cmp-to BGMain.exe --out t-patch.tpp) to generate a non-smart patch for the .exe. It generated a ~480kb file over 19,000 lines, meaning it patches that many bytes in the .exe . But, the good news:

• The patch compresses to about 36kb - around half the compressed bspatch at ~78kb
  
• I diffed the .exes patched by the original bspatch and the WeiDU patch. They were identical.
  
• Rather amazingly, WeiDU patches all those bytes in the .exe in like a second or two
  
• It is called from about 3 lines of code in the .tp2, so it won't bog it down
  
• Adding new patches or changing this one is dead easy, even easier than directly editing the .exe (though you may need that as a reference).

So I think that addresses any objections. Except it's WeiDU .

I can try to compress the raw patch a bit with WRITE_LONGs or WRITE_ASCIIs instead of WRITE_BYTEs, but given the existing compression rate and execution speed, I don't think it's worth it. Plus it'd probably make it tougher to maintain. I will however make it a bit smarter to check for a few existing hex addresses have the expected data before applying the patch. Checking every single one would be a waste of time in my opinion, but we could check three or five or ten or whatever, in different spots. Unless you know of some bytes that are more significant, I'll just pick a few at random to check, something like first, middle and last blocks probably.

But that makes me think of something. Yes, this requires the patched ToB executable, but is it possible to do the same or similar for the patched SoA executable? The main reason I'm asking is because I know at least a couple folks are playing Aurora with the svirfneblin animations on SoA. And though I was going to "unstack" the existing slots we use and use this scheme instead for the next release, I don't want to foist ToB on them if they don't want to run out and get it (and some don't). And I'm not sure about the megamods, but I think at least one or two might be SoA-compatible too, or other smaller mods that add animations are anyhow.

So I'll work on making the _LOW patch a bit smarter after that, but I'm still wondering about this...

It should be possible (already does the latter, no?) but ideally the whole patch should be converted to WeiDU. Also then other WeiDU mods using new animations can do a REQUIRE_COMPONENT or similar on this, whereas it's difficult or impossible now.

It's possible, but I wanted to avoid the huge .tp2 (bsdiff compresses extremely well) and keep it simple for me to maintain. You could still REQUIRE_COMPONENT if you check for the _LOW fix or the other (eventual) WeiDU components of this. You could also make a component check for the WeiDU component executing the current patch, now that I think about it.

It doesn't quite do the latter, it just replaces any _LOW (5xxx) animations it finds during install. A smart patch would replace all in the original game files and for override/mod files check if a creature looks like it should have a character animation (is it a humanoid race, does it have a corresponding class, etc.) and patch those while keeping new references (which could also be 5xxx) intact where they're used.

Still not quite sure why the existing patch isn't good enough. If it finds creatures that have the _LOW animations, it just changes them to the equivalent non-LOW versions, aye? Isn't that sufficient? Character-type creatures who don't have the _LOW animations should be fine as they are, shouldn't they? The only exceptions I can think of offhand are svirfneblin and maybe a few other gnomes, but we patch these in Aurora anyway, and I don't think that's what you're talking about.

Edit: ok, I think I see what you're getting at re: mod _LOW creatures. I suspect there are very few of those, but I can check with WeiDU anyway, and see if there's some reason any that turn up should not get converted the same way as vanilla _LOW creatures.

So after that, that would leave:

.automatically detecting mega mods and replacing their animation references with compatible ones
.going through the vanilla game files to make use of new animations (i.e. placing the new content in the game)

The first item should be easy after Arkenor finishes his analysis. I can do part of the second since, as I said, we already do it in Aurora for the UD svirfneblin, goblin captains, etc. But someone should be able to help with this too, by suggesting other existing CREs who would more appropriately get IWD or other animations. And I'm thinking that someone (or one of the someones at least) should be Jarno, since he was champing at the bit for this and now has the chance to contribute something . An easy way to start is to just grep through the creature names with DLTCEP and see if there's an obvious targets, for stuff like Orc Shaman (if there are such CREs and animations, for example).

Edited by Miloch, 22 December 2009 - 10:45 AM.

[Erephine]

So I used WeiDU command line functions (--cmp-from BGMain_unpatched.exe --cmp-to BGMain.exe --out t-patch.tpp) to generate a non-smart patch for the .exe. It generated a ~480kb file over 19,000 lines, meaning it patches that many bytes in the .exe . But, the good news:

• The patch compresses to about 36kb - around half the compressed bspatch at ~78kb
  
• I diffed the .exes patched by the original bspatch and the WeiDU patch. They were identical.
  
• Rather amazingly, WeiDU patches all those bytes in the .exe in like a second or two
  
• It is called from about 3 lines of code in the .tp2, so it won't bog it down
  
• Adding new patches or changing this one is dead easy, even easier than directly editing the .exe (though you may need that as a reference).

So I think that addresses any objections. Except it's WeiDU .

I can try to compress the raw patch a bit with WRITE_LONGs or WRITE_ASCIIs instead of WRITE_BYTEs, but given the existing compression rate and execution speed, I don't think it's worth it. Plus it'd probably make it tougher to maintain. I will however make it a bit smarter to check for a few existing hex addresses have the expected data before applying the patch. Checking every single one would be a waste of time in my opinion, but we could check three or five or ten or whatever, in different spots. Unless you know of some bytes that are more significant, I'll just pick a few at random to check, something like first, middle and last blocks probably.

The patch file itself is actually 6.54 KB But yes.

If there's a simple way to repeat the process you used (that I would understand) I can include it if you like. I'm just concerned about updating the patch, which (obviously) is something I should be able to do without external help If it included the validity checks as well it'd definitely be worth it.

But that makes me think of something. Yes, this requires the patched ToB executable, but is it possible to do the same or similar for the patched SoA executable? The main reason I'm asking is because I know at least a couple folks are playing Aurora with the svirfneblin animations on SoA. And though I was going to "unstack" the existing slots we use and use this scheme instead for the next release, I don't want to foist ToB on them if they don't want to run out and get it (and some don't). And I'm not sure about the megamods, but I think at least one or two might be SoA-compatible too, or other smaller mods that add animations are anyhow.

It's possible, but not trivial (to the point of basically having to redo it from scratch for SoA) since the executables are quite different, and I liberally use ToB's animation code as well.

I might think about it once this implementation is definitely final (having to maintain two would be a nightmare!), but it's a big 'might' at this point.

Sorry.

Still not quite sure why the existing patch isn't good enough. If it finds creatures that have the _LOW animations, it just changes them to the equivalent non-LOW versions, aye? Isn't that sufficient? Character-type creatures who don't have the _LOW animations should be fine as they are, shouldn't they? The only exceptions I can think of offhand are svirfneblin and maybe a few other gnomes, but we patch these in Aurora anyway, and I don't think that's what you're talking about.

Edit: ok, I think I see what you're getting at re: mod _LOW creatures. I suspect there are very few of those, but I can check with WeiDU anyway, and see if there's some reason any that turn up should not get converted the same way as vanilla _LOW creatures.

The problem with the existing patch is the following situation (in that order):

- Player installs a mod using IA containing new animation references (let's say a female drider at 0x5001)
- Player installs the _LOW patching component (patching the 0x5001 .cre reference to 0x6001 which is a male elf cleric)

If the _LOW patching component is installed after new animations were added, it might break their references (since it doesn't try to distinguish between 5xxx entries that were LOW entries and those that were actually IA references). If it's installed in the correct order, the current patch is indeed fine.

[Miloch]

The patch file itself is actually 6.54 KB But yes.

Well, but you also need the bspatch.exe, whereas you need WeiDU.exe either way (to fix the _LOW CREs at least). But my main point was they're both small either way, compressed.

If there's a simple way to repeat the process you used (that I would understand) I can include it if you like. I'm just concerned about updating the patch, which (obviously) is something I should be able to do without external help If it included the validity checks as well it'd definitely be worth it.

It includes the validity checks now, fully tested (it failed only when I used it on an .exe that had already been patched the old way, but I may account for that too). Regenerating the patch from a modded .exe is as simple as double-clicking on a batch file I've included. Updating the patch manually is just as easy. I've included instructions, which I'll replicate here:

newpatch_readme.txt
This folder contains the files needed for the executable patch. To generate a new patch from a changed .exe, double-click on newpatch.bat. Open the resulting new t-exe_patch.tpp with a text editor and delete the first line (with COPY ...) then save the file. You may wish to clean it up a bit (remove comments, etc.) as I did with the initial patch (which is now t-old_patch.tpp) but it is not really necessary except to reduce patch size.

Alternately, you can back up the existing patch manually and enter the following from a command prompt in your game folder (which is what newpatch.bat does):

weidu.exe --cmp-from BGMain_unpatched.exe --cmp-to BGMain.exe --out "infinityanimations\patch\t-exe_patch.tpp"

HOWEVER, it is much more efficient just to edit the existing patch directly to make changes. For example, the file contains the following syntax (example):

WRITE_BYTE 0x407438 0x00
WRITE_BYTE 0x40743a 0x81

Let's say you need to clear the byte at hex address 407439 as well. You simply enter here:
WRITE_BYTE 0x407439 0x00

And you can put this *anywhere* in the file, though it is more orderly I suppose to put it where it belongs, in between the two existing lines. If your text editor has a sort function (as MS Word does in its Table menu), you can put your patches in any order and just sort it when you're done editing (though again, this isn't strictly necessary except to avoid duplicate patching).

I might think about it once this implementation is definitely final (having to maintain two would be a nightmare!), but it's a big 'might' at this point.

Well I guess I can retain the "stacked" slots for those who have SoA and the new ones for ToB. Though now that I think about it, there are probably unused slots in SoA I could use (though I'm not sure which, offhand). Those folks on SoA won't have the same slot conflicts as the megamod folks.

If the _LOW patching component is installed after new animations were added, it might break their references (since it doesn't try to distinguish between 5xxx entries that were LOW entries and those that were actually IA references). If it's installed in the correct order, the current patch is indeed fine.

I've revised it so it can be installed in any order. Though the whole thing should be installed after other mods that add animations, so they don't overwrite slots (it'll be a distant day before all those megamods are updated for this new scheme).

The obvious missing function is to allow this to install actual content (animations). It'd be an easy task to make it so that *any* animations extracted to the infinityanimations/content folder (or whatever) will be installed with the mod. The only caveat is that you'd have to extract the content archives with a "Don't Extract Paths" option so they don't end up in a bunch of different subfolders. I suppose it could be possible to get WeiDU to look through a bunch of unknown subfolders, but it'd probably be best to avoid that.

Apart from the megamod CRE patch, there is one more component I can think of. Some of the megamods changed vanilla CREs to some other animation to "free up" slots they wrote new animations to. Well, we can restore these to what they should have been once the correct animations are in place.

Edit: regarding documentation... are you married to the purple background in your readme? I find it quite hard to read. I could convert it to the SHS template if you want, and update it with what I've done. Or there's also the readme generator.

Edited by Miloch, 04 December 2009 - 02:47 PM.

[Miloch]

So I think I have this pretty much coded. Dunno if it even parses yet. Haven't tested it at all. I guess I should (should I? ).

It includes the following so far:

Infinity Animations (main component, does all the checking, copying, patching necessary etc.)

Distinctive Genies (recolours efreet and dao... you're welcome, Arkenor )

Distinctive Fiends
- Balors get the proper male tanar'ri animation
- Some mariliths get the IWD marilith animation
- Darkens the tanar'ri (balor) animation of nabassu as per PnP
- Assigns yochlols a darkened otyugh animation (except those in drow form)
- Makes the chromatic demon somewhat chromatic
- Fade-demon (if present) gets a greyish marilith animation
- Glabrezu get proper animations instead of balor or cornugon

Correctly Coloured Abishai (recoloured white abishai currently - will be updated with red/green abishai when available)

Gelugons Get the Osyluth Animation

Pit Fiend Animations
1) All get the NWN animation
2) Some get the NWN animation

Cambion Animations
1) All get the IWD Isair animation
2) Some get the IWD Isair animation

Variable Goblins (uses all goblin animations for different creatures)

Svirfneblin Animations (like in Aurora only slotted rather than stacked)

Variable Lizardmen (shamans get lizardman2, captain gets lizard_king animation)

All is open to discussion. It's rather arbitrarily grouped and ordered now, and some folks might not like everything in a particular component (the Fiend one comes to mind). But I wanted to avoid the BG2 Tweaks-SCS "ask 100 questions to get a mod installed" syndrome. Not slamming those mods, they probably need to be the way they are, but excessive component splitting could really make this one tedious. The only reason I made the Gelugon and Abishai components separate from Fiends is because they requires non-standard content and have unique requisites (only games with certain mods will have gelugons anyhow). Still, flexibility rules, as long as it doesn't override usability.

The 2da files are the only things really missing that I can think of. They will get installed if packaged with any of the content files, but I imagine quite a few are missing. We can clone them from existing ones pretty easily, if someone can go through the slots and pick appropriate 2da files or sounds for them.

I have incremented the version number (=> r.09102253). Is there some significance to this or is it a joke? It's funny, but maybe v0.92 or whatever would be easier until a comfortable v1 is done.

I haven't done anything with documentation because you didn't say what you wanted to do with it. And well, it's always the last thing anyhow, isn't it. But it needs to be written for the above... probably should do some screen shots or frame samples too.

Also, there isn't any actual content packaged with this. Rather, it will detect if any required content is missing for whatever components and direct you where to get it and where to extract it.

[Arkenor]

I have incremented the version number (=> r.09102253). Is there some significance to this or is it a joke? It's funny, but maybe v0.92 or whatever would be easier until a comfortable v1 is done.

Looks like a date and a time to me.

I think Goblins should be the same colour set within any given area, unless there is a reason that multiple tribes are present.

Presumably the various IWD and IWD2 critters have sound effects and 2da back home in their original games that we can use. The NWN ones, maybe we'll need to do some jiggerypokery. What sort of sound format did they have?

Edited by Arkenor, 08 December 2009 - 08:16 AM.

[Erephine]

Yes, it is a time and date stamp.

Mm, don't go overboard with the .2da files just yet. The problem with assigning anything and everything via .2DA is, certain animations may share a .2DA (if they have the same prefix) currently, so the soundset will have to be kept fairly neutral and sounds assigned to the .cre instead.

I haven't really mentioned this anywhere because I'm still not sure whether I like it that way or not. It's the way the original game does this as well (e.g. Minotaur/Mind Flayer; Tanar'ri/Marilith; etc.) but it does limit you somewhat. I've thought about patching in a workaround but I'm still indecisive about the whole thing.

Miloch, the list looks fine, but remove components like these:

- Darkens the tanar'ri (balor) animation of nabassu as per PnP
- Assigns yochlols a darkened otyugh animation (except those in drow form)
- Makes the chromatic demon somewhat chromatic
- Fade-demon (if present) gets a greyish marilith animation
Distinctive Genies (recolours efreet and dao... you're welcome, Arkenor )
Correctly Coloured Abishai (recoloured white abishai currently - will be updated with red/green abishai when available)

Effect/item recolours aren't really part of this mod.

-edit: Just thinking about it; you could move those to a separate installer maybe?

Edited by Erephine, 08 December 2009 - 08:52 AM.

[Arkenor]

How well is the exe patch going to work with no-cd executable patches? If they won't work together, do you think it might be worth including one?

I guess it's already working with the widescreen patch.

Distinctive Genies (recolours efreet and dao... you're welcome, Arkenor smile.gif )

That's great and all, but it was kind of MY project. Still, I guess it's done now, and probably about a hundred times faster than I'd have got it finished.

Edited by Arkenor, 08 December 2009 - 09:34 AM.

[Miloch]

Effect/item recolours aren't really part of this mod.

-edit: Just thinking about it; you could move those to a separate installer maybe?

But they are separate, optional components. Move them to a separate installer? You mean another mod? How is that different from removing them?

This does nothing with items. It just assign creatures somewhat more appropriate animations, as does the rest of the mod. If they had separate ones, I'd use those. Also, all those components require the main component (except Distinctive Genies, which does not require new animations). Moreover, the Abishai will get proper animations just as soon as you've slotted them, so why remove it just to put it back? It's mainly a placeholder for now, and one that requires very little changing to update. (For that matter there's about 5 lines of commented-out code for the Lady of Pain when that's available.)

If I spun it off to another mod, it'd probably end up in a tweak mod I won't be able to finish anytime soon (a number of components of which are, currently and rather inappropriately, in Aurora). Maybe I can peddle it to Avenger's aTweaks, since it already has a similar component. Though it'd need to require this as a prerequisite.

Distinctive Genies (recolours efreet and dao... you're welcome, Arkenor smile.gif )

That's great and all, but it was kind of MY project. Still, I guess it's done now, and probably about a hundred times faster than I'd have got it finished.

For all I care, you can have it. You didn't indicate you were going to definitely do something with it or even that you had any source files anymore. Plus, what you described (combat tweaks) does indeed go beyond the scope of this mod, though I don't think modding animations does. But whatever. The main reason I put it in there (and I said I might, to no objections) is so you could take a look and tweak it if you like, to see if it's something you could work with. And, well, I think it's a good idea and one that seems to be lacking.

I'm not trying to hijack this mod or pinch someone's work. I don't even care if I'm mentioned in the credits. Just trying to help. I wonder why I spend hours poring through spreadsheets and WeiDU code... some sort of masochism I guess. Such gratefulness .

How well is the exe patch going to work with no-cd executable patches? If they won't work together, do you think it might be worth including one?

I don't know, but I suspect it works with those that don't increase the .exe size (it checks for that). I don't think we want to include one because some people get uneasy with that sort of thing. But for all I know it could already have one - I just replicated the exact same patch Erephine had.

I guess it's already working with the widescreen patch.

If you say so. I don't use WSM. It's not needed with my monitor, but I haven't tested any of this one way or another.

Mm, don't go overboard with the .2da files just yet. The problem with assigning anything and everything via .2DA is, certain animations may share a .2DA (if they have the same prefix) currently, so the soundset will have to be kept fairly neutral and sounds assigned to the .cre instead.

I haven't really mentioned this anywhere because I'm still not sure whether I like it that way or not. It's the way the original game does this as well (e.g. Minotaur/Mind Flayer; Tanar'ri/Marilith; etc.) but it does limit you somewhat. I've thought about patching in a workaround but I'm still indecisive about the whole thing.

Tutu handles this by wiping the 2da files and assigning sounds directly on the creature files. Not sure if it's the best solution either, but I guess it's possible since we're patching all the CREs anyway for animations, but we'd need to know all the proper sound references, and port missing ones where necessary. Now, dealing with sounds certainly seems as if it would go beyond the scope here, but unfortunately animation slots and sounds are intrinsically linked. They're also linked to creature genders, I believe.

[Erephine]

Tutu handles this by wiping the 2da files and assigning sounds directly on the creature files. Not sure if it's the best solution either, but I guess it's possible since we're patching all the CREs anyway for animations, but we'd need to know all the proper sound references, and port missing ones where necessary. Now, dealing with sounds certainly seems as if it would go beyond the scope here, but unfortunately animation slots and sounds are intrinsically linked. They're also linked to creature genders, I believe.

No, animation soundsets are not linked to gender.

Can you set walking sounds from .cre files? If you can that sounds like a good idea. I don't think they have to be incredibly accurate. I'm testing without .2DA files in my override (so they all link to the same sound information for the slot) with player characters (so some .cre references are there) and it seems fine apart from those for the most part.

[Miloch]

No, animation soundsets are not linked to gender.

I meant that creature gender (the byte at 0x275) determines the casting voice (as noted here). I assume the actual references are hardcoded somewhere. The game also checks it for the summoning cap, so this is a bit of an engine bug - you can't have, say, a summoned marilith (an obviously female demon) without it sounding like a bloke (e.g. voicing the cas_pm1m.wav clip when casting, instead of cas_pm1f.wav). There is probably some hackaround for this (like putting a script or time limit on your demon) but apart from that, you want to make sure creature genders are set correctly where it's clear which they should be.

Can you set walking sounds from .cre files? If you can that sounds like a good idea. I don't think they have to be incredibly accurate. I'm testing without .2DA files in my override (so they all link to the same sound information for the slot) with player characters (so some .cre references are there) and it seems fine apart from those for the most part.

I don't know. Where are they set normally? Are these the wal_*.wav clips? There's no obvious soundslots for them, but they don't appear in any of the animation .2da files I looked at either. Walking sounds are normally very low in volume (and you can turn them off in configuration) but if I crank the volume up in Tutu, I still hear them even with blank .2da files. I'm guessing the engine probably handles those too, since I'd doubt the CRE files do.

If you want to go this route, we would need some sort of list of default sounds, for the new animations at least. It could be coded such that if a CRE doesn't have a sound assigned in a standard slot (attack, damage, etc.), it could get the default one (there are READ/WRITE_SOUNDSET macros in WeiDU too but that route could be messy, particularly if someone's botched the sound references for vanilla CREs, which I wouldn't doubt).

I don't know how either IWD handles creature sounds. There's no equivalent .2da files, and the few CREs I looked at (such as svirfneblin) had nothing slotted for sounds - only a few animals like bears did (though I only looked at a few CREs). And I wouldn't know where to start looking in NWN - don't even have it installed, though I think I do have it somewhere.

[Erephine]

If there's no .2da file, the game uses the sounds specified in the .exe. The .2da is sort of an 'override folder' for the information in there.

I'm assuming walking sounds can be set from the .2da because it does have entries for them:

WALK ***
WALK 0

[Miloch]

I'm assuming walking sounds can be set from the .2da because it does have entries for them

I would assume so too, but I didn't see any 2DAs that actually had them set. Most walking sounds shouldn't sound too different anyway, but I suppose there should be some difference between a marilith "slithering" and a balor tanar'ri "hoofing it" or whatever. In which case, we're hosed either way since the MTAN.2da would apply to both, but it can't be any worse than the existing situation, unless those walking sounds are hardcoded in the .exe by gender (like casting sounds) or something else. And if so, then why bother messing with those particular sounds?