216 replies to this topic

[Helborn]

There are a large number of files that get modified over and over during a megamod. During installation, TDD, SOS, CtB, RoT, NEJ, DSotSC, Bonehill, and NTotSC and at least one other all biff everything in the override folder. Which means that later mods must bring them back (copy them to Override) to make their changes. This means that certain files wind up in several bif files. Which means that the chitin.key & dialog.tlk get overwritten several times with essentially the same reference data (same file, new location).

Wouldn't it be easier to remove the biffing actions and let The Bigg's generalized biffing do it all? It would also save time and space in the long run for an install.

Even BGT could leave the files in the override until biffing, which would allow mods simple access to all the BG1 files

The same can be said of all the ogg and tiz files - there is no reason to unpack them on a per mod basis. Since all of the unpacking is done via batch files it's not unreasonable to make the changes to the tp2 files and eliminate the call for running the install bats. It would also mean that the latest versions of oggdec and tisunpack could be used which would mean a smaller batch file with wild cards.

The main change to the tp2's would be to put all new files into override instead of the specialized Folders. This would require a larger effort, but once the folder names are identified it only requires a global replace to fix and most mods do the move in the batch file anyway.

[Lollorian]

I love the idea (always wondered about this, but gave up thinking "They're BIG, they probably require it or something" )

Which means that the chitin.key & dialog.tlk get overwritten several times with essentially the same reference data (same file, new location).

I think the new reference overwrites (or takes precedence over) the old one otherwise, we'll see dupes of Irenicus, Ellesime and whatnot

On another note, I've seen many mods introduce strings (in the .tra's) that are already present in vanilla BGII-ToB, especially the unidentified item strings. Can something be done to remove the dupes from the dialog.tlk??

Cheers,
Lol

EDIT: The results of my experiments in this thread (if you don't wanna go through boring lists and stuff ) can now be found in conveneient patchy form in the form of the BWPTrimpack!

If you like history and stuff, read the thread

Edited by Lollorian, 04 April 2010 - 09:52 AM.

[Jarno Mikkola]

This could be done in the BiG World Fixpack's .tp2 fixes, or at the mod release themselves, but it would require a hell of a lot testing, about a 39,916,800 times. Narf.

If it's good to know what you are looking for, the bigg said:

Generalized Biffing can be uninstalled and reinstalled like any other mod. Of course, the various mods that use AT_EXIT ~weidu --make-biff~ rather than MAKE_BIFF cannot be uninstalled that easily (all the big BG1 and BG2 mods for instance).

So potentially, if you find the AT_EXIT ~weidu --make-biff~, you just remove that and redirect the COPY'es etc. to the ~override~ instead of the set folder, and you are quite much set.

During installation, TDD, SOS, CtB, RoT, NEJ, DSotSC, Bonehill, and NTotSC

And TS-BP, BP, Drizztsaga, BG1NPC... if my info is correct.

And let's remember with the current Biffing Scheme, the game takes about 5 to 10 minutes to start if you do not use the Generalized Biffing, so, so, t might take a bit more without all the biffing, and lets remember that the whole original game data is also biffed.

The good thing about this is that as one now can uninstall the Generalized Biffing...

Edited by Jarno Mikkola, 17 November 2009 - 11:02 PM.

[Wisp]

On another note, I've seen many mods introduce strings (in the .tra's) that are already present in vanilla BGII-ToB, especially the unidentified item strings. Can something be done to remove the dupes from the dialog.tlk??

If a new string is identical to an old string, WeiDU will use the old string instead of adding the new string.

[Helborn]

This could be done in the BiG World Fixpack's .tp2 fixes, or at the mod release themselves, but it would require a hell of a lot testing, about a 39,916,800 times. Narf.

I really don't think that it would require quite that much testing since I have been doing it for my manual installs for a while now.

I am quite willing to undertake the project to see what changes exactly need to be made.

Whether to add it to the BW Fixpack or create replacement tp2's that could be offered as a separate download I will leave to Leomar and the rest of the BW group.

[Helborn]

Which means that the chitin.key & dialog.tlk get overwritten several times with essentially the same reference data (same file, new location).

I think the new reference overwrites (or takes precedence over) the old one otherwise, we'll see dupes of Irenicus, Ellesime and whatnot

My understanding is that the new reference takes precedence but does not overwrite, which is why they both keep growing and growing and growing. This way, the Vanilla stuff would get copied once and the mod stuff would be introduced once and the generalized biffing would package it all up very nicely at the end. (My hat is off to you Bigg that is one nice piece of coding).

This is not for those who continually add and subtract mods, but for folks like me who want to play one massive game then reinstall for a different massive game (I do Nej and RoT separate from other games)

[Hoppy]

Which means that the chitin.key & dialog.tlk get overwritten several times with essentially the same reference data (same file, new location).

I think the new reference overwrites (or takes precedence over) the old one otherwise, we'll see dupes of Irenicus, Ellesime and whatnot

Cheers,
Lol

I think the multiple spawns of creatures is not due to having them in different BIFS and if it does contribute it is more rare than say scripting conflicts or a looped script. I have had these doubles before but could never reproduce them no matter what mod introduced the CRE/material or even if it was a vanilla game. Something there just points to a script that does not finish and update the global or two scripts that are almost the same and thus both fire.

My game is biffed twice, once at the end of the whole installation and then again after I had added mods and made improvements and fixes. and it does not run any differently than with one BIF. Not better and not worse. Now with this in mind I have files in all different BIFs and it is not giving me any problems. Do you want to do all the work of reorganizing TP2's to reformat the chitin.key when it really may have a minor improvement?

There is a saying that "if it isn't broke, don't fix it."


@helborn
Chitin.key does not get overwritten, it gets patched by respected mods that add BIF files to data (TDD, SOS. CtB, etc.) same as dialog talk if we are talking about current mods that patch dialog.tlk. When it is patched it just points to the new location of the reference.


EDIT:

I think reinstalling a whole new modded game is a lot easier by making a complete copy of the vanilla game folder and use that for the new installation. Faster than uninstalling one medium size mod.

Edited by Hoppy, 18 November 2009 - 09:03 PM.

[Jarno Mikkola]

My game is biffed twice, once at the end of the whole installation and then again after I had added mods and made improvements and fixes. and it does not run any differently than with one BIF. Not better and not worse. Now with this in mind I have files in all different BIFs and it is not giving me any problems. Do you want to do all the work of reorganizing TP2's to reformat the chitin.key when it really may have a minor improvement?

Yeah, but then you can't use any of the mods in the list: TDD, SOS, CtB, RoT, NEJ, DSotSC, Bonehill, and NTotSC, TS-BP, BP, Drizztsaga and BG1NPC...
As they all biff some or most of their files before you make the grand Gen_Biff:ing, or you'll end up biffing it 14 times at least.

We are talking about leaving the mods own biffing not done, so when we install the Megamod, we just do the Biffing once at the end. The benefit it gives use is less time consumed on the Biffing, hard drive space saving, and you also get better --change-log's, and if you don't want to do that, you get better results with the Clean-up.bat .

I think reinstalling a whole new modded game is a lot easier by making a complete copy of the vanilla game folder and use that for the new installation. Faster than uninstalling one medium size mod.

Eeaah, no, as if you need to just uninstall one mod, it's faster to uninstall all the previous mods and then remove the one and reinstall the rest, than start from fresh install, even with BWS... we are talking about Megamods here... in most cases, if none of the mods use the old biffing function that cannot be uninstalled by WeiDU.exe . Of couse what you probably meant is that it's easier to use backups... but when we are installing 300 mods, one cannot expect the gamer to have that many current backups...

I really don't think that it would require quite that much testing since I have been doing it for my manual installs for a while now.

I am quite willing to undertake the project to see what changes exactly need to be made.

Whether to add it to the BW Fixpack or create replacement tp2's that could be offered as a separate download I will leave to Leomar and the rest of the BW group.

Well, if you can provide us the laitest mod versions .tp2's that you have 'fixed', I am sure the BWP team will be able to differentiate the needed lines for the BWFixpack (so put the .tp2 into an archive, and upload the file to this thread).

Edited by Jarno Mikkola, 19 November 2009 - 04:41 AM.

[Leomar]

We are talking about leaving the mods own biffing not done, so when we install the Megamod, we just do the Biffing once at the end. The benefit it gives use is less time consumed on the Biffing, hard drive space saving, and you also get better --change-log's, and if you don't want to do that, you get better results with the Clean-up.bat.

Hmmm, normally it is not less time consuming (it makes no different if you biff at the end or step by step). The biff process is always normally the same. And the installation would be bigger and bigger, because all files were backuped (what is really not needed). And old computers could freeze, because at the end there are to much files to biff (the memory gets to big).

At the moment we see no sense to change the mods which biffed itself. Normally it is good, that the big mods and BGT does this...

Greetings Leomar

Edited by Leomar, 25 November 2009 - 12:10 AM.

[Lollorian]

Actually, lemme put forth this (hypothetical ... I think ) situation:

• Suppose megamods M1 and M2 biff their files at the end of each one's installation, so now all their creatures, spells, and stores are inside their own bifs like M1_CRE.bif, M1_SPL.bif, M2_STO.bif etc etc.
  
• Now, the override doesn't have any cre's, spl's or sto's from M1 and M2 (since they're now in the data folder as bif's)
  
• In comes a HUGE cre/spl/sto patcher (P5Tweaks?? SCS?? Crefixer?? LOL ) ... so it takes each cre/spl/sto from each .bif, makes its patchety-patches, and puts the patched cre/spl/sto into the override
  
• After everything's done, in comes Gen_Biff and gobbles up all the stuff in the override and puts them into one of its many trademark TB#XXX.bif's

I dunno if this hypotherical situation actually happens in the BWP (although I'm kinda sure it does ) But, dya see what's kookie here??

Since all the cre/spl/sto's from M1 and M2 are patched and re-biffed by Gen_Biff into the TB#XXX.bif's, the original cre/spl/sto's in the M1_XXX and M2_XXX.bif's are now redundant, and hence just take up space (which is why the Cleanup,bat was created for ... to minimize install size )

(Although, yeah, the memory issue would seem to be a problem for HUGE biffjobs like a 13 GB override )

Cheers,
Lol

Edited by Lollorian, 25 November 2009 - 12:33 AM.

[Jarno Mikkola]

We are talking about leaving the mods own biffing not done, so when we install the Megamod, we just do the Biffing once at the end. The benefit it gives use is less time consumed on the Biffing, hard drive space saving, and you also get better --change-log's, and if you don't want to do that, you get better results with the Clean-up.bat.

Hmmm, normally it is not less time consuming (it makes no different if you biff at the end or step by step). The biff process is always normally the same. And the installation would be bigger and bigger, because all files were backuped (what is really not needed). And old computers could freeze, because at the end there are to much files to biff (the memory gets to big).

Well, if the files is edited a single time, and then biffed, then the time consumed is the same, but if it changed multiple time(by different mods), and each time biffed into separate .bif file, it will take as much or at least the almost the same amount of space even with the backups counted, and as the Clean-up.bat can erase the backups, but it can't remove the .bif file themselves, as some of the info would still be needed, and what comes to the timing, multiple Biffings always loose.

Biffing only once is cleaner solution in my mind, and there will be much less game data to be read, when they are in a clean package, and not in 23 packages, that all have correct data but only in some sections, while the others are just spare data left by the other installs that has been 'overwritten' by the others.
It's like having 5 years old 1TB's hard drive always in use, and so it has never been De-fragmented, and now you try to find the 100MB's that was written last week. You'll have to find the data with a torch and a hand pick, good luck.

Edit: Lollorian is very right about the situation...

Edited by Jarno Mikkola, 25 November 2009 - 12:55 AM.

[Helborn]

We are talking about leaving the mods own biffing not done, so when we install the Megamod, we just do the Biffing once at the end. The benefit it gives use is less time consumed on the Biffing, hard drive space saving, and you also get better --change-log's, and if you don't want to do that, you get better results with the Clean-up.bat.

Hmmm, normally it is not less time consuming (it makes no different if you biff at the end or step by step). The biff process is always normally the same. And the installation would be bigger and bigger, because all files were backuped (what is really not needed). And old computers could freeze, because at the end there are to much files to biff (the memory gets to big).

At the moment we see no sense to change the mods which biffed itself. Normally it is good, that the big mods and BGT does this...

Greetings Leomar

I am in the process of editing the tp2's to reflect no biffing (and to move all ogg and tiz processing to the end) (am doing it for the latest versions of every mod I have downloaded). Am I correct in understanding that there is no longer any reason to backup the override folder? That is an easy addition to the changes I am making and would definitely speed up the installation.

Edited by Helborn, 25 November 2009 - 12:19 PM.

[Leomar]

Thanks for your feedback and thank you, Helborn, for doing the job.

I think, if Helborn finished the job, we can test and see how long it will need to biff the whole megamod with Generalized Biffing. Additional we'll add a note in the Install.bat, so the user get the info that the biffing needs a lot of time during Generalized Biffing does its job. At the end, it sounds after a good idea and we hope that the memory on older computers can handle this.

Greetings Leomar

[Lollorian]

I recently put together an install with basically anything I could think of that pulls things outta .bif's and puts them into the override, but don't add content, stuff like SCS, BG2Fixpack, BG2Tweaks, RR, aTweaks etc. When I finished, I gen_biffed them all to see the difference (took 15 minutes)

Vanilla Install: (before Tweaking)
Data - 939 MB
Override - 25.7 MB

After Tweaks:
Data - 1.52 GB
Override - 202 kB

So, final data folder (1556.5 MB) is almost double of initial data+override (964.7) That's almost 600 MB of wasted space!!! (well, let's say 100 MB goes to the new spells/other content that gets added ... but still 500 MB!!! )

But, interesting paradox here, this space gain will be GREAT for older systems (with not-so-huge-disks ), but at the same time, it's the memory of the older systems that prevents it from happening

Another thing, the 202 kB in my final override contains stuff I'm not familiar with: .mrk, .xxx, .g3, .slh, .rr, .txt's, some .baf's, .d and a .dlg (wonder why it didn't get biffed ... MONKTU 8.DLG ... is it the extra space in the name??)

Saw that the file gets copied by BG2_Tweaks, so attached its .DEBUG

Cheers,
Lol

Attached Files

•  SETUP_BG2_TWEAKS_DEBUG.7z   85K   672 downloads

[Jarno Mikkola]

Another thing, the 202 kB in my final override contains stuff I'm not familiar with: .mrk, .xxx, .g3, .slh, .rr, .txt's, some .baf's, .d and a .dlg (wonder why it didn't get biffed ... MONKTU 8.DLG ... is it the extra space in the name??)

Well, the .txt's, .g3's and .xxx's probably aren't game files themselves, while the .baf and .d are not-compiled game scripts, and dialog files... so the files just shouldn't be copied over to the override folder.

[Leomar]

Another thing, the 202 kB in my final override contains stuff I'm not familiar with: .mrk, .xxx, .g3, .slh, .rr, .txt's, some .baf's, .d and a .dlg (wonder why it didn't get biffed ... MONKTU 8.DLG ... is it the extra space in the name??)

Well, the .txt's, .g3's and .xxx's probably aren't game files themselves, while the .baf and .d are not-compiled game scripts, and dialog files... so the files just shouldn't be copied over to the override folder.

That's right and therefore we delete them with the BW Clean-Up.bat:

:: Step 2
:: Now files that should not be in the override will be deleted. Don't worry about the "not found" messages!

SET OV=.\override\

@echo off
FOR %%f IN (
%OV%z#misc.*
%OV%*.aup
%OV%*.B
%OV%*.baf
%OV%*.bak
%OV%*.bat
%OV%*.d
%OV%*.exe
%OV%*.g3
%OV%*.ks
%OV%*.mp3
%OV%*.mrk
%OV%*.ogg
%OV%*.rpgd
%OV%*.RR
%OV%*.sfk
%OV%*.slh
%OV%*.tiz
%OV%*.txt
%OV%*.UB
%OV%*.xxx
%OV%.DS_Store
%OV%AR0500-bcs
%OV%fhlplay.fh
%OV%BBlade1
%OV%BBlade2
:: %OV%BP-BGT_WM6.wm6
%OV%CELES
%OV%chitem.log
%OV%CLOUDKA
%OV%Desktop.ini
%OV%DFOGA
%OV%dummy.blarg
%OV%dw#fixasc
%OV%dw#fixkelsey
%OV%dw#imsta
%OV%dw#restorestring.tpa
%OV%ESHIELC
%OV%firebrn
%OV%ICARMOR
%OV%ICINHIT
%OV%ICWRATI
%OV%ION1
%OV%ION1B
%OV%ION1BZ
%OV%ION1G
%OV%ION1GD
%OV%ION1S
%OV%ION1W
%OV%ION2
%OV%ION2B
%OV%ION2BZ
%OV%ION2G
%OV%ION2GD
%OV%ION2S
%OV%ION2W
%OV%IOUNXA7
%OV%IOUNXA8
%OV%IOUNXA9
%OV%LPOOMV
%OV%otthelm
%OV%PGDARN.bam.gif
%OV%PSHELLC
%OV%RPGSolaFlirtPack
%OV%S#Eye
%OV%sharea
%OV%spheart
%OV%spwi910cre
%OV%spwm104
%OV%SSHELLC
%OV%TRBLOBA
%OV%va#tod.nul
%OV%"Service-Version 1.txt"
) DO del %%f 2>nul

Greetings Leomar

[Jarno Mikkola]

...

That's right and therefore we delete them with the BW Clean-Up.bat:

But could you do a small BWS related fix mod/tool that just deletes those file, while not all the others... you can set the BWS/BiG World Install.bat to run it before the Generalized Biffing. Call it Pre-Bif.bat or something.

By the way, I am quite sure the MONKTU 8.DLG is a discrazed game file that the Generalized Biffing does not pack to the .bif cause it does not have a proper name syntax(a,b,c...), and someone should --change-log the file(to find out where it comes from) and then ask the mod maker to adjust the files name to not have a space in it.

[the bigg]

Suggestion: if memory consumption is a problem for lower end computers, each big mod can still use gen_biff's code for biffing its own tis/wav/bam files; that way, only gen_biff will have to biff the various cre/itm/spl/bcs/dlg files (and the tis/wav/bam files from various small mods), and most importantly the various cre/itm/spl/... files will be biffed no more than twice (once in the standard game biffs, once by gen_biff).

As Lollorian says here, 90% of all cre/itm/... files biffed by BGT will be brought in the override by one of the 30000 existing tweak mods no matter what, so them being biffed by <big mod> will have no real advantage for any installation worthy of consideration (and you can still have <big mod> retain its old school biffing and use BWFixpack to change it to tis/wav/bam gen_biff-style biffing, so that the freaks of nature playing only <big mod> will still have <big mod>'s files biffed).

Edited by the bigg, 01 December 2009 - 03:19 AM.

[Lollorian]

About MONKTU 8.DLG, it can't be changelogged because of the space (the .bat tries to changelog 8.DLG)

Anyway, as seen in the debug file, it gets copied by BG2Tweaks, due to this regex:

COPY_EXISTING_REGEXP GLOB ~^.+\.dlg$~ ~override~
  PATCH_IF (SOURCE_SIZE > 0x2f) THEN BEGIN // protects against invalid files
	READ_LONG  0x0c "offset_state"
	PATCH_IF ("%offset_state%" = 0x30) BEGIN // adds pause field to older dlg formats
	  READ_LONG 0x14 "offset_response"
	  READ_LONG 0x1c "num_s_trigger"
	  READ_LONG 0x18 "offset_s_trigger"
	  READ_LONG 0x20 "offset_r_trigger"
	  READ_LONG 0x24 "num_r_trigger"
	  READ_LONG 0x28 "offset_action"
	  READ_LONG 0x2c "num_action"
	  FOR (index = 0; index < num_s_trigger; index = index + 1) BEGIN
		READ_LONG  ("%offset_s_trigger%" + ("%index%" * 0x08)) "indiv_offset"
		WRITE_LONG ("%offset_s_trigger%" + ("%index%" * 0x08)) ("%indiv_offset%" + 0x04)
	  END
	  FOR (index = 0; index < num_r_trigger; index = index + 1) BEGIN
		READ_LONG  ("%offset_r_trigger%" + ("%index%" * 0x08)) "indiv_offset"
		WRITE_LONG ("%offset_r_trigger%" + ("%index%" * 0x08)) ("%indiv_offset%" + 0x04)
	  END
	  FOR (index = 0; index < num_action; index = index + 1) BEGIN
		READ_LONG  ("%offset_action%" + ("%index%" * 0x08)) "indiv_offset"
		WRITE_LONG ("%offset_action%" + ("%index%" * 0x08)) ("%indiv_offset%" + 0x04)
	  END
	  WRITE_LONG 0x0c ("%offset_state%"	 + 0x04)
	  WRITE_LONG 0x14 ("%offset_response%"  + 0x04)
	  WRITE_LONG 0x18 ("%offset_s_trigger%" + 0x04)
	  WRITE_LONG 0x20 ("%offset_r_trigger%" + 0x04)
	  WRITE_LONG 0x28 ("%offset_action%"	+ 0x04)
	  INSERT_BYTES 0x30 4
	END ELSE BEGIN // otherwise just sets to pause
	  WRITE_LONG 0x30 0
	END
  END
  BUT_ONLY_IF_IT_CHANGES

and the result being:

...
Copied [MONKEN.DLG] to [override/MONKEN.DLG]
Copied [MONKTU 8.DLG] to [override/MONKTU 8.DLG]
Copied [MONKTU1.DLG] to [override/MONKTU1.DLG]
...

Now, funny thing is, the CHITIN.KEY holds a reference to that file, and further investigation shows that it's also present in vanilla BGII-ToB and it has no reference in any other file (shouldn't the BG2Fixpack fix stuff like this?? )

So, the problem ultimately lies in vanilla BGII's chitin.key (it's probably safe to delete, non??)

PS: The file only has one state, "It's not as much as the fiend had, but it's plenty, and it's at a significant discount for you. Just speak to Bernard...I still have him handling the store." which is also found in a MONKTU8.DLG (without the space )

And before I forget, I totally agree with jarno's suggestion of an "Override Cleanup.bat" (but is it necessary for it to run before biffing?? I don't think those kinds of files get biffed )

Cheers,
Lol

Edited by Lollorian, 01 December 2009 - 05:08 AM.

[Leomar]

Now, funny thing is, the CHITIN.KEY holds a reference to that file, and further investigation shows that it's also present in vanilla BGII-ToB and it has no reference in any other file (shouldn't the BG2Fixpack fix stuff like this?? )

So, the problem ultimately lies in vanilla BGII's chitin.key (it's probably safe to delete, non??)

PS: The file only has one state, "It's not as much as the fiend had, but it's plenty, and it's at a significant discount for you. Just speak to Bernard...I still have him handling the store." which is also found in a MONKTU8.DLG (without the space )

Nice catch. Lollorian, can you report this at the BG2 Fixpack forum?

Greetings Leomar

Edited by Leomar, 02 December 2009 - 02:31 PM.