19 replies to this topic

[Sam.]

My little pet project, PS BAM, is now on GitHub.  It is an Infinity Engine BAM compressor, builder, and editor.  If you're interested, go read the readme and give it a try.  Feel free to ask questions, make requests, and let me know here when it catches your computer on fire.  Use at your own risk!

 

Happy modding,

Sam.

[Sam.]

Real world use cases
 
I understand that the full utility of PS BAM may not be apparent at first glance, and also that the particulars of the parameters and their options may not be overly intuitive.  In fact, many seem to find using command-line utilities a rather daunting prospect in general.  Because of this, I have reserved this post to demonstrate real use cases as they are requested or as I encounter them.  As I have said before, if there is something you'd like PS BAM to do, just ask and I'll be happy to explain how it is done or do my best to implement the requested behavior.  With that said, here are some examples:
 
1)  Montage – Combining and Extracting Paperdoll Images
 
Background
I wrote a tool, PaletteGenerator (Forum), that can take an uncompressed, 8-bit, 256-color, paletted BMP V3 file and apply Infinity Engine color gradients to it. 

Realistically, we would probably want to do this to inventory paperdoll BAMs in order to see the effects of applying different character or item colors without having to load the game on each iteration.  The problem is that we need to convert inventory BAMs into a single frame BMP for use in PaletteGenerator.  The problem becomes more complex because in the inventory paperdoll BAMs, the actual image of the paperdoll is broken into two frames, a top half and bottom half, and then the transparent pixels around each are cropped.
 
In order to make them a single image, we'll need to un-crop them and combine them (easier said than done).  PS BAM can do this!
 
Usage
I extracted all of the *INV.bam files from ToSC into a folder ("C:\Users\Sam\Desktop\tmp test\BG1 INV"), then ran the following to extract them as inventory paperdolls padded to the proper dimensions of a paperdoll.

"D:\AutoHotkey Scripts\PS BAM\PS BAM_x64.exe" --Compress 1 --CompressionProfile Quick --DropUnusedPaletteEntries 0 --DropDuplicatePaletteEntries 0 --FixPaletteColorErrors 1 --TrimFrameData 0 --IntelligentRLE 0 --Montage "1x2" --OrderOfOperations "CPE" --ExportFrames "BMP,8V3" --DebugLevelL 1 --DebugLevelP 2 --DebugLevelS 1 --LogFile "C:\Users\Sam\Desktop\tmp test\Log.txt" --OutPath "C:\Users\Sam\Desktop\tmp test\BG1 INV\extracted" --Save "" "C:\Users\Sam\Desktop\tmp test\BG1 INV\*.bam"

 
So what the hell is all this?

"D:\AutoHotkey Scripts\PS BAM\PS BAM_x64.exe"

The path to the PS BAM executable.  Use the 64-bit one if you can (it should be a bit faster).

--Compress 1 --CompressionProfile Quick

We want to enable some basic compression to pre-process the frames before they will be combined.  CompressionProfile can be Max, Recommended, Safe, Quick, or None.  We choose Quick because we want to do some basic things like drop unused frames, but don't want to spend too much computing power achieving optimum compression (as we don't need to save the compressed BAMs anyway).  CompressionProfile is the only input that is position dependent (you can overwrite its defaults by by putting relevant parameters after it).

--DropUnusedPaletteEntries 0 --DropDuplicatePaletteEntries 0

These are paletted BAMs, so we don't want to mess with the palette.  The "Quick" CompressionProfile would have auto-detected that they are paletted and done the same thing, but we can can manually specify it to be sure.

--FixPaletteColorErrors 1

This will fix the BAMWorkshop teal and pink transparent and shadow colors and set them to the proper green and black.  This is the only palette edit we want to make.

--TrimFrameData 0

This turns off trimming of excess transparent pixels around the frames.  The Montage function already performs its own trimming internally so we can save a bit of time by not trying to do it twice, but more importantly we do not want to perform any "extra" trimming of the frames because for these images, we are likely to delete "good" pixels.

--IntelligentRLE 0

As will be specified later, we do not need to save the BAMs to file, so applying run-length-encoding of the frames is useless.  More importantly, frames can only be manipulated (montaged, exported, etc.) BEFORE applying RLE.  As will be seen later, we will be compressing, and then processing (Montage is included in this step) and then exporting.  Therefore, we want to disable RLE.

--Montage "1x2"

Montage is what combines the multiple frames into one.  The format is RowsxColumns.  However, "1x2" is the only value currently supported.  Give me a specific use-case for other values, and I'll see about implementing them.

--OrderOfOperations "CPE"

There are three distinct steps that can be performed in any order.  C is compression, P is processing (montage happens during this step), and E is export.  Here we are compressing first to perform some pre-processing, then processing to montage the frames together, then exporting the montaged frames.

--ExportFrames "BMP,8V3"

Frames can be exported in a wide variety of formats.  In this case, we will be exporting them as Bitmaps [BMP] that are 8-bit (256 color and uncompressed) [8] with a BMP V3 header.  This is the format the PaletteGenerator expects.

--DebugLevelL 1 --DebugLevelP 2 --DebugLevelS 1

This tells the program how much debug output to display.  DebugLevelL is for the loading step, DebugLevelP is for the processing step (which includes compression), and DebugLevelS is for the saving step.  Values of:

• -1 = very silent (no log)
• 0 = Silent  (errors only)
• 1 = Normal  (errors and warnings)
• 2 = Verbose (errors, warnings, extra info)

--LogFile "C:\Users\Sam\Desktop\tmp test\Log.txt"

Where to save the log file.

--OutPath "C:\Users\Sam\Desktop\tmp test\BG1 INV\extracted"

Where to put all the output files including exported frames, palettes, sequences, and any saved files like BAMs, BAMDs, GIFs, etc.

--Save ""

What format to save the processed/generated BAMs in.  In this case, blank is none.

"C:\Users\Sam\Desktop\tmp test\BG1 INV\*.bam"

Very last should come the files to process.  Wildcards are accepted, and any number of input files can be specified.
 

Result
The end result may look something like this:

Now, using an image editor that can make the background color transparent and maintain the same palette, we can directly overlap the paperdolls like so:
+ + + =

Edited by Sam., 18 November 2018 - 03:01 PM.

[Sam.]

New Version: v0.0.0.3a

I improved montage and replaced the --CompressFirst and --ProcessFirst options with the hopefully more useful/intuitive --OrderOfOperations.  Parameters for this option are any ordering of P, C, and E:

P = Process; C = Compress; E = Export; in any order.  e.g. | CPE | CEP | PCE | PEC | EPC | ECP |

Usage would be e.g.

 

--OrderOfOperations "PCE"

but could even be something like

 

--OrderOfOperations "CPCE"

to Compress, Process, Compress again, and finally Export.  In this case, however, you would probably want to disable RLE (because reasons) via

 

--IntelligentRLE 0

 

[Sam.]

Real world use cases
 
I understand that the full utility of PS BAM may not be apparent at first glance, and also that the particulars of the parameters and their options may not be overly intuitive.  In fact, many seem to find using command-line utilities a rather daunting prospect in general.  Because of this, I will demonstrate real use cases as they are requested or as I encounter them.  As I have said before, if there is something you'd like PS BAM to do, just ask and I'll be happy to explain how it is done or do my best to implement the requested behavior.  With that said, here are some examples:
 
2)  Very Good (Hyper) Compression of BAM V1 Files
 
Background
I needed to add three new BAM files to the Thrown Hammers mod, and wanted to compress them as much as possible to save space and minimize download bandwidth.  PS BAM can achieve very good compression at the cost of processing time.
 
Usage
I copied the three BAMs I wanted to compress to "C:\Users\Sam\Desktop\TH_Dark_Theme\" and ran the following:

"D:\Program Files\GitHub Projects\PS-BAM\PS BAM_x64.exe" --AdvancedFLTCompression 1 --AdvancedZlibCompress 2 --AllowShortPalette 1 --AlphaCutoff 10 --AutodetectPalettedBAM 1 --AutodetectPalettedThreshold 500 --BAMProfile "" --Compress 1 --DebugLevelL 1 --DebugLevelP 2 --DebugLevelS 1 --DropDuplicateFrameData 1 --DropDuplicateFrameEntries 1 --DropDuplicatePaletteEntries 1 --DropEmptyCycleEntries 1 --DropUnusedFrameData 1 --DropUnusedFrameEntries 1 --DropUnusedPaletteEntries 1 --ExtraTrimBuffer 2 --ExtraTrimDepth 3 --FindBestRLEIndex 0 --FixPaletteColorErrors 1 --FLTSanityCutoff 5040 --ForceShadowColor 0 --ForceTransColor 1 --IntelligentRLE 1 --MaxRLERun 255 --OrderOfOperations C --ReduceFrameColumnLT 1 --ReduceFramePixelLT 1 --ReduceFrameRowLT 1 --SearchTransColor 1 --TrimFrameData 1 --zopfliIterations 1000000 --LogFile "C:\Users\Sam\Desktop\TH_Dark_Theme\Log.txt" --OutPath "C:\Users\Sam\Desktop\TH_Dark_Theme\compressed" --Save "BAM" "C:\Users\Sam\Desktop\TH_Dark_Theme\*.bam"

 
So what the hell is all this?  I'll go into some detail.  There are multiple levels of indirection built into the BAM V1 file format.  Many of the structures contain pointers to other "data", and because of this, duplicate or unused "data" can be removed without affecting the BAM in any visual manner (from the player's perspective in the game).
 

"D:\Program Files\GitHub Projects\PS-BAM\PS BAM_x64.exe"

The path to the PS BAM executable.  Use the 64-bit one if you can (it should be a bit faster).
 
 

--AdvancedFLTCompression 1

From the IESDP:

The frame lookup table is an array of frame indices. A cycle specifies a sequence of entries in this table. For instance, an animation might start at the 0th element in this array, and travel over 6 frames. (The next animation, then, would typically start at the 6th element and run over some number of indices.) If the first 6 entries in this table were { 0, 1, 1, 2, 3, 4 }, the animation would display frame #0, followed by frame #1 for two time periods, followed by frames 2, 3, 4 in order.

So let's say the 0th cycle uses frames [3, 4, 5, 6, 7, 8, 9] and the 1st cycle uses frames [0, 1, 2, 3, 4, 5].  These runs of frame indices are stored in the Frame Lookup Table (FLT for short).  Each cycle entry is merely a pointer into the FLT and a count of how many frame entries to use.  Typically, these runs of frame indices are merely stored end-to-end in the FLT like [3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5].  The 0th cycle entry would point to index 0 of the FLT and span 7 frame indices, and the 1st cycle entry would point to index 7 and span 6 frame entries.  Because of this level of indirection, however, this need not be the case.  Instead we can rearrange and overlap the runs of frame indices to make it shorter: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9].  Now the 0th cycle entry would point to index 3 of the FLT and span 7 frame indices, and the 1st cycle entry would point to index 0 and span 6 frame entries.  Now instead of our FLT being 13 frame indices, it is only 10.  The problem is that there may be more than one possible way to combine multiple arrays of frame indices.  Consider the following hypothetical situation:

You can see that one possible concatenation is better than the other.  --AdvancedFLTCompression attempts to find the smallest possible concatenation of all arrays of frame indices.  It does this by attempting to combine them in every possible order (every permutation).  The issue we run into is that the number of possible permutations is equal to the factorial of the number of cycles.  So if there are 5 cycles, 5!=120 possible permutations; 7 cycles is 5,040 permutations, 9!=362,888; for a creature BAM like AMOOG11 that has 54 cycles there are over 2x10^71 possible orders to attempt to combine the arrays of frame indices.  Not even I have that kind of time, so

--FLTSanityCutoff

can be used to stop attempting different permutations after the specified number in the event things extrapolate out of control.  PS BAM is actually a bit smarter than blindly attempting every permutation from the beginning.  It starts by dropping any array (of frame indices) which is wholly contained within another array.  It then removes any array that has no frame indices in common with any other arrays from the remaining pool.  Finally the arrays are sorted by the average value of the frame indices in attempt to achieve better overlap with fewer permutations.  Only after this does it start attempting to concatenate every permutation of the remaining pool.  A good initial value for --FLTSanityCutoff is 120, but when I have the time I tend to use 720 or 5040.  AdvancedFLTCompression is mathematically lossless.

 

--AdvancedZlibCompress

This applies z-lib compliant compression to compress a BAM V1 file to a BAMC V1 file.  A value of 1 uses the actual z-lib dll on the highest setting to perform the compression.  A value of 2 will perform the compression with zopfli which performs z-lib compliant compression, but is capable of better compression that the the original z-lib library itself.  Additionally,

--zopfliIterations

specifies the number of iterations zopfli will perform searching for better compression.  The more iterations, the better the compression but the longer it takes.  500 is ok for the casual user.  10,000 is more than enough.  Anything higher than that might save you a couple of bytes, but will take a long time to complete.  AdvancedzlibCompress is mathematically lossless.
 
 

--AllowShortPalette

After dropping unused and duplicate palette entries, this option will make the BAM's palette only big enough to hold all of the used colors (thus storing less than the 256 color, 1024 byte maximum).  If this option is disabled, the palette will be padded with zeros (equating to pure black).  The way PS BAM loads BAMs, AllowShortPalette itself should be mathematically lossless.
 
 

--AlphaCutoff

This used to transform nearly transparent pixels into the fully transparent background color, thus freeing up palette entries that can be dropped and potentially increasing how many transparent pixels can be trimmed from the frame.  In BAMs, alpha values range from 1 (nearly transparent) to 255 (fully opaque) with 0 also representing fully opaque for backwards compatibility.  With a value of, say 10, we are setting any color with an alpha value between 1 and 10 inclusive to the background (transparent) color.  AlphaCutoff is visually lossy.
 
 

--AutodetectPalettedBAM

"Paletted" BAMs have a special palette that usually look like this:

Each of the special color gradients in this palette are remapped to other color gradients assigned by character or item colors (this is what the PaletteGenerator mentioned above simulates).  If --AutodetectPalettedBAM is enabled, the sum of the Euclidean distance in RGBA colorspace is calculated between the BAM's palette and this special one.  If a paletted BAM is detected, duplicate and unused palette entries will not be dropped form the palette.

--AutodetectPalettedThreshold

Provides the opportunity to set a threshold of what is and is not considered a "paletted" BAM.  A value of 0 (zero) means the palettes must match exactly.  A value of 500 will identify a BAM as paletted even if it uses the teal and pink transparent and shadow colors from BAMWorkshop.  A handful of paletted BAMs in unmodded games use a similar but different palette:
 
A value of 14100 will identify vanilla off-paletted palettes as paletted (at least all the variants I tested), but it can be set to any number.  These settings have no direct impact on lossless vs lossy compression; instead see the descriptions for --DropDuplicatePaletteEntries and --DropUnusedPaletteEntries.
 
 

--BAMProfile

This can be used to set some predefined compression profiles, but as the development of PS BAM progresses, these
profiles are subject to change.  The current accepted profiles are Max, Recommended, Safe, Quick, and None.  This is the only setting that is position dependent, meaning you can set a compression profile and then overwrite specific settings by specifying them individually after it.  Additionally/alternately, you may specify a target game engine for some additional engine-specific toggling of settings (although this feature may not handle every engine's specific quirks under every circumstance).  Accepted engine variants are BG1, PST, IWD, BG2, IWD2, and EE.  As of 20181125, the current code looks like:

	Arr:=StrSplit(Settings.CompressionProfile,A_Space)
	Loop, % Arr.Length()
		{
		If (Arr[A_Index]="Recommended")
			Settings.Compress:=1, Settings.FixPaletteColorErrors:=1, Settings.AutodetectPalettedBAM:=1, Settings.DropDuplicatePaletteEntries:=1, Settings.DropUnusedPaletteEntries:=1, Settings.SearchTransColor:=1, Settings.ForceTransColor:=1, Settings.ForceShadowColor:=0, Settings.AlphaCutoff:=10, Settings.AllowShortPalette:=1, Settings.TrimFrameData:=1, Settings.ExtraTrimBuffer:=2, Settings.ExtraTrimDepth:=3, Settings.ReduceFrameRowLT:=1, Settings.ReduceFrameColumnLT:=1, Settings.ReduceFramePixelLT:=1, Settings.DropDuplicateFrameData:=1, Settings.DropUnusedFrameData:=1, Settings.IntelligentRLE:=1, Settings.MaxRLERun:=255, Settings.FindBestRLEIndex:=0, Settings.DropDuplicateFrameEntries:=1, Settings.DropUnusedFrameEntries:=1, Settings.AdvancedFLTCompression:=1, Settings.FLTSanityCutoff:=720, Settings.DropEmptyCycleEntries:=1, Settings.AdvancedZlibCompress:=2, Settings.zopfliIterations:=1000
		Else If (Arr[A_Index]="Max")
			Settings.Compress:=1, Settings.FixPaletteColorErrors:=1, Settings.AutodetectPalettedBAM:=0, Settings.DropDuplicatePaletteEntries:=1, Settings.DropUnusedPaletteEntries:=1, Settings.SearchTransColor:=1, Settings.ForceTransColor:=1, Settings.AlphaCutoff:=10, Settings.AllowShortPalette:=1, Settings.TrimFrameData:=1, Settings.ExtraTrimBuffer:=2, Settings.ExtraTrimDepth:=3, Settings.ReduceFrameRowLT:=1, Settings.ReduceFrameColumnLT:=1, Settings.ReduceFramePixelLT:=1, Settings.DropDuplicateFrameData:=1, Settings.DropUnusedFrameData:=1, Settings.IntelligentRLE:=1, Settings.MaxRLERun:=255, Settings.FindBestRLEIndex:=1, Settings.DropDuplicateFrameEntries:=1, Settings.DropUnusedFrameEntries:=1, Settings.AdvancedFLTCompression:=1, Settings.FLTSanityCutoff:=5040, Settings.DropEmptyCycleEntries:=1, Settings.AdvancedZlibCompress:=2, Settings.zopfliIterations:=1000
		Else If (Arr[A_Index]="Safe")
			Settings.Compress:=1, Settings.FixPaletteColorErrors:=1, Settings.AutodetectPalettedBAM:=1, Settings.DropDuplicatePaletteEntries:=0, Settings.DropUnusedPaletteEntries:=0, Settings.SearchTransColor:=1, Settings.ForceTransColor:=0, Settings.ForceShadowColor:=0, Settings.AlphaCutoff:=10, Settings.AllowShortPalette:=0, Settings.TrimFrameData:=1, Settings.ExtraTrimBuffer:=0, Settings.ExtraTrimDepth:=0, Settings.ReduceFrameRowLT:=1, Settings.ReduceFrameColumnLT:=1, Settings.ReduceFramePixelLT:=1, Settings.DropDuplicateFrameData:=1, Settings.DropUnusedFrameData:=1, Settings.IntelligentRLE:=1, Settings.MaxRLERun:=254, Settings.FindBestRLEIndex:=0, Settings.DropDuplicateFrameEntries:=1, Settings.DropUnusedFrameEntries:=1, Settings.AdvancedFLTCompression:=1, Settings.FLTSanityCutoff:=720, Settings.AdvancedZlibCompress:=0
		Else If (Arr[A_Index]="Quick")
			Settings.Compress:=1, Settings.FixPaletteColorErrors:=1, Settings.AutodetectPalettedBAM:=1, Settings.DropDuplicatePaletteEntries:=1, Settings.DropUnusedPaletteEntries:=1, Settings.SearchTransColor:=1, Settings.ForceTransColor:=1, Settings.ForceShadowColor:=0, Settings.AlphaCutoff:=10, Settings.AllowShortPalette:=1, Settings.TrimFrameData:=1, Settings.ExtraTrimBuffer:=2, Settings.ExtraTrimDepth:=3, Settings.ReduceFrameRowLT:=1, Settings.ReduceFrameColumnLT:=1, Settings.ReduceFramePixelLT:=1, Settings.DropDuplicateFrameData:=1, Settings.DropUnusedFrameData:=1, Settings.IntelligentRLE:=1, Settings.MaxRLERun:=254, Settings.FindBestRLEIndex:=0, Settings.DropDuplicateFrameEntries:=1, Settings.DropUnusedFrameEntries:=1, Settings.AdvancedFLTCompression:=1, Settings.FLTSanityCutoff:=1, Settings.DropEmptyCycleEntries:=1, Settings.AdvancedZlibCompress:=1
		Else If (Arr[A_Index]="None")
			Settings.Compress:=0
		; BG1 | PST | IWD | BG2 | IWD2 | EE
		Else If (Arr[A_Index]="BG1") OR (Arr[A_Index]="PST")
			{
			Settings.AdvancedZlibCompress:=0
			Settings.ForceTransColor:=1	; May also want to include Settings.SearchTransColor:=1 here.
			}
		Else If (Arr[A_Index]="EE")
			Settings.FindBestRLEIndex:=0
		}

For example, you might use '--BAMProfile Quick' to enable the quick compression profile which is tailored to fast but not-as good compression.  Or you might use '--BAMProfile "Max EE"' to attempt to achieve the best possible (but much slower) compression while maintaining compatibility with the EE engine.  The scope of these engine-specific toggles may need to be expanded (I haven't looked at or used them in a while).
 
If you have additional questions or suggestions for changes to these defaults, just ask.
 
 

--Compress

This setting globally toggles compression as a whole on or off.  In the near future I am strongly considering depreciating it and relying exclusively on the presence of "C" in --OrderOfOperations to enable/disable compression.  As of 20181125, you must still use '--Compress 1 --OrderOfOperations "C"' to enable compression.
 
 

--DebugLevelL
--DebugLevelP
--DebugLevelS

This tells the program how much debug output to display.  DebugLevelL is for the loading step, DebugLevelP is for the processing step (which includes compression), and DebugLevelS is for the saving step.  Values of:

• -1 = very silent (no log)
• 0 = Silent  (errors only)
• 1 = Normal  (errors and warnings)
• 2 = Verbose (errors, warnings, extra info)

--LogFile

Specifies where to save this debug log.  If a blank value is specified, e.g. '--LogFile ""', then no log will be saved.  Otherwise it should be the path and filename of where to save the log file.
 
 

--DropDuplicateFrameData

This allows the BAM compressor to drop duplicate frame data.  Because of the way PS BAM reads BAMs, this operation should be mathematically lossless.
 
 

--DropDuplicateFrameEntries

This allows the BAM compressor to drop duplicate frame entries.  Because of the way PS BAM reads BAMs, this operation should be mathematically lossless.
 
 

--DropDuplicatePaletteEntries

This allows the BAM compressor to drop duplicate palette entries.  This operation is visually lossless but should not be used with paletted BAMs.
 
 

--DropEmptyCycleEntries

This allows the BAM compressor to drop cycles which contain no frames.  Depending on the type of BAM you are working with (item, spell, GUI, creature animation, etc.), this may or may not be a desirable thing to do and may or may not alter how the game engine uses or displays a BAM.  Under some circumstances, this operation may be visually lossless, but it is not mathematically lossless.
 
 

--DropUnusedFrameData 1

This allows the BAM compressor to drop unused frame data.  This operation is visually lossless.
 
 

--DropUnusedFrameEntries

This allows the BAM compressor to drop unused frame data.  This operation is visually lossless.
 
 

--DropUnusedPaletteEntries

This allows the BAM compressor to drop unused palette entries.  As with the other Boolean options, a value of 0=FALSE/OFF and 1=TRUE/ON.  However, with --DropUnusedPaletteEntries you may instead pass it a value of 2 which will only drop unused palette entries from the end of the palette.  This operation is visually lossless, but should not be used on paletted BAMs (except with a value of 2 which should probably be okay).
 
 

--ExtraTrimBuffer
--ExtraTrimDepth

These can enable special trimming of non-transparent pixels if the specified conditions are met.  See this post for more details and an example.  This operation is visually lossy.
 
 

--IntelligentRLE

This option enables intelligent run-length-encoding of frame data.  It is "intelligent RLE" for a number of reasons.  First, RLE will never be applied to a frame unless doing so will decrease its total size.  Next, RLE'd data may consist of runs of the RLEColorIndex of up to 255 pixels.  However, BAMWorkshop (BWI) has a know bug where it can only handle runs of up to 254 pixels.  To add support for this popular legacy BAM editor, I have provided the option

--MaxRLERun

which should generally be given a value of 254 (to maintain support for BWI) or 255 (to allow for maximum possible compression).  However, even if you allow PS BAM to use runs of up to 255, it will only do so if it provides better compression than using 254 (again, to maximize compatibility with BWI which for many people may be a priority).
 
By default, the transparent color will be used as the RLE color.  This behavior may optionally be modified using:

--FindBestRLEIndex

According to the actual BAM file format specifications, the palette entry to use for RLE need not necessarily be the same palette entry as the transparent color.  Some engine variants like BG1 handle this properly while others like ToB and the EEs do not.  I have made a post about it here and a Beamdog bug ticket hoping to get this behavior corrected in a future EE patch.  Turning on this option will allow PS BAM to search every used palette entry for the one that provides the best compression as the RLE color.  RLE is always mathematically lossless.
 
 

--FixPaletteColorErrors

This will detect if the teal/cyan background color and pink shadow color from BAMWorkshop are present in the BAM's palette, and if so correct them to the proper green and black colors.  It will also correct the brown background color and dark grey shadow color occasionally produced by BAM Batcher.
 
 

--SearchTransColor

Early engine variants sometimes allowed a palette entry other than the first to be the transparent color.  From the IESDP:

The transparency index is set to the first occurence of RGB(0,255,0). If RGB(0,255,0) does not exist in the palette then transparency index is set to 0

However later engine variants (ToB and the EEs) make palette entry 0 the transparent color, regardless of what color is there.  Some discussion on the subject starts in this post.  --SearchTransColor tells PS BAM to search the palette for the transparent color (green).  If it doesn't find it, the transparent index will be set to palette entry 0.
 
 

--ForceTransColor

This setting tells PS BAM to force palette entry 0 to be the proper transparent color.  If --SearchTransColor was enabled and the transparent color was found at a palette entry other than 0, it will move it to palette entry 0.  Otherwise it will simply overwrite palette entry 0 with the transparent color (green).
 
 

--ForceShadowColor

This will force PS BAM to make the shadow color (palette entry 1) to be the proper black color.  Depending on what value is passed to --ForceShadowColor determines how this is to be achieved.   Available options are 0=None | 1=Force | 2=Move | 3=Insert (move will insert if fails).  0 will disable forcing the shadow color.  1 will overwrite palette entry 1 with the shadow color (black).  2 will search the palette for the shadow color (black) and move it to palette entry 1 if it finds it (or resort to method 3 if it does not).  3 will find an unused palette entry if there is one, move it to palette entry 0, and set it to the shadow color (black).  If all palette entries are used, it will resort to method 1.
 
 

--OrderOfOperations

This is used to enable the distinct categories of processes that PS BAM can perform, and specifies the order in which to perform them.  Parameters for this option are any ordering of P, C, and E where P=Process, C=Compress, and E=Export.  For example if you used "--OrderOfOperations C" you would compress the BAM without performing any additional processing or exporting operations.  Alternatively, if you used "--OrderOfOperations PCE" you would perform preliminary processing of the BAM, then compress it, then perform any requested export operations.
 
 

--ReduceFrameColumnLT
--ReduceFramePixelLT
--ReduceFrameRowLT

These options will reduce a frame with no more than the specified number of columns, pixels, or rows to a single transparent pixel at offset (0,0).  For example "--ReduceFramePixelLT 1" would make any frame consisting of a single pixel (no matter the color) to a single transparent pixel located at horizontal and vertical offsets of 0.  Use Case:  Creature animations often contain many frames consisting of a single orange pixel which are not even displayed by the engine and manipulating them in this way will often improve overall compression.  These options are at worst visually lossy.
 
 

--TrimFrameData

Trims rows and columns of transparent pixels from around a frame, manipulating it's offsets to compensate accordingly.
 
 

--OutPath

Give it the directory specifying where to put all the output files including exported frames, palettes, sequences, and any saved files like BAMs, BAMDs, GIFs, etc.
 
 

--Save

Used to specify in what format to save the processed/generated BAMs in.  Accepted format are BAM, GIF, and BAMD where BAMD takes the filetype for each frame from Settings.ExportFrames.  Blank is do not save (but exports will still be processed).
 
 
Very last should come the files to process.  Wildcards are accepted, and any number of input files can be specified.
 
 
 

Result
These BAMs were quite simple (1 frame in 1 sequence) so they aren't a good example of just how much better this hyper compression can be, but here are the results:

        Name         OriginalSize UncompressedSize CompressedSize %OfOriginalSize %OfUncompressedSize       Time      
    BDDHAM03.BAM         2374           9690            1887        79.486099 %       19.473684 %     1140.532352 sec.
    BDDHAM04.BAM         2589           9440            1930        74.546157 %       20.444915 %     1037.944160 sec.
    CHAMM07.BAM          4019          16906            3577        89.002239 %       21.158169 %     1348.988349 sec.

The hyper compressed BAMs were about 81% of the size NearInfinity produced and 20% compared to being fully uncompressed.  The times are artificially high because I ran a stupidly high number (1,000,000) of zopfli iterations (z-lib compliant compression to a BAMC file), which probably only saved a handful of bytes compared to say 500 iterations.  That's just my style, tho.

Edited by Sam., 25 November 2018 - 05:21 PM.

[Creepin]

Sam., can you please explain why would one want to compress bams at all especially considering hindrances you mentioned in the readme? Bams that are bmps made with photoshop and converted with Miloch's BAM Batcher, are they not compressed? Is it a bad thing? Why?

[Sam.]

Sam., can you please explain why would one want to compress bams at all especially considering hindrances you mentioned in the readme? Bams that are bmps made with photoshop and converted with Miloch's BAM Batcher, are they not compressed? Is it a bad thing? Why?

In a broad sense, compressing BAMs that are packaged with a mod saves on bandwidth on the upload and each download, and saves on HD space in each user's override folder.  In the era of EEs where the recommended practice is to not biff your override folder, PS BAM is capable of compressing BAMs even smaller than can be achieved by biffing them.  From a more functional standpoint, compressing/simplifying a BAM (such as dropping duplicate frames) not only decreases the memory required by the engine to load the BAM, but should also reduce the computational resources required to display it (the exception being RLE which most game BAMs use anyway).  A filesize reduction of 20% may seem trivial for the odd BAM or two, but 20% of something like Infinity Animations that totals several to many gigs in all becomes more significant.

 

As to the "hindrances" I mentioned, I assume you refer to potential incompatibilities with game engines and legacy tools?  Each iteration of the Infinity Engine from the original Baldur's Gate to the latest EEs tend to be somewhat inconsistent in how they render BAMs.  For instance, BG1/ToSC and PST do not support BAMC files (BAM files that are compressed with z-lib complaint compression) while other engine variants do.  ToSC and PST did not require the transparent color to be the 1st palette entry, whereas many of the other engine variants do.  ScottBrooks (a Beamdog programmer) is quoted as saying:

 

2: When the bam is rendering, it asks the palette system how many entries are reserved. If the 2nd entry matches the shadow color of 0x00000000, then it returns 2 Otherwise it returns 1. So basically if the 2nd entry is the shadow color, then it will treat it as such. Otherwise it just uses the 1 reserved entry of the color key green.

However, the EEs include many components for Erephine's 1pp mod.  Erephine used BAMWorkshop which used teal for the transparent color and pink for the shadow color, and these 1pp BAMs display fine in the EE engine so this is clearly no longer true.

 

It has long been known that BAMWorkshop can not properly handle BAMs with runs of 255 RLE'd transparent pixels, even tho the game engines can.  I personally have authored edits to BAM Batcher and bamresizer to limit RLE runs to 254 pixels to ensure compatibility with this legacy editor.  BAMWorkshop II is notorious for being unstable and corrupting BAMs.  Neither BAMWorkshop nor BAMWorkshop II can handle BAMs that use the alpha channel, although the EE engines can.  After a recent NearInfinity update fixing a regression with short palettes, I have yet to encounter a BAM which PS BAM compressed that NI can't load properly.

 

PS BAM does not provide compression options that are not compatible with at least one engine variant.  Moreover, it strives to support the best possible compression for each engine variant, and with a couple caveats in the event of BAM corruption, never forces any "compression" compared to the original file.  My point of the warnings is not that PS BAM can or will magically produce incompatible BAMs, it is that the compression it can produce may not necessarily be compatible with every engine variant and that every legacy BAM editor may not be capable of properly viewing/editing what it produces.  The only guaranteed method of testing whether a compressed BAM will work properly is to test it in your target game(s).

 

Because of the shortcomings of legacy BAM editors, you will generally want to run PS BAM's hyper compression after all editing has been completed.  You can always use PS BAM to uncompress it again if further edits are needed.

 

---

 

BAM Batcher is only capable of using BMPs as inputs.  I faked an image quantizer for BAM Batcher a while back, but it will produce very bad posterization if much more than 256 colors are used.  PS BAM is capable of doing everything BAM Batcher is, but it supports a real quantizer and many more file format (such as GIF, PNG, JPG, etc).  BAM Batcher supports unintelligent RLE and z-lib compression to BAMC files, but the compression offered by PS BAM is much more extensive.

Edited by Sam., 20 November 2018 - 05:58 AM.

[The Imp]

In the era of EEs where the recommended practice is to not biff your override folder, PS BAM is capable of compressing BAMs even smaller than can be achieved by biffing them.  

Biffing had nothing to do with the size... it's about the speed at which the data can be read. At time when RAM was not a thing ... yet.

[Creepin]

Thank you for this exhaustive explanation Sam!

[Gwendolyne]

Sam, it's been a long time I wanted to test PS BAM to compress the thousands creature animation files I need to install, but always postponed it because of the lack of documentation.

 

What would be the best command line options, let's say to decompress-recompress all bam files in MYMOD/bam/cre/list of directories, that would produce a significant files size reduction?

 

Edit : games compatibility (ToB and BG2EE might be enough to be fully compatible with BGT and EET).

Edited by Gwendolyne, 20 November 2018 - 12:52 AM.

[Sam.]

In the era of EEs where the recommended practice is to not biff your override folder, PS BAM is capable of compressing BAMs even smaller than can be achieved by biffing them.  

Biffing had nothing to do with the size... it's about the speed at which the data can be read. At time when RAM was not a thing ... yet.

This statement is simply not true, for multiple reasons.  I was tempted to just ignore it, but an explanation has some relevance to the topic at hand.

 

Biffing had nothing to do with the size...

When files are added to a BIFC V1 or BIFC V1.0 archive file (aka biffed), they are z-lib compressed.  This reduces the filesize (with a caveat I'll explain in a minute).  Furthermore, it is more efficient (in terms of size) to store one large file instead of many small ones.  Quoth IESDP concerning the BIFF file format:

 

This file format is a simple archive format, used mainly both to simplify organization of the files by grouping logically related files together (especially for areas). There is also a gain from having few large files rather than many small files, due to the wastage in the FAT and NTFS file systems.

Go look up the difference between file size and size on disk.

 

Now the caveat:  When a BAM V1 (as opposed to a BAMC V1) is added to a BIF it becomes z-lib compressed which reduces its size.  A BAMC file, however, is already z-lib compressed and adding it to a BIF will force it to be z-lib compressed again which does not gain you anything.  PS BAM uses zopfli to perform zlib-compatible compression.  Zopfli offers better compression ratios than the z-lib library itself (at the cost of more processing time).  If you try to biff a file compressed with zopfli, the resulting filesize will increase not decrease, so don't do it.

 

 

At time when RAM was not a thing ... yet.

RAM has been around since like the 1970s and is almost exclusively volatile memory.  Perhaps you are referring to SSDs which usually use non-volatile NAND flash memory?

 

it's about the speed at which the data can be read.

Storing a single file instead of many small files does have the added benefit of faster data access due to consecutive vs nonconsecutive RW operations on electromechanical and optical drives.  However, as pointed out earlier, this is not the only benefit.

[Sam.]

Sam, it's been a long time I wanted to test PS BAM to compress the thousands creature animation files I need to install, but always postponed it because of the lack of documentation.

 

What would be the best command line options, let's say to decompress-recompress all bam files in MYMOD/bam/cre/list of directories, that would produce a significant files size reduction?

 

Edit : games compatibility (ToB and BG2EE might be enough to be fully compatible with BGT and EET).

I'm slowly trying to work on documentation, but there is a lot to document.  Some options are fairly strait forward (I bet you can guess what --DropUnusedFrameData does), but others require a much more technical and detailed explanation in order to get a good understanding of what it does or how it should be used (--AdvancedFLTCompression and the corresponding --FLTSanityCutoff for instance).  I'll try to do some more this week/weekend.

 

As for what are the best options for compression, that really depends on what your goals are and how much time you want to spend on compression.  When I get around to compressing Infinity Animations, my plan is to crank compression way up and use three quad-core computers with a different instance of PS BAM running on each core and let it run for a few days.  That is undoubtedly excessive compared to what most people will want to do.  What I used in tutorial "2)  Very Good (Hyper) Compression of BAM V1 Files" is a good place to start, but consider turning --zopfliIterations down a bit as I suggested.  There are enough options it's worth taking the time to make some informed decisions about which options you want to use (again, working on documentation...).

 

I'll try to give you an idea of the possible scope of your options.  I personally think of the various compression options as divided into three categories:  mathematically lossless, visually lossless, and visually lossy.  Examples of mathematically lossless include RLE, z-lib compression, and because of how PS BAM reads BAMs, usually even dropping duplicate frames and frame entries.  I term them mathematically lossless because these things can be undone to get exactly the same file you started with.  Visually lossless include things like trimming transparent pixels from frames and dropping unused palette entries, frame entries, frames, etc.  Visually lossless means pixel per pixel, the displayed BAMs will be identical before and after the operation, but the operations can never be undone.  Lastly, visually lossy includes operations that actually alter the appearance of a BAM.  For example,

--AlphaCutoff 10

is used to transform nearly transparent pixels into the fully transparent background color, thus freeing up palette entries that can be dropped and potentially increasing how many transparent pixels can be trimmed from the frame.  In BAMs, alpha values range from 1 (nearly transparent) to 255 (fully opaque) with 0 also representing fully opaque for backwards compatibility.  In this particular case, we are setting any color with an alpha value between 1 and 10 inclusive to the background (transparent) color.  It is technically visually lossy, but I challenge you to see the difference.

 

Here's a second example of a visually lossy option.  I exported the frames from MDR11207.bam using:

"D:\AutoHotkey Scripts\PS BAM\PS BAM.ahk" --Compress 0 --OrderOfOperations "E" --ExportFrames "PNG" --DebugLevelL 1 --DebugLevelP 2 --DebugLevelS 1 --LogFile "C:\Users\Sam\Desktop\Export\Log.txt" --OutPath "C:\Users\Sam\Desktop\Export\Extract" --Save "" "C:\Users\Sam\Desktop\Export\MDR11207.bam"

You will notice some "garbage" pixels towards the upper right corner of this frame.  These pixels are not part of the animation, and would block the frame from being trimmed from the top using the standard

--TrimFrameData 1

However, PS BAM provides some more advanced options capable of trimming these garbage pixels:

--ExtraTrimBuffer 2 --ExtraTrimDepth 3

ExtraTrimDepth specifies the maximum number of rows that can contain non-transparent color pixels to be considered for trimming.  ExtraTrimBuffer specifies the minimum number of fully transparent rows that must separate the non-transparent rows from any others before they will be trimmed.  To put it another way, going from the top edge down, we can have up to three rows of pixels that are not fully transparent.  They are followed by at least two rows that are fully transparent, therefore they will be removed.

"D:\AutoHotkey Scripts\PS BAM\PS BAM.ahk" --OrderOfOperations "CE" --TrimFrameData 1 --ExtraTrimBuffer 2 --ExtraTrimDepth 3 --ExportFrames "PNG" --DebugLevelL 1 --DebugLevelP 2 --DebugLevelS 1 --LogFile "C:\Users\Sam\Desktop\Export\Log.txt" --OutPath "C:\Users\Sam\Desktop\Export\Extract" --Save "" "C:\Users\Sam\Desktop\Export\MDR11207.bam"

Yields

Beware, however, because this can sometimes trim legitimate pixels also.  Consider a character's sword animation where the pommel is only a few pixels followed by several rows of transparent pixels where the character's hand would go.

Edited by Sam., 20 November 2018 - 06:32 PM.

[Sam.]

I have updated 2)  Very Good (Hyper) Compression of BAM V1 Files to include at least a cursory explanation for all the compression options.  That probably covers half of what there is to document considering I've barely scratched the surface of import, processing, and export operations .  As always, if you have questions or need more clarification, please ask.

 

@Gwendolyne,

Have you had a chance to play around with compressing your animations yet?

[Gwendolyne]

No, I did not test it yet as I was identifying the available EE animations slots per format. It is done (I will PM them as soon as possible) and I will test PSBam as soon as I have installed my new slots, in a couple of days.

 

Thanks for the documentation. It is a bit more clearer now. and will avoid a few mistakes, like moving all pure black pixels into shadow index. Indeed not a good idea to compress my black paladin mount...

 

Edit: I am currently making animations gif files for my readme. Is there a way to export a bam file into fig with a transparent background and a semi-transparent shadow color? I guess not, but who knows, maybe you are a magician...

Edited by Gwendolyne, 26 November 2018 - 03:07 PM.

[Sam.]

Edit: I am currently making animations gif files for my readme. Is there a way to export a bam file into fig with a transparent background and a semi-transparent shadow color? I guess not, but who knows, maybe you are a magician...

Well, the GIF file format doesn't support semi-transparency, so that's out.  You can specify a transparent color for the background, tho (which PS BAM is supposed to do).

 

You can export each sequence as a separate animated GIF using:

"D:\Program Files\GitHub Projects\PS-BAM\PS BAM_x64.exe" --Compress 0 --OrderOfOperations C --SingleGIF 0 --DebugLevelL 1 --DebugLevelP 2 --DebugLevelS 1 --LogFile "D:\AutoHotkey Scripts\PS BAM\MooseN\Log.txt" --OutPath "D:\AutoHotkey Scripts\PS BAM\MooseN\GIF Example" --Save "GIF" "D:\AutoHotkey Scripts\PS BAM\test\AMOOG11.BAM"

or you can export EVERY frame in the BAM into a single GIF via:

"D:\Program Files\GitHub Projects\PS-BAM\PS BAM_x64.exe" --Compress 0 --OrderOfOperations C --SingleGIF 1 --DebugLevelL 1 --DebugLevelP 2 --DebugLevelS 1 --LogFile "D:\AutoHotkey Scripts\PS BAM\MooseN\Log.txt" --OutPath "D:\AutoHotkey Scripts\PS BAM\MooseN\GIF Example" --Save "GIF" "D:\AutoHotkey Scripts\PS BAM\test\AMOOG11.BAM"

 

Edit:  Well, I'm not sure why the forum software isn't playing the animations...

Edit2:  Uploaded externally instead.

Edited by Sam., 26 November 2018 - 06:47 PM.

[Sam.]

I have committed some updates to PS BAM on GitHub.  Updates include:

• Improved exception handling
• Some code modernization
• A couple significant speed improvements
• A minor bugfix
• The ability to read BAM V2 files (only with the x64 EXE)

Note that due largely to the fact that PS BAM is written in AHK (which is an interpreted language), reading BAM V2 files from their PVRZs isn't particularly fast.  If the BAMs only have a handful of frames (as AFAIK all existing BAM V2 files do), it won't be an issue.  If you try reading a BAM V2 that has multiple thousands of frames, it works but will take a long time (from hours to overnight).  Writing to BAM V2 files is not supported at this time.

 

Let me know if you have any questions or issues.  Feature suggestions are welcome.

[Sam.]

I've push a few updates lately.  I've broken backwards compatibility in that I've renamed the previous '--Montage 1x2' to '--Montage Paperdoll' for clarity.  '--Montage mxn' and variants now work for a much wider variety of inputs.  See commit history on GitHub for more info.

[Sam.]

Pushed an update to v0.0.0.23a
 
Better support for exporting transparency

• Export PNG as 32-bit if PaletteHasAlpha=1
• Add ability to export palette as PNGV
• Add --ExportWithTransparency option to fine tune transparency handling
• _SetOpaquePalette0() and _AlphaCutoff() update Stats.PaletteHasAlpha

Improved shadow searching

• Added additional parameter to --SearchShadowColor option to search for whichever color is used for the shadow
• Fixed bug preventing specified parameter being passed to --SearchShadowColor

Fixed var not set in PVRz routine

Edited by Sam., 06 April 2022 - 05:35 AM.

[Sam.]

I've added two new options, --RemapGradientSet and --RemapGradient.  With something like this:

Spoiler

SET prefix=1PP-chft2inv

"D:\Program Files\GitHub Projects\PS-BAM\PS BAM_x64.exe" ^
--DebugLevelL 1 ^
--DebugLevelP 2 ^
--DebugLevelS 1 ^
--Save "" ^
--ExportPalette "bmpv" ^
--ExportFrames "bmp" ^
--ExportFramesAsSequences 0 ^
--OrderOfOperations "CPE" ^
--FixPaletteColorErrors 1 ^
--SearchTransColor 1 ^
--ForceTransColor 1 ^
--ForceShadowColor 0 ^
--AlphaCutoff 0 ^
--TrimFrameData 0 ^
--ExtraTrimBuffer 0 ^
--ExtraTrimDepth 0 ^
--ReduceFrameRowLT 1 ^
--ReduceFrameColumnLT 1 ^
--ReduceFramePixelLT 1 ^
--Montage "Paperdoll" ^
--RemapGradientSet "Pink2Yellow" ^
--RemapGradient "Pink_Yellow2Yellow" ^
--LogFile "%~dp0%Log.txt" ^
--OutPath "%~dp0framesalt3" ^
"%~dp0%prefix%*.bam"

Pause
Exit

You can remap the primary and/or mixed gradients in paletted BAMs to make changes like this:
->

[Salk]

Nice option to have, Sam! 

[Sam.]

Nice option to have, Sam! 

I've been toying with the idea of being able to export each gradient set and/or each primary/mixed gradient (of each frame) as a separate image.  I can see how it would be useful for editing a frame or two for a paperdoll, but it probably becomes a bit unwieldy for an entire animation.  Also, reliably importing them back in becomes troublesome considering the myriad ways one might edit them vs how persnickety palette indexing needs to be to behave properly in-game for paletted animations.  Any thoughts?