60 replies to this topic

[Suslik]

I have tried to explain the bug in details in my thread: http://www.shsforums...tection-glitch/
And then i was told to move here, to TobEx forum, because it seems that you guys are the only ones who can fix it.

Should i repost the contents from that thread here?

And i will have a lot of free time till february as well as a lot of enthusiasm to help you any way i can. I am a C++ programmer(about 7 years of experience), familiar with asm and disassemblers(much less experience). It would be great if my skills could be of any use to TobEx team.

Edited by Ascension64, 21 January 2012 - 05:30 AM.

[Ascension64]

I have yet to properly look at the problem in code... been too busy trying to fix up BGT at the moment. Also, check your PM.

[Suslik]

I was investigating TobEx code. It's all fine and readable in spite of some weird things(such as mixed __cdecl and __stdcall functions and sometimes strange code convention lol), but its cool. Ok, now the only thing left is reversing the executable itself, hehe. Hello, ollydbg.

[Suslik]

I was tinkering with ollydbg/IDA 5.0 trying to find those functions responsible for reflection/absorption. Proved to be quite challenging, hehe.

[Ascension64]

The reflect/absorb code sits in

5006AF BOOL CEffect::TryApplyEffect(creTarget, rollSaveDeath, rollSaveWands, rollSavePoly, rollSaveBreath, rollSaveSpells, rollResistMagic, rollEffectProb)

Bounced/protected spells get added to a CPtrList via

464A50 void CProtectedSplsList::AddTail(CEffect*, nPower, nOpcodeEffect, CCreatureObject*, int, STRREF, int, int)

That might help you get started.

[i30817]

I'm in linux, otherwise i would try to implement my new "Trigger()" replacement idea.

Seems like it should be simple if adding triggers and actions is simple (and not limited).

[Suslik]

okay, i was trying to do the following thing:
1) put the breakpoint to the procedure the effect is added to to list upon hitting on spell turning. thanks to you i know the adress.
2) set a memory breakpoint to void *this pointer with some range in order to find the next point where the list is accessed
3) this will give me the adress of procedure responsible for actually reflecting spells back. as far as i understand the problem is that this procedure is sometimes not called. check calling condition.

fxd:
okay, stages 1) and 2) are complete. i have found the point where spell list is modified, but i am still unable to determine the exact condition when it happens - the condition when stored in spell turning effects are fired back.

next strange thing: why are at least two effects added when a spell hits the target? "breach" adds two effects, but "chromatic orb" for example adds about 7-10, what for?

Edited by Suslik, 30 December 2011 - 12:32 AM.

[Suslik]

I have finally found the exact block responsible for firing spells back. The cycle responsible for firing all spells from the list hides near address 464C48. Now the only thing left is to determine why this block sometimes stops firing.

[really raw info]ok kids, now the tough shet. somehow i was unable to reproduce the bug while debugging. i wonder wtf? i can reproduce it while playing normally and cannot do it while under IDA debugger(under debugger everything works fine). what can cause such behaviour?[really raw info]

Edited by Suslik, 30 December 2011 - 05:32 AM.

[Ascension64]

Keep the updates coming...

[Suslik]

Ok, now i'm really stuck.

Test case is following:
I have a savegame where two partymembers(A and B) have cast spell turning on themselves.

1) I normally run BGMain.exe/TobExLoader.exe(BWP 9.10 standard) load the savegame, and char A casts "breach" on B. After that "breach" bounces from B to A and back randomly 0-3 times. If it bounces 3 times, it's ok, and spell turning wears off according to rules. But sometimes(actually 80% of times i load the game) when the spell hits a char, it is absorbed and never reflected back, "spell turning" starts absorbing spells infinitely. Behavior i have already described in the first post of the thread.
2) I run OllyDbg/IDA(same effect for both) and BGMain.exe from under the debugger. No breakpoints, no modifications to the executable, absolutely same actions, but the spell(breach in my case) is reflected back and forth 3 times, strictly as expected, no matter how many times i try. Furthermore, spells are reflected with a very strict timer of about 1 second. Let me remind, that even in vanilla game reflected spells sometimes lag for a few seconds-dozens of seconds before reflecting back(but they always get reflected, instead of getting absorbed infinitely as in BWP).

There's also a strange effect: if i set a breakpoint on spell reflecting code(464C48 for example) and resume the program after that, there's a specific animation(i remember the same in vanilla game long before) called spsturni.bam(jeez, i have managed to find it with bruteforcely browsing all images in resources lol) when a spell is reflected back: some kind of shiny silver halo or a disk and a reflected projectile is spawned from it. If i do not set the breakpoint at this exact place, i cannot see this halo - reflected spell's projectile is spawned from nowhere.

Questions:
- How come that under debugger spell reflection code works perfectly and gets bugged when executed normally.
- Minor issue, but is it possible to fix that halo which is supposed to appear when reflecting spells?

Edited by Suslik, 30 December 2011 - 05:32 AM.

[Suslik]

Anyway, if i attach debugger to running BGMain.exe, problem still can be reproduced. Problematic code is presumably here:

int __thiscall CheckReflectingSpells_464BFB(void *this, int a2)
{
  int v26; // [sp+18h] [bp-E8h]@1
  int v77; // [sp+ECh] [bp-14h]@1
  v26 = this;
  v77 = 0;
  while ( *(v26 + 12) != 0 )
  {
    /*464C43: */ v78 = CheckList_A4E60A(v26); //this line stops triggering when spell reflection glitches
    //...
  }
  //...
}

the problem is that function(464BFB) runs way too often, because it is fired for every creature. So i find it hard to debug the one fired for an exact character who should actually reflect the spell.

Question: do you(asc64, probably i'm asking you : D) do you know what that function(464BFB) is? Probably it is some kind of CCreature::UpdateAI or something. If so, what is in this CCreature with 12 offset? How often is that UpdateAI fired and under which circumstances?

[Suslik]

If the process runs without a bug under debugger and with it if executed normally, I can try and run it as if it was started with debugger attached. The only difference I know that causes such behavior is that debugger overrides global memory allocation procedures(malloc? VirtualAlloc?) with its own. If we assume that those debugger allocators set blocks of allocated memory to zeros, it can change behavior of running program if the memory is broken somewhere(uninitialized variable in heap?). So what if we try and replace current allocator(A50608) with our own, that will force all allocated memory to zero values?

We can also set the allocated memory to values like 0xcdcdcdcd so that such errors may be easily tracked(memory access breakpoints on "read" from block 0xcdcdcdcd). What do you say? Is it possible to replace global memory allocator? Is it worth trying?

Edited by Suslik, 30 December 2011 - 02:39 PM.

[Suslik]

There were some compilation errors(MSVS 2010 professional, tobex 0.20):

void* IECPtrList::GetHead() const { return (*this).ToCPtrList().GetHead(); }

cptrlistex.cpp(54): error C2440: 'return' : cannot convert from 'const void *' to 'void *' Conversion loses qualifiers

And a few similar methods. I have fixed them simply commenting. they still persist in v0.22, check it out.

When compiled in debug, it says:

Debug(): CBaldurChitin has incorrect size 0x724C (expected 0x720C)
and same for CTlkTbl

then program fails without any messages. i have managed to attach a debugger to tobex.dll, and it seems that crash occurs in CRuleTable.

maybe it's because i am using headers and libs from detoured 3.0(i have somehow failed to compile 2.1)? if i do not add any patches/hooks at all(comment all of them), BG still crashes when i try to load a game. that's a shame, but because of these errors i cannot test my nullifying allocator.

is it safe to take tobex.dll and tobexloader.exe from version 22 and test them or should i stick to v20 i'm currently using(with bwp 9.10)? i have tried updating current tobex 0.20 to version 0.22 via setup-tobex, but it attempted to install/reinstall a few millions of other mods and failed somewhere in the middle lol. of course i have a backup, so that's no problem.

Edited by Suslik, 30 December 2011 - 08:03 PM.

[Ascension64]

- How come that under debugger spell reflection code works perfectly and gets bugged when executed normally.

Are you using IDA? I'm not familiar with the OnFocus() code, but there may be some graphical changes or other wacky stuff going on when you lose/regain focus of the main window. I've had the game crash on me sometimes when switching back to the game window due to some sound code issues.

- Minor issue, but is it possible to fix that halo which is supposed to appear when reflecting spells?

Don't know what this is. I haven't had any time to look at the issue overall yet, so reading your progress is interesting.

the problem is that function(464BFB) runs way too often, because it is fired for every creature. So i find it hard to debug the one fired for an exact character who should actually reflect the spell.

Question: do you(asc64, probably i'm asking you : D) do you know what that function(464BFB) is? Probably it is some kind of CCreature::UpdateAI or something. If so, what is in this CCreature with 12 offset? How often is that UpdateAI fired and under which circumstances?

464BFB void CProtectedSplList::Update(CCreatureObject&)

It is called during 88BFE3 CCreatureObject::AIUpdate(), pretty much exclusively. Can't remember what it does, been a while since I looked at that proc. AI update is performed every tick if I remember correctly (15 ticks [also called AI updates] to one second)

If the process runs without a bug under debugger and with it if executed normally, I can try and run it as if it was started with debugger attached. The only difference I know that causes such behavior is that debugger overrides global memory allocation procedures(malloc? VirtualAlloc?) with its own. If we assume that those debugger allocators set blocks of allocated memory to zeros, it can change behavior of running program if the memory is broken somewhere(uninitialized variable in heap?). So what if we try and replace current allocator(A50608) with our own, that will force all allocated memory to zero values?

We can also set the allocated memory to values like 0xcdcdcdcd so that such errors may be easily tracked(memory access breakpoints on "read" from block 0xcdcdcdcd). What do you say? Is it possible to replace global memory allocator? Is it worth trying?

TobEx tries to use the embedded memory allocator whenever new resources are added to the game, especially instances that TobEx doesn't remove later. Saved me problems of dealing with more than one heap space. Local instances are still handled in TobEx.dll heap space. Confusing, I know. Maybe there is a better way to do this.

There were some compilation errors(MSVS 2010 professional, tobex 0.20):

void* IECPtrList::GetHead() const { return (*this).ToCPtrList().GetHead(); }


cptrlistex.cpp(54): error C2440: 'return' : cannot convert from 'const void *' to 'void *' Conversion loses qualifiers

And a few similar methods. I have fixed them simply commenting. they still persist in v0.22, check it out.

When compiled in debug, it says:

Debug(): CBaldurChitin has incorrect size 0x724C (expected 0x720C)
and same for CTlkTbl



then program fails without any messages. i have managed to attach a debugger to tobex.dll, and it seems that crash occurs in CRuleTable.

maybe it's because i am using headers and libs from detoured 3.0(i have somehow failed to compile 2.1)? if i do not add any patches/hooks at all(comment all of them), BG still crashes when i try to load a game. that's a shame, but because of these errors i cannot test my nullifying allocator.

is it safe to take tobex.dll and tobexloader.exe from version 22 and test them or should i stick to v20 i'm currently using(with bwp 9.10)? i have tried updating current tobex 0.20 to version 0.22 via setup-tobex, but it attempted to install/reinstall a few millions of other mods and failed somewhere in the middle lol. of course i have a backup, so that's no problem.

Are you compiling under MFC4.2 libraries? Newer version of the library have lots of changes to structs and stuffs the whole thing up. I spent a huge amount of time wrestling with this, since it also appears that the libraries I am using have slightly different code from that statically linked to BGMain. I haven't worked out exactly which MFC4.2 build it is using.

TobEx 0022 uses Detours Express 3.0, so you can simply slot that in. Probably better to use the most current revision anyway. Can you understand WeiDU code? If you can, you can manually update. Copy over the new TobEx.dll and make any changes as specified in the TobEx.tpa for an update.

If it is CTlkTbl size that is incorrect causing CBaldurChitin size to be wrong, it could be some of the vector objects being the wrong size in your MFC version you are trying to statically link. CMapPtrToString and such.

Edited by Ascension64, 30 December 2011 - 08:51 PM.

[Suslik]

Are you using IDA? I'm not familiar with the onfocus() code, but there may be some graphical changes or other wacky stuff going on when you lose/regain focus of the main window.

olly or IDA - does not matter. effect is the same. i am currently working on reproducing some aspects of a process being run under a debugger(minor modifications to CreateProcess in the launcher). I will tell you the results.

i doubt that onfocus() stuff has to do anything with that behavior - i can run the process from under debugger, then detach debugger, and spells will be reflected back properly. so i hope there's some way to figure out under which conditions is the process executed under debugger so that it fixes the bug.

464BFB void CProtectedSplList::Update(CCreatureObject&)

It is called during 88BFE3 CCreatureObject::AIUpdate(), pretty much exclusively. Can't remember what it does, been a while since I looked at that proc. AI update is performed every tick if I remember correctly (15 ticks [also called AI updates] to one second)

whoa, i feel so pro reversed it almost correctly : D

TobEx tries to use the embedded memory allocator whenever new resources are added to the game, especially instances that TobEx doesn't remove later. Saved me problems of dealing with more than one heap space. Local instances are still handled in TobEx.dll heap space. Confusing, I know. Maybe there is a better way to do this.

of course tobex should use allocator of BGMain.exe's heap. the thing i want is to modify it to automatically nullify all allocated memory(if it is the reason of working properly with debugger attached). i have already hooked BGMain's malloc and have written my modified wrapper with memset(ptr, 0, size) for it, but i cannot test it due to those runtime issues yet.

Are you compiling under MFC4.2 libraries? Newer version of the library have lots of changes to structs and stuffs the whole thing up. I spent a huge amount of time wrestling with this, since it also appears that the libraries I am using have slightly different code from that statically linked to BGMain. I haven't worked out exactly which MFC4.2 build it is using.

no idea. i mean i just compile it with native to MSVS 2010 libs and that's it. should i replace MFC manually to that of 4.2?

Edited by Suslik, 30 December 2011 - 09:00 PM.

[Suslik]

TobEx 0022 uses Detours Express 3.0, so you can simply slot that in. Probably better to use the most current revision anyway. Can you understand WeiDU code? If you can, you can manually update. Copy over the new TobEx.dll and make any changes as specified in the TobEx.tpa for an update.

well, yeah, i've thought that i understand it. more or less. recovering from backup again : D

Don't know what this is. I haven't had any time to look at the issue overall yet, so reading your progress is interesting.

never mind, it seems that it is a completely another issue, not related to reflection bugs. i have asked it here: http://forums.gibber...showtopic=17611

cptrlistex.cpp(54): error C2440: 'return' : cannot convert from 'const void *' to 'void *' Conversion loses qualifiers

and this is a compiletime error, has nothing to do with MFC or detours version. take a look when you have the time.

and thanks a lot for your info and help

Edited by Suslik, 30 December 2011 - 09:12 PM.

[Ascension64]

[quote name='Suslik' date='31 December 2011 - 02:56 PM' timestamp='1325307409' post='530371']
[quote]olly or IDA - does not matter. effect is the same. i am currently working on reproducing some aspects of a process being run under a debugger(minor modifications to CreateProcess in the launcher). I will tell you the results.[/quote]Oh OK. You do know that TobEx modifies the executable to LoadLibraryA() as well, so if you just launched bgmain.exe under debugger it should be OK, or maybe I miss the point. Anyway, I'm not a C++ guru, so have a play around. Don't forget the latest master is on github if you want to pull the most updated code. I committed just then with some minor updates.

[quote]whoa, i feel so pro reversed it almost correctly : D[/quote]Lol, reading the disassembly isn't really hard because there are no code optimisations, and there is the awesome IESDP project that made everything possible in the first place.

[quote]of course tobex should use allocator of BGMain.exe's heap. the thing i want is to modify it to automatically nullify all allocated memory(if it is the reason of working properly with debugger attached). i have already hooked BGMain's malloc and have written my modified wrapper with memset(ptr, 0, size) for it, but i cannot test it due to those runtime issues yet.[/quote]
[quote]no idea. i mean i just compile it with native to MSVS 2010 libs and that's it. should i replace MFC manually to that of 4.2?
[/quote]Don't know. If you use the attached project file in the src, then it is set up to manually to statically link with MFC lib, so you'll need to grab a copy of the MFC4.2 libs. I'm pretty sure the Microsoft SDK has it somewhere, but I pulled mine off ancient MSVS6.0 CDs.

[Suslik]

You do know that TobEx modifies the executable to LoadLibraryA() as well, so if you just launched bgmain.exe under debugger it should be OK, or maybe I miss the point.

Yep. If i launch BGMain.exe from under olly, ida, or MSVS debugger(i have just tested that), there is no reflection bug at all. If i lauch it normally, there it is bugged.

Lol, reading the disassembly isn't really hard because there are no code optimisations, and there is the awesome IESDP project that made everything possible in the first place.

boo, you're just jelly, because you will never reverse your first chunk of code again. and i just did it : )

Don't know. If you use the attached project file in the src, then it is set up to manually to statically link with MFC lib, so you'll need to grab a copy of the MFC4.2 libs. I'm pretty sure the Microsoft SDK has it somewhere, but I pulled mine off ancient MSVS6.0 CDs.

Unfortunately i dont have VC6.0 right now and my internets are too slow to download one. Would you kindly post the libs you use, or even include them to the src.rar inside the tobex archive?

offtop: FFFFUUUu, during the restore process i have accidentally overwritten my quicksave and autosave. previous saves are about a month ago, yay.

[Suslik]

Okay, i have found a workaround for the problem. i create a debuggable process instead of normal one. Check it out:
a little modified create process and a necessary dbg message loop:

	if ( CreateProcessA(
		reinterpret_cast<LPCSTR>(szFileName),
		NULL, //pointer to cmd arguments
		NULL, //process security
		NULL, //default thread security
		FALSE, //inherit handles?
		CREATE_SUSPENDED | DEBUG_PROCESS,
		NULL, //environment
		NULL, //current directory
		&si,
		&pi )
	) {
		DWORD dwExitCode = injectDLL(pi.hProcess, szLibPath);
		ResumeThread(pi.hThread);

		DEBUG_EVENT dbgEvent;
		while(WaitForDebugEvent(&dbgEvent, INFINITE))
		{
			ContinueDebugEvent(dbgEvent.dwProcessId, 
				dbgEvent.dwThreadId, 
				DBG_CONTINUE);
			if(dbgEvent.dwDebugEventCode == EXIT_PROCESS_DEBUG_EVENT) break;
		}

		CloseHandle(pi.hThread);
	    CloseHandle(pi.hProcess);


		if (!dwExitCode) return -1;
	}

then minor change in injection code. i have to receive dbg messages, because the process gets suspended and WaitForSingleObject is never fired:

	bool done = 0;
	while(!done)
	{
		DEBUG_EVENT dbgEvent;
		while(WaitForDebugEvent(&dbgEvent, 10))
		{
			ContinueDebugEvent(dbgEvent.dwProcessId, 
				dbgEvent.dwThreadId, 
				DBG_CONTINUE);
		}
		done = 1;
		switch (WaitForSingleObject(hRemoteThread, 10)) {
			case WAIT_ABANDONED:
				break;
			case WAIT_OBJECT_0:
				break;
			case WAIT_TIMEOUT:
				done = 0;
				break;
			case WAIT_FAILED:
				displayErrorMessage("injectDLL(): WaitForSingleObject() failed.", GetLastError());
				return 0;
			default:
				displayErrorMessage("injectDLL(): WaitForSingleObject() returned invalid value.", 0);
				return 0;
		}
	}

that's it, a workaround. no idea why it works, but it does, spells are reflected and absorbed perfectly.

i am still unable to test any other solutions(probably more explainable lol) due to lack of MFC42 headers/libs, but at least now we have a temporary workaround to compare with.

Edited by Suslik, 30 December 2011 - 10:51 PM.

[aqrit]

nice work Suslik

I think this is the laziest solution:

// set the HEAP_ZERO_MEMORY flag for calls to HeapAlloc
BACKUP ~blah_backup~
AUTHOR ~blah~
README ~~
VERSION ~v0~

BEGIN ~tob_spell_turning_workaround~
	COPY ~bgmain.exe~ ~bgmain.exe~ 
		WRITE_BYTE "0x0063951B" 0x08
		WRITE_BYTE "0x0063AE6E" 0x08
		WRITE_BYTE "0x00640E4B" 0x08