ESP Register not set properly / Calling Conventions...

[Milby7]

I've got a BIG problem when trying to call a dll, from a test console app!! I've written a simple dll that delegates calls to a 3rd party library, but is necessary because it will eventually be used in a C# app, & some 3rd party functions are actually in import libraries (inc. a DllMain, so i can't create my own), some are mapped via standard libs to dlls @ runtime.

(I've copied quite a bit of code below that should help to show what's going on)

When i call a function in the dll called SendMessageA i get a nasty error saying:
"Run-time check failure #0 - The value of ESP was not properly saved across a function call. This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention"

But when i call another function called GetEnvironment everything's fine!!

CLIENT APP:

int _tmain(int argc, _TCHAR* argv[])
{
	TCHAR szMsg[255];
	long lRes	  = 0;
	long lStatus  = 0;
	long lProcErr = 0;
	
	OutputSystemRegisters();
	
	cout << "Loading Uniface runtime API library..." << endl;
	
	hModule = ::LoadLibrary("urtapi7.dll");
	
	OutputSystemRegisters();
	
	cout << "Enter message to send to Uniface:";
	cin >> szMsg;
	
	cout << "Sending message " << "\"" << szMsg << "\" to Uniface via the asynchronous interrupt trigger..." << endl;
	
	UnifaceRuntimeApi::SendMessageA sendMessage = (UnifaceRuntimeApi::SendMessageA) ::GetProcAddress(hModule, "SendMessageA");
	
	OutputSystemRegisters();
	
        // * * * B A N G * * *
	lRes = sendMessage(szMsg);
	
	cout << "Message sent with status " << lRes << endl;
	
        // just while testing...
	TidyUp(); // (free uniface handles & free library)
	return 0;
	
	system("PAUSE");
	
	cout << "Getting Uniface environment handle..." << endl;
	
	GetEnvironment getEnv = (GetEnvironment) ::GetProcAddress(hModule, "GetEnvironment");
	
	// get uniface environment...
        // * * * T H I S   W O R K S * * *
	lRes = getEnv(&hUniEnv);
	
        // etc...

DLL STUFF:

// ** used by client app **


typedef long (__stdcall *SendMessageA) (LPSTR sMessage);

typedef long (__stdcall *GetEnvironment) (int* phUniEnv);


// ** some dll code **


long __stdcall SendMessageA(LPSTR sMessage)
{
	return uputames((unsigned char*) sMessage, (short) strlen(sMessage));	
}

long __stdcall GetEnvironment(int* phUniEnv)
{
	return ueopen((UHACT*) phUniEnv);
}


// ** 3rd party stuff **


extern long ueopen(UHACT *);
short __cdecl uputames(USTRING *pMsg, short sLen);

The 3rd party stuff is always pretty painfull to work with & i'm not sure if they're doing something in their DllMain that's screwing around with memory or something!? Their functions perform communication & other things with an executable program, which is running from a mapped network drive.

I've tried changing the calling conventions of the dll function SendMessageA (inc. typedef) to __cdecl, but that didn't work.

Should i be using pointers to pointers?? I don't really know when & when not to use them!!

If anyone has ANY ideas then i'd really appreciate the help as i'm pretty stuck!!

 

[Milby7]

I've found my way round that!! By simply renaming SendMessageA to SendAsyncMessageA!! Don't really know what was going on there as it was in its own namespace to avoid collisions with the Windows SendMessageA function. Anyway... it's still not actually fixed. I tried it with the function empty of code (ok), the i stuck the call back in as follows:

long __stdcall SendAsyncMessageA(LPSTR sMessage)
{
	return uputames((unsigned char*) sMessage, (short) strlen(sMessage));	
}

and i get a brand new error!!


Unhandled exception at 0x1000621f in Client.exe: 0xC0000005: Access violation reading location 0x00000038


Why would it be referencing system memory??

Has anyone seen anything like this before??

 

[Milby7]

I've just changed the function (SendAsyncMessageA) to do a simple printf & i get the same error. I get the message above except that it's trying to read 0x00000000 this time. But the "Locals" window says:

sMessage | 0x0012fdd8 "test" | char *

Is this where i should use a pointer to a pointer?? As I've said above I'm not too clear on when to use them, or why.

 

[Milby7]

I've just realised that the 3rd party headers redefine printf as one of their own functions. Grrrrrr!! I tested std:cout & that works, so it seems just to be the 3rd party stuff. If anyone knows anything that could cause this behaviour, then please let me know.

Thanks!!