C#•2mo ago

P/Invoke boolean call always returns true in Release builds

I have a simple boolean function in a native C++ DLL (for keyboard input in my case), which I P/Invoke from C#. In debug builds, everything works fine. But in a release build (of both the C# and C++ projects), the C# side always receives true from the P/Invoke call, even when the native side returns false. I've checked my build settings but I can't figure out what is happening here- any recommendations for how to debug this or what could be the potential cause? For reference - my code looks a little bit something like this: C++:

bool sb_IsKeyDown(int key)
{
  return keyboardState[key];
}

bool sb_IsKeyDown(int key)
{
  return keyboardState[key];
}

C#:

[DllImport(DLL_NAME, CallingConvention = CallingConvention.Cdecl)]
private static extern bool sb_IsKeyDown(int key); // Returns true when it shouldn't in Release builds.

[DllImport(DLL_NAME, CallingConvention = CallingConvention.Cdecl)]
private static extern bool sb_IsKeyDown(int key); // Returns true when it shouldn't in Release builds.

Please ping me on reply, thank you!

9 Replies

reflectronic•2mo ago

you are not marshalling the bool correctly bool in a P/Invoke is a 4 byte bool this means that whatever garbage is left in the upper bits of the return register can cause the marshaller to treat it as true use [return: MarshalAs(UnmanagedType.I1)]