Calling convention mismatch between call-site and callee for architecture plugins

[Lukas-Dresel]

When I built an architecture plugin for a custom VM challenge from DiceCTF this weekend, I noticed that most (if not all) calls did not have any arguments when seen in ML/HL-IL. This was despite the function itself clearly having recognized the arguments correctly.

So somehow the results of the function call site and function definition do not seem to be merged correctly. Ideally, the expected behavior on my end would be that the (especially if manually entered) calling convention defined in the function prototype is used, unless a Force Call Override interferes at the call-site. On that note, forcing the call type at the call-site also had no effect on the result of the higher level ILs which is unexpected.

Example:

Function call site with no arguments shown (incorrectly):

Function definition with correctly detected arguments in the definition.

These screenshots are from the exact same session, simply once after clicking inside the function to view the actual definition.

Of note: Once I implemented a calling convention, functions started showing up as having arguments, oftentimes still with incorrect arguments however (what the calling convention specified vs. what the definition said) This matters, since for this VM no real calling convention exists, some functions take arguments in R2,R3,R4 others in R8,R9,R10. For the CTF, I fell back on a most-common calling convention with R8-R15 set as arguments, and had to remember which function signatures were incorrect.

The slack discussion surrounding this can be seen @ https://binaryninja.slack.com/archives/C1E616ZSQ/p1644114031680159.

[Lukas-Dresel]

I just pushed the disassembler and the plugin. With the current version of the plugin, the calls do resolve arguments after I added a custom calling convention, however the results are incorrect in the sense that some functions take arguments that are not given in the calling convention. These do not show up.

However, to reproduce the original issue, simply unimport the calling convention and platform from the __init__.py

[fuzyll]

Sorry it took so long to get back to you on this.

My understanding is that we look at function calls and try to match them to calling conventions that have been defined for the architecture. If one isn't defined, or doesn't match, we won't be able to determine what the parameters are for the function automatically. So, for your specific use-case here, we would expect you to define calling conventions for anything common and then mark up the rest manually (e.g. with var @ reg in the function prototype). It's possible you might be able to use a Workflow in the future to do some additional analysis to catch these yourself, but I don't think we currently support that.

I think this means the behavior shown above is "working as intended". If "some functions take arguments that are not given in the calling convention", we'd expect that to be resolved manually. But, if the opposite happens (a function takes arguments that are in the calling convention, but aren't recognized and don't show up), that would be a bug and we should get it fixed. If you have an example of the latter, please file an issue and let us know.

[Lukas-Dresel]

Sorry, just now saw the reply.

I would argue that the vunction signature should have precedence over whatever a generic calling convention says. Otherwise you're also going to run into issues with compilers that can change register use in function prototype for internal functions.

The problem is not the arguments of the calling convention not showing up, it's that the arguments in the function signature do not show up at the call site. I would argue that a function signature (especially a manually specified one) should supersede anything else (bar a manual call type annotation ala IDAs Force call type).

In general, the calling convention can inform the signature analysis, e.g. as a default, but should not dictate it IMO

Also, I did specify the registers with var @ reg, e.g. see arg1 @ R2 in the function prototype, this happened despite that.

[Lukas-Dresel]

Sorry this sounded a bit saltier than expected 😅

Long story short, this even happened with a custom, manually annotated function signature.