Skip to content

StructPackingAndApiCallingConvention

Jump to bottom
Wiki converter edited this page Sep 24, 2020 · 2 revisions

Struct Packing and API Calling Convention

Ticket #58 is a patch requesting that portaudio.h include explicit calling convention and structure packing #pragmas (on Windows). Since PA is cross-platform we need to make a cross-platform decision. This page documents the considerations relating to accepting/modifying/rejecting this patch.

We are discussing this issue in the "[LIST-REVIEW] #58 API calling convention and pack" mailing list thread, started here: http://music.columbia.edu/pipermail/portaudio/2011-February/011534.html

The goal is to definitively choose a position for V19 and beyond.

Executive Summary

Currently portaudio.h doesn't specify explicit struct packing or calling convention. This means that a DLL or shared library will be built according to prevailing compiler settings and defaults. It also means that a user of the DLL or shared library needs to use the same calling conventions and struct packing settings used when the DLL was built. This is usually OK if the DLL is built by the same developer who uses the DLL, but not ideal if the consumer of the DLL is a third party, especially if the user in a different language or environment where the native C compiler defaults may be unsupported.

There are really two here issues:

  1. Choice of whether and how to specify structure packing
  2. Choice of whether to specify calling convention

These are addressed separately below.

It has been noted that these are completely separate issues. They are related in that they are both compiler settings that can get out of sync between DLLs and dependent executables, and they would both be configured in portaudio.h

Known Issues with Current PA Situation

  • Ticket #58 notes that for DLLs calling convention and pack needs to be specified in the header file if clients want or need to use different default settings in their projects.

  • PA build instructions have previously suggested 4-byte struct alignment flag /Zp4 for making DLLs compatible with some environments (VisualBasic, PureBasic etc). Using this compiler setting creates DLLs that are incompatible with the usual MSVC default 8 byte alignment. Objections have been raised to having struct packing settings (eg /Zp4) in the build instructions and it has been suggested that using #pragma pack(4) in the header files would be preferable. [ NOTE: the build instructions have been revised to suggest /Zp4 only when needed ].

Thomas Blom notes that much of the controversy arises from providing or supporting binary libs:

What you say seems sensible if we are in the business of providing binary libs. If instead we officially only offer the source then it would seem people either build everything with compiler defaults, and it works, or people build everything with custom packing, and it works.

Thus there is a subtext to this discussion that we may provide binary DLLs on Windows in the future.

Choice of Structure Packing

Background Reading, facts, options and opinions

Phil writes:

Very good discussion of packing pragmas, padding and alignment here: http://en.wikipedia.org/wiki/Data_structure_alignment

VisualBasic/COM type libraries assumes 4 byte packing. MSVC default packing is 8 bytes. Default on Linux x86 is 4 bytes.

In C there are 4 different mechanisms that can be used to ensure consistent struct packing:

  1. Compiler flags: These apply to the whole project and all #included header files where no packing is explicitly specified.
  2. #pragma pack() directive: This is compiler-specific but available with MSVC and gcc at least. similar mechanisms are available in other compilers.
  3. Manually pad structs with extra fields to ensure alignment.
  4. Sort fields by size, largest first

In many (but not all) cases it is possible to make (2), (3) and (4) immune to compiler flags (1). These options are discussed further below.

Rob Bielik notes:

gcc (as well as MSVC) supports the #pragma pack(push, n) and #pragma pack(pop) directives, which would remove (override) the need for /Zp4 setting.

#pragmas can only reduce alignment below compiler settings so #pragma pack(8) is not sufficient to provide compatibility with compilers with /Zp4 set.

Phil notes:

We could also control alignment by reordering the members in our structures or padding the structures. This might break the binary interface between an application and a DLL. But so will changing structure alignment using pragmas. Once we do the reordering we will be pretty much immune to compiler settings.

Note that manual padding is only legitimate if the size of all types is known. The PA API currently uses ints (for example), which may be 2, 4, or 8 bytes in size depending on compiler settings. We would need to update the PortAudio API to use types that have known size (ie PaInt32) for this to work.

Peter thinks that using #pragma pack is better than manual padding and notes that Delphi and Lazarus provides the $Align directive for this purpose: http://www.delphibasics.co.uk/RTL.asp?Name=$Align

Chris and Phil determined that for PureBasic it is possible to manually pad the struct definitions to conform alignment (either on the PA side or the PB side).

Peter, Robert, Michael and Thomas dislike the padding approach, mostly for aesthetic reasons it appears:

I would not recommend "padding the wholes".

I think pad members are an ugly way to solve it

I also consider padding parameters as very ugly and a long term maintainance load

+1 to Michael. I think manual padding is a hack.

Regarding manual padding, Phil counters:

Well. It is kind of a hack. But a common one and somewhat useful. I sometimes call pad members "reserved". Then it looks like I am planning ahead and not just resorting to an ugly hack. My first choice for avoiding alignment problems is to sort the members from big to small. But it may be too late for that in PortAudio.

Note that these alignment problems are not restricted to foreign language bindings. They can bite you even when calling from a 'C' application to a 'C' DLL.

Note that calling fields "reserved" raises the issue of whether they are required to be set to zero and whether we do anticipate using them in the future or whether they are guaranteed to remain unused (and hence OK to contain garbage.

Thomas adds some analysis:

There is a tradeoff in simplicity/transparency of implementation and requiring a user to do some amount of work to use the library. If I were designing such a library, I'd opt for the simplest implementation, making it easiest to understand, working "out of the box" for the most common use-cases, and require that external bindings with special requirements use the facilities of the external language/platform to satisfy those requirements. For me this would mean no pragmas and no manual padding, unless their are performance reasons even in the common use cases to do so.

This probably depends a lot on how portable the lib aims to be. I am amused when I see patches added to libs to support "modula-2 on my amiga 1000" ... this is obviously really cool in one sense (that example is actually from my own programming past) but is the kind of thing that makes portable libraries choc-full of #ifdef bs that make them seem way more complicated than they actually are, which has it's own set of problems in terms of maintenance by open-source contributors etc...
I suppose PortAudio in particular may be precisely the kind of library that folks with ancient hardware/software want to use in the realm of "retro" music synthesis etc...so my rant may be off base. But you can always fork PA and mod it yourself if you're going to geek out with your old amiga. ;)

Informal Survey of Other Libs

Looking at other libs we would not be alone to leave structure pack unspecified in our header files:

library Specifies pragma pack pads/sorts to align data notes
GLFW NO ?
freetype NO ?
freeimage pragma pack(1)
libsndfile NO ? (and the win32 lib is provided as a binary)
ASIO pragma pack(4) ? possibly due to ASIO using COM
Lua NO ?
TinyXML NO ?
Boost YES ? (see config\abi_prefix.h config\abi_suffix.h)
mingw /include/*.h YES ? (1,2,4,8 for different files)
zlib NO NO (structs also interleave bytes and ints etc)

Side note: Boost provides a BOOST_DISABLE_ABI_HEADERS macro to turn off enforcing pack.

Ross writes:

http://www.zlib.net/DLL_FAQ.txt

zlib is very widely used (in libpng for example).

Interestingly zlib doesn't appear to specify any packing in the header files, and their data structures include unsorted fields with layout like this:

typedef struct z_stream_s {
    Bytef    *next_in;  /* next input byte */
    uInt     avail_in;  /* number of bytes available at next_in */
    uLong    total_in;  /* total nb of input bytes read so far */

Impact

It has been noted that any change to structure packing will break binary compatibility with clients. Some bindings can be updated with a simple change to set alignment. Others may require manual padding.

Reordering fields would break bindings a lot more (they would need to also re-do their struct declarations). Peter called for a version number API to deal with this. Ross noted that struct versioning is due for review under ticket #78.

The following PA structs contain 8 byte doubles that would not be 8-byte aligned with pack(4):

Note that structVersion fields are slated for removal under ticket #78 so only PaDeviceInfo may end up needing explicit padding.

Pros / Cons

Pros Cons
\2. Document current status quo (ie canonicalise dominant 'C' platform defaults)
* No source code change. * Sensitive to user changing compiler settings.
* Don't have to decide on specifics, just use defaults. * Windows default of pack(8) not directly compatible with some non-C environments.
\2. #pragma pack() enforced status quo (ie pack(8) on Windows, pack(4) on Linux x86)
* pack(8) can be broken by /Zp4 on Windows -- so no real benefit, also risks confusion.
\2. #pragma pack(4)
* Makes things clear. * 8 byte doubles are not perfectly aligned given current struct layouts.
* Compatible with VisualBasic, PureBasic etc.
* Enforces consistency with both Windows and Linux
\2. Manual padding
* Gives tight as possible padding and correct alignment of doubles. * Requires switching to known size types (e.g. defining PaInt32) to be meaningful on ia64
* Shouldn't require #pragma pack() * adds extra fields to current structs (won't break binary compatibilty with pack(8) users though.
* Compatible with VisualBasic, PureBasic etc. * ugly. at least 50% of devs think it's a hack.
* if fields are marked "reserved" and are expected to be zero, client code would need to change to init them to zero.
\2. Re-sorting fields by size
* doesn't require #pragma pack() * Guarantees natural alignment.
* Compatible with VisualBasic, PureBasic etc. * Requires non-C bindings structure definitions to be re-written.

Choice of Calling Convention

So far this is a less contentious topic than structure alignment. It also seems more common for calling convention to be specified in library headers.

Background Reading

On Windows, calling convention not only affects the actual calling convention mechanics, but also how exported function names are "decorated" (ie name mangling rules). Each compiler treats things differently as documented here: http://wyw.dcweb.cn/stdcall.htm

The zlib FAQ provides extensive discussion of the calling convention choices they made: http://www.zlib.net/DLL_FAQ.txt

Zlib provides multiple options for calling convention. By default it specifies none ( _ cedecl on Windows) but provides an option to use WINAPI ( _stdcall). The advantage of using the default is that clients don't need to specify a specific calling convention with their callback functions. The link makes a good argument that _ _cdecl should be/is the default for 'C' standard libraries. It means that user defined callbacks will just work. See their FAQ for further argument.

Using default (_ _cdecl) means you can't use their official zlib DLL with VisualBasic.

Ross notes:

In addition to calling convention I was also thinking that on Windows we want to give the option to use _ _declspec(dllimport) in the header for clients using the dll:

This goes before the return value specification like this:

_ _declspec( dllimport ) int _ _stdcall Pa_GetVersion( void );

So the macros would look like either:

PA_DLL_IMPORT int PA_API_CC Pa_GetVersion( void );

or as I have seen elsewhere:

PA_API_FUNC(int) Pa_GetVersion( void );

e.g. Python's PyAPI_FUNC macro: http://svn.python.org/projects/python/branches/release25-maint/Include/pyport.h

PortAudio

Clone this wiki locally