[ccan] Using ccan with a C python extension

[Cody P Schafer]

On Mon, Jun 15, 2015 at 6:01 PM, Rusty Russell <rusty at rustcorp.com.au> wrote:
> Stuart Longland <stuartl at longlandclan.yi.org> writes:
>> On 15/06/15 12:01, Rusty Russell wrote:
>>> Naveen Nathan <naveen at lastninja.net> writes:
>>>> It is setup to be installed as a python package via pip/easy_install
>>>> which will:
>>>>     (1) fetch the python module
>>>>     (2) build the C extension, and finally
>>>>     (3) install it on the system (or virtual environment).
>>>>
>>>> I specifically want to use ccan/{bitmap,endian} but unsure how
>>>> to integrate the code into our project's build process.
>>>>
>>>> I envision at step (2) above, we will build libccan and statically
>>>> link it to the C extension. However, I will need a ccan config.h
>>>> which will vary per system. So this will involve generating the
>>>> file. Does that mean I should include the tools / configurator
>>>> as part of our python package?
>>>
>>>         I think you should use autoconf, as it's the standard.  I keep
>>> hoping someone will write the autotools stuff for CCAN for me :)
>>
>> Yeah, the trouble with that approach is that autoconf does a pretty poor
>> job of integrating into Python module builds.
>>
>> The usual build system in Python is the `distutils` module, which
>> basically gets included into a Python script called 'setup.py' and called.
>>
>> Then it's a *magic happens here* affair, where out the other end is your
>> Python module.
>>
>> You'd have to hook something in that could call ./configure at the
>> appropriate time, and somehow convince the automated packaging tools
>> (stdeb, gsourcery-pypi, etc) to ignore ./configure and use setup.py.
>>
>> I'm not sure what's needed in config.h; maybe there's some way to
>> generate this using some basic Python calls before building?
>
> It depends very much which modules.  For bitmap and endian you need:
>
> HAVE_BUILTIN_CLZL
> HAVE_BYTESWAP_H
> HAVE_BSWAP_64
> HAVE_LITTLE_ENDIAN / HAVE_BIG_ENDIAN
>
> So you could get away with something like:
>
> if sys.byteorder == 'little':
>    print "#define HAVE_LITTLE_ENDIAN 1"
> else:
>    print "#define HAVE_BIG_ENDIAN 1"

Do note that doing this will make any people trying to cross-compile
across endian (not common due to the current decline in BE usage, but
possible).
Of course, the current configurator has the same issue :)
(HAVE_*ENDIAN is a executed test).

I'm not sure how autotools solves this for cross compilation, but I'd
guess it's something they are guessing based on a target's triple.

In this case, I don't think it really pays off to try to guess based
on triples (the above check is going to be correct most of the time)
To be nice to packagers, though, it would be useful to provide a way
to override the endian auto-detect.

>
> print """#if __GNUC__ >= 4
> #define HAVE_BUILTIN_CLZL 1
> #endif
> #ifdef __linux__
> #define HAVE_BYTESWAP_H 1
> #define HAVE_BSWAP_64 1
> #endif"""

I've considered a few times that it might be useful to have a config.h
that trys it's best to guess at the HAVE_* status based on the
compiler provided defines.

Also relevant to the above: clang & gcc do predefine some macros for
byte order that might be useful, if you go this route.

x at arnold ~ $ echo | clang -E -dM  -x c - | grep END
#define __BYTE_ORDER__ __ORDER_LITTLE_ENDIAN__
#define __LITTLE_ENDIAN__ 1
#define __ORDER_BIG_ENDIAN__ 4321
#define __ORDER_LITTLE_ENDIAN__ 1234
#define __ORDER_PDP_ENDIAN__ 3412
x at arnold ~ $ echo | gcc -E -dM  -x c - | grep END
#define __ORDER_LITTLE_ENDIAN__ 1234
#define __FLOAT_WORD_ORDER__ __ORDER_LITTLE_ENDIAN__
#define __ORDER_PDP_ENDIAN__ 3412
#define __ORDER_BIG_ENDIAN__ 4321
#define __BYTE_ORDER__ __ORDER_LITTLE_ENDIAN__