Preprocessor tomfoolery (stringifying a #include)

Question

Note: This question has nothing to do with OpenCL per se... check the last paragraph for a succinct statement of my question. But to provide some background:

I'm writing some C++ code that makes use of OpenCL. I like to keep the source for my OpenCL kernels in their own files, to keep coding and maintenance easy (as opposed to embedding the sources directly as string constants in associated C++ code). This inevitably leads to the question of how to load them into the OpenCL runtime once it comes time to distribute binaries---ideally, the OpenCL source is included in the binary, so that the binary doesn't need to be in a specific place within some directory structure to know where the OpenCL source code is.

I'd like to include the OpenCL files as string constants somewhere, and preferably without the use of additional build steps or external tools (for cross-compiler/cross-platform ease of use... i.e., no to xxd and the like). I thought I'd stumbled on a technique based on the second answer in this thread, like so:

#define STRINGIFY(src) #src

inline const char* Kernels() {
  static const char* kernels = STRINGIFY(
    #include "kernels/util.cl"
    #include "kernels/basic.cl"
  );
  return kernels;
}

Note that I'd prefer not to embed the STRINGIFY macro in my OpenCL code if at all possible (as was done in the above referenced SO question). Now, this works wonderfully on the Clang/LLVM compiler, but GCC dies a horrible death ("Unterminated argument list invoking macro STRINGIFY" and various syntax "errors" related to the contents of the .cl files appear). So, clearly this exact technique isn't usable across compilers (haven't tried MSVC, but I'd like it to work there too)... How could I massage it minimally so that it works across compilers?

In summary, I'd like a standards-compliant technique for including the contents of a file as a C/C++ string constant without invoking external tools or polluting the files with extraneous code. Ideas?

EDIT: As Potatoswatter pointed out, the behavior of the above is undefined, so a truly cross-compiler preprocessor technique that doesn't involve touching the files-to-be-stringified probably isn't possible (first person to figure out a heinous hack that does work for most/all compilers gets the answer points). For the curious, I ended up doing what was suggested in the second response here... that is, I added the STRINGIFY macro directly to the OpenCL files I was including:

In somefile.cl:

STRINGIFY(
  ... // Lots of OpenCL code
)

In somefile.cpp:

#define STRINGIFY(src) #src

inline const char* Kernels() {
  static const char* kernels =
    #include "somefile.cl"
    ;
  return kernels;
}

This works in the compilers I've tried it in (Clang and GCC as well, since it doesn't have preprocessor directives inside the macro), and isn't too large a burden at least in my context (i.e., it doesn't interfere with syntax highlighting/editing the OpenCL files). One feature of preprocessor approaches like this one is that, since adjacent strings get concatenated, you can write

inline const char* Kernels() {
  static const char* kernels =
    #include "utility_functions.cl"
    #include "somefile.cl"
    ;
  return kernels;
}

and as long as the STRINGIFY macro is in both .cl files, the strings get concatenated, allowing you to modularize your OpenCL code.

Jackjackadandy

As for concatenating the CL code, clCreateProgramWithSource accepts a list of strings. – Anechoic

Ah true... had forgotten about that. Though I think the above approach is still easier overall, since the code to compile the cl files is extremely simple. – Jackjackadandy

Why not just have an #include as the source code? OpenCL preprocesses the code too, and you can specify the include folder via the -I compiler option. Makes for very comfortable file loading... just let the runtime to do it for you! Not sure if it would handle binaries, but I suppose it would work the same. – Rambler

Answer 1

The most relevant part of the Standard is §16.3/10:

The sequence of preprocessing tokens bounded by the outside-most matching parentheses forms the list of arguments for the function-like macro. The individual arguments within the list are separated by comma preprocessing tokens, but comma preprocessing tokens between matching inner parentheses do not separate arguments. If (before argument substitution) any argument consists of no preprocessing tokens, the behavior is undefined. If there are sequences of preprocessing tokens within the list of arguments that would otherwise act as preprocessing directives, the behavior is undefined.

Extracting the key points:

• You need to enclose the header files within a pair of parentheses so the macro doesn't think that every comma character in the file introduces another argument. These parentheses will also be stringized, but shouldn't be hard to work around.
• Putting #include in an argument list at all is officially undefined behavior, so this is going to be unportable. The compiler officially doesn't know whether you want the resulting string to be "#include \"kernels/util.cl\"".

Donner

"If there are sequences of preprocessing tokens within the list of arguments that would otherwise act as preprocessing directives, the behavior is undefined" Ah... :( – Jackjackadandy

Answer 2

The conventional technique is using a program like bin2c, usually hastily written. Another method is using objcopy from GNU binutils:

$ objcopy -I binary extensions.cfg -O elf32-little -B i386 --rename-section .data=.rodata extensions.o
$ objdump -x extensions.o

extensions.o:     file format elf32-i386
extensions.o
architecture: i386, flags 0x00000010:
HAS_SYMS
start address 0x00000000

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .rodata       00000447  00000000  00000000  00000034  2**0
                  CONTENTS, ALLOC, LOAD, DATA
SYMBOL TABLE:
00000000 l    d  .rodata        00000000 .rodata
00000000 g       .rodata        00000000 _binary_extensions_cfg_start
00000447 g       .rodata        00000000 _binary_extensions_cfg_end
00000447 g       *ABS*  00000000 _binary_extensions_cfg_size

The -O and -B flags have to match the objdump output for one of your compiled object files, to satisfy the linker, while the section renaming is just to inform the runtime linker this data is read-only. Note the symbols, mapping to start address, end address and data size. They each count as addresses, so in C you'd use them with something like:

extern const char _binary_extensions_cfg_start, _binary_extensions_cfg_end;
extern const char _binary_extensions_cfg_size;
for (const char *p=&_binary_extensions_cfg_start; p<&_binary_extensions_cfg_end; p++)
    do_something(p);
memcpy(somewhere, &_binary_extensions_cfg_start, (intptr_t)&_binary_extensions_cfg_size);

I realize neither of these is the preprocessor thing you're asking for, but it simply wasn't designed to do that. Nevertheless, I would be interested to know if it can.

Anechoic

Answer 3

You can't do it that way; I'm surprised it worked in clang. If you want to include text files directly in your binary, you have some options:

1: A preprocess that runs before compilation that converts your .cl files into a .cpp file that defines the string.

2: Storing the string data via compiler-specific storage in your compiled executable. This is how tools like Visual Studio include .rc, .ico, and other files that are used to build Windows applications. Of course, as stated, these are compiler-specific.

The safest way is option 1.

Kylstra

As I said, I'd prefer not to add external tools/extra build steps. What I find odd is that the technique I used above works for GCC as well if I add the macro directly to the file being #included, so something close to this must be workable. – Jackjackadandy

Answer 4

Here's what I do in xcode C:

const char oursource[60000];
const char * oursourceptr = oursource;
const char * * oursourceptrptr = & oursourceptr;

// in function "readfileintostring":

char *fylnm = "/Developer/projs/myproj/mykernel.cl";

long enby; short pathref;

FSRef dink; FSPathMakeRef( (const UInt8 *) &fylnm, &dink, NULL );
SInt16 forkRefNum;  HFSUniStr255 dataForkName;  FSGetDataForkName(&dataForkName);
FSOpenFork( &dink, dataForkName.length, dataForkName.unicode, fsRdPerm, (FSIORefNum *) &pathref );

enby = 100000;  FSRead( pathref, &enby, (void *) oursourceptr );

// .. then later ..

program = clCreateProgramWithSource(context, 1, (const char **) oursourceptrptr, NULL, &err);

... it's not preprocessor voodoo, but it works for me, I can see syntax highlighted in my .cl file, and I can even copy my .cl over into a .c, change one #define, and it runs as xcode C....

Sarmiento

Unless I'm missing something, does' this read the .cl file at runtime...? The OP is asking for a solution that includes the file's content as a string at compile time. – Mariettamariette