Transferring data between executables

Question

I have two executables written in C++ on Windows. I generate some data in one, and want to call the other executable to process this data. I could write the data out to a file then read it in the other executable, but that seems rather expensive in terms of disk I/O. What is a better way of doing this? It seems like a simple enough question but google just isn't helping!

Let's say the data is around 100MB, and is generated in its entirety before needing to be sent (i.e. no streaming is needed).

Answers that work when mixing 32 bit and 64 bit processes gain bonus points.

Pitanga

named pipes, local winsockets, memory mapped files, shared memory, mail slot, registry, window messages (not good for large data), what else? take your pick. if you tell us a little more about the data (size, etc.) and the nature to which you want to share the data, we can give better advice as far as which method is best. – Anile

In general, this is called inter-process communication. – Dakotadal

Sure, I've added a little information about the data. Originally I just wrote "large amount" but edited my question down before posting, I should really have just given some numbers :) – Pitanga

General term for what you are looking for is interprocess communication, I'm sure Google can help with that. Or be more specific, explain your goals and requirements, so more technical answer might be provided here. – Lap

Answer 1

If your processes can easily write to and read from file, just go ahead. Create the file with CreateFile and mark it as temporary & shareable. Windows uses this hint to delay physical writes, but all file semantics are still obeyed. Since your file is only 100 MB and actively in use, Windows is almost certainly able to keep its contents fully in RAM.

Millisecond

Ok, I didn't know this kind of caching was this reliable. Definitely the simplest solution, thanks :) – Pitanga

This sort of caching is just as reliable as keeping the data in RAM. Windows can page RAM to swapfile if needed. And behind the scenes, this is in fact the same mechanism. The Virtual Memory Manager keeps an association between RAM and the backing location on disk. – Millisecond

Answer 2

You can use Boost.MPI. It is from Boost, which has high quality standard, and the code sample seems pretty explicit:

http://www.boost.org/doc/libs/1_53_0/doc/html/mpi/tutorial.html#mpi.point_to_point

// The following program uses two MPI processes to write "Hello, world!"
// to the screen (hello_world.cpp):

int main(int argc, char* argv[])
{
  mpi::environment env(argc, argv);
  mpi::communicator world;

  if (world.rank() == 0) {
    world.send(1, 0, std::string("Hello"));
    std::string msg;
    world.recv(1, 1, msg);
    std::cout << msg << "!" << std::endl;
  } else {
    std::string msg;
    world.recv(0, 0, msg);
    std::cout << msg << ", ";
    std::cout.flush();
    world.send(0, 1, std::string("world"));
  }
  return 0;
}

Venture

Boost never ceases to amaze me! This seems like a good option, especially if I extend this to work over multiple machines, but for now I'll go with the simpler option of relying on file caching as MSalters suggests. – Pitanga

Answer 3

Assuming you only want to go "one direction" (that is, you don't need to get data BACK from the child process), you could use _popen(). You write your data to the pipe and the child process reads the data from stdin.

If you need bidirectional flow of data, then you will need to use two pipes, one as input and one as output, and you will need to set up a scheme for how the child process connects to those pipes [you can still set up the stdin/stdout to be the data path, but you could also use a pair of named pipes].

A third option is a shared memory region. I've never done this in Windows, but the principle is pretty much the same as what I've used in Linux [and many years back in OS/2]: 1. Create a memory region with a given name in your parent process. 2. The child process opens the same memory region. 3. Data is stored by parent process and read by child process. 4. If necessary, semaphores or similar can be used to signal completion/results ready/etc.

Mcmurry

I've learnt something new :) Thanks! Pipes sound like a good option but I'm guessing shared memory implies shared address space, which wouldn't work when mixing 32 and 64 bit processes. – Pitanga

As I haven't tried, I don't know, but I believe as long as you use a named shared memory, and don't exceed the address limit of 32-bit [so less than 2GB], it should work. Just remember to not put pointers within your shared memory. The memory isn't (necessarily) shared at the same address. – Mcmurry

Ah right, I misunderstood the code sample. Yes, it does look like it works that way. Thanks x2! – Pitanga

Using boost::interprocess::managed_windows_shared_memory did not work properly for me when mixing 32 and 64 bit. – Tactual