How to construct a c++ fstream from a POSIX file descriptor?

Question

I'm basically looking for a C++ version of fdopen(). I did a bit of research on this and it is one of those things that seems like it should be easy, but turns out to be very complicated. Am I missing something in this belief (i.e. it really is easy)? If not, is there a good library out there somewhere to handle this?

EDIT: Moved my example solution to a separate answer.

Alyse

Windows and Linux can do mmap to the file and exposed its content as byte array. – Punchboard

Answer 1

From the answer given by Éric Malenfant:

AFAIK, there is no way to do this in standard C++. Depending on your platform, your implementation of the standard library may offer (as a nonstandard extension) a fstream constructor taking a file descriptor as input. (This is the case for libstdc++, IIRC) or a FILE*.

Based on above observations and my research below there's working code in two variants; one for libstdc++ and another one for Microsoft Visual C++.

---

libstdc++

There's non-standard __gnu_cxx::stdio_filebuf class template which inherits std::basic_streambuf and has the following constructor

stdio_filebuf (int __fd, std::ios_base::openmode __mode, size_t __size=static_cast< size_t >(BUFSIZ)) 

with description This constructor associates a file stream buffer with an open POSIX file descriptor.

We create it passing POSIX handle (line 1) and then we pass it to istream's constructor as basic_streambuf (line 2):

#include <ext/stdio_filebuf.h>
#include <iostream>
#include <fstream>
#include <string>

using namespace std;

int main()
{
    ofstream ofs("test.txt");
    ofs << "Writing to a basic_ofstream object..." << endl;
    ofs.close();

    int posix_handle = fileno(::fopen("test.txt", "r"));

    __gnu_cxx::stdio_filebuf<char> filebuf(posix_handle, std::ios::in); // 1
    istream is(&filebuf); // 2

    string line;
    getline(is, line);
    cout << "line: " << line << std::endl;
    return 0;
}

---

Microsoft Visual C++

There used to be non-standard version of ifstream's constructor taking POSIX file descriptor but it's missing both from current docs and from code. There is another non-standard version of ifstream's constructor taking FILE*

explicit basic_ifstream(_Filet *_File)
    : _Mybase(&_Filebuffer),
        _Filebuffer(_File)
    {   // construct with specified C stream
    }

and it's not documented (I couldn't even find any old documentation where it would be present). We call it (line 1) with the parameter being the result of calling _fdopen to get C stream FILE* from POSIX file handle.

#include <cstdio>
#include <iostream>
#include <fstream>
#include <string>

using namespace std;

int main()
{
    ofstream ofs("test.txt");
    ofs << "Writing to a basic_ofstream object..." << endl;
    ofs.close();

    int posix_handle = ::_fileno(::fopen("test.txt", "r"));

    ifstream ifs(::_fdopen(posix_handle, "r")); // 1

    string line;
    getline(ifs, line);
    ifs.close();
    cout << "line: " << line << endl;
    return 0;
}

Hibachi

Now the accepted answer due to completeness. Others may be interested in my solution using boost, which has been moved to a separate answer. – Alyse

For linux: If you look at ios_init.cc in gcc (the source I have is for version 4.1.1) std::cout is initialised by initialising a stdio_sync_filebuf<char> around your file descriptor, then initialising on ostream around your stdio_sync_filebuf<char>. I can't claim that this is going to be stable though. – Fulbright

@Fulbright Looking into std::cout implementation is a good idea. I'm wondering what's the difference between stdio_filebuf and stdio_sync_filebuf? – Hibachi

POSIX fds in MSVC are emulation. Windows API for file operations differs from POSIX ones in many ways - different functions names and data types of parameters.Windows internally uses so called "handles" to identify various Windows API objects, and Windows API type HANDLE is defined as void*, so at minimum it will not fit into "int" (which is 32-bit) on 64-bit platforms. So for Windows you may be interested in looking for stream that allows to work over Windows API file HANDLE. – Atiptoe

@PiotrDobrogost: the _sync_... version translates calls directly to the underlying FILE*, unlike normal iostreams which do some buffering. I'd like to add that this code leaks the FILEs -- you wouldn't use it like that normally (or, you could initialize the buffers with the FILE* itself), but the whole thing is useful to get an iostream talking to a socket(), pipe(), memfd_create()'d fd. – Quag

Answer 2

AFAIK, there is no way to do this in standard C++. Depending on your platform, your implementation of the standard library may offer (as a nonstandard extension) a fstream constructor taking a file descriptor (This is the case for libstdc++, IIRC) or a FILE* as an input.

Another alternative would be to use a boost::iostreams::file_descriptor device, which you could wrap in a boost::iostreams::stream if you want to have an std::stream interface to it.

Dejecta

Considering that this is the only portable solution, I don't understand why this isn't the accepted or top-rated answer. – Princely

I think this answer shows an example: #1958261 – Aerology

Note that as of boost 1.81 however, boost::iostreams seemed to not support moves however. So its usability in modern C++ seems decreased significantly IMO. – Matazzoni

Answer 3

Part of the original (unstated) motivation of this question is to have the ability to pass data either between programs or between two parts of a test program using a safely created temporary file, but tmpnam() throws a warning in gcc, so I wanted to use mkstemp() instead. Here is a test program that I wrote based on the answer given by Éric Malenfant but using mkstemp() instead of fdopen(); this works on my Ubuntu system with Boost libraries installed:

#include <stdlib.h>
#include <string.h>
#include <assert.h>
#include <string>
#include <iostream>
#include <boost/filesystem.hpp>
#include <boost/iostreams/device/file_descriptor.hpp>
#include <boost/iostreams/stream.hpp>

using boost::iostreams::stream;
using boost::iostreams::file_descriptor_sink;
using boost::filesystem::path;
using boost::filesystem::exists;
using boost::filesystem::status;
using boost::filesystem::remove;

int main(int argc, const char *argv[]) {
  char tmpTemplate[13];
  strncpy(tmpTemplate, "/tmp/XXXXXX", 13);
  stream<file_descriptor_sink> tmp(mkstemp(tmpTemplate));
  assert(tmp.is_open());
  tmp << "Hello mkstemp!" << std::endl;
  tmp.close();
  path tmpPath(tmpTemplate);
  if (exists(status(tmpPath))) {
    std::cout << "Output is in " << tmpPath.file_string() << std::endl;
    std::string cmd("cat ");
    cmd += tmpPath.file_string();
    system(cmd.c_str());
    std::cout << "Removing " << tmpPath.file_string() << std::endl;
    remove(tmpPath);
  }
}

Alyse

Answer 4

There's a good chance your compiler offers a FILE-based fstream constructor, even though it's non-standard. For example:

FILE* f = fdopen(my_fd, "a");
std::fstream fstr(f);
fstr << "Greetings\n";

But as far as I know, there's no portable way to do this.

Cortisone

Note that g++ (correctly) won't allow this in c++11 mode – Excess

Answer 5

It actually is quite easy. Nicolai M. Josuttis has released fdstream in conjunction with his book The C++ Standard Library - A Tutorial and Reference. You can find the 184 line implementation here.

Hysterectomize

This is an interesting application of the phrase "quite easy". – Barranquilla

It becomes easier if you use the Boost class that incorporates that code, as suggested in another answer. – Apothem

Answer 6

I've tried the solution proposed above for libstdc++ by Piotr Dobrogost, and found that it had a painful flaw: Due to the lack of a proper move constructor for istream, it's very difficult to get the newly constructed istream object out of the creating function. Another issue with it is that it leaks a FILE object (even thought not the underlying posix file descriptor). Here's an alternative solution that avoids these issues:

#include <fstream>
#include <string>
#include <ext/stdio_filebuf.h>
#include <type_traits>

bool OpenFileForSequentialInput(ifstream& ifs, const string& fname)
{
    ifs.open(fname.c_str(), ios::in);
    if (! ifs.is_open()) {
        return false;
    }

    using FilebufType = __gnu_cxx::stdio_filebuf<std::ifstream::char_type>;
    static_assert(  std::is_base_of<ifstream::__filebuf_type, FilebufType>::value &&
                    (sizeof(FilebufType) == sizeof(ifstream::__filebuf_type)),
            "The filebuf type appears to have extra data members, the cast might be unsafe");

    const int fd = static_cast<FilebufType*>(ifs.rdbuf())->fd();
    assert(fd >= 0);
    if (0 != posix_fadvise(fd, 0, 0, POSIX_FADV_SEQUENTIAL)) {
        ifs.close();
        return false;
    }

    return true;
}

The call to posix_fadvise() demonstrates a potential use. Also note that the example uses static_assert and using which are C++ 11, other than that it should build just fine in C++ 03 mode.

Inconveniency

What do you mean by proper version of move constructor? What version of gcc did you use? Maybe this version did not have move constructors implemented yet – see Is the move constructor of ifsteam implicitly deleted?? – Hibachi

This is a hack that is dependent on underlying implementation details. I would hope nobody ever uses this in production code. – Ting

...not only it is a hack which might not work, but if you want it, why not use dynamic_cast? – Quag

Answer 7

Another non-portable solution is to use mmap (or its Windows' analogue) and then construct std::iostream from a pointer that mmap gave like so.

Yeah, it does not construct exactly an std::fstream, but this requirement rarely needs to be met because every piece of code should depend on stream interfaces (e.g. std::istream) rather than on their implementations.

I think this solution is more portable than use of STL implementation-specific hacks, because this way you only depend on an operating system, rather than on a specific implementation of STL for the same OS.

Percival

Answer 8

My understanding is that there is no association with FILE pointers or file descriptors in the C++ iostream object model in order to keep code portable.

That said, I saw several places refer to the mds-utils or boost to help bridge that gap.

Thorvald

FILE* is standard C and thus C++ so I don't see how enabling C++ streams to work with C streams could harm portability – Hibachi