Why can I not mmap /proc/self/maps?
Asked Answered
S

3

4

To be specific: why can I do this:

FILE *fp = fopen("/proc/self/maps", "r");
char buf[513]; buf[512] = NULL;
while(fgets(buf, 512, fp) > NULL) printf("%s", buf);

but not this:

int fd = open("/proc/self/maps", O_RDONLY);
struct stat s;
fstat(fd, &s); // st_size = 0 -> why?
char *file = mmap(0, s.st_size /*or any fixed size*/, PROT_READ, MAP_PRIVATE, fd, 0); // gives EINVAL for st_size (because 0) and ENODEV for any fixed block
write(1, file, st_size);

I know that /proc files are not really files, but it seems to have some defined size and content for the FILE* version. Is it secretly generating it on-the-fly for read or something? What am I missing here?

EDIT: as I can clearly read() from them, is there any way to get the possible available bytes? or am I stuck to read until EOF?

Skep answered 16/2, 2021 at 13:37 Comment(3)
tldp.org/LDP/Linux-Filesystem-Hierarchy/html/proc.htmlBotelho
You probably can't mmap any of the fake files. Because then the kernel would have a hard time making that work because they aren't real files.Copaiba
Yes, the kernel generates the content on the fly for read. It's not secret. That's what a "not really file" isCopaiba
P
0

As already explained by others, /proc and /sys are pseudo-filesystems, consisting of data provided by the kernel, that does not really exist until it is read – the kernel generates the data then and there. Since the size varies, and really is unknown until the file is opened for reading, it is not provided to userspace at all.

It is not "unfortunate", however. The same situation occurs very often, for example with character devices (under /dev), pipes, FIFOs (named pipes), and sockets.

We can trivially write a helper function to read pseudofiles completely, using dynamic memory management. For example:

// SPDX-License-Identifier: CC0-1.0
//
#define  _POSIX_C_SOURCE  200809L
#define  _ATFILE_SOURCE
#define  _GNU_SOURCE
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <string.h>
#include <errno.h>

/* For example main() */
#include <stdio.h>

/* Return a directory handle for a specific relative directory.
   For absolute paths and paths relative to current directory, use dirfd==AT_FDCWD.
*/
int at_dir(const int dirfd, const char *dirpath)
{
    if (dirfd == -1 || !dirpath || !*dirpath) {
        errno = EINVAL;
        return -1;
    }
    return openat(dirfd, dirpath, O_DIRECTORY | O_PATH | O_CLOEXEC);
}

/* Read the (pseudofile) contents to a dynamically allocated buffer.
   For absolute paths and paths relative to current durectory, use dirfd==AT_FDCWD.
   You can safely initialize *dataptr=NULL,*sizeptr=0 for dynamic allocation,
   or reuse the buffer from a previous call or e.g. getline().
   Returns 0 with errno set if an error occurs.  If the file is empty, errno==0.
   In all cases, remember to free (*dataptr) after it is no longer needed.
*/
size_t read_pseudofile_at(const int dirfd, const char *path, char **dataptr, size_t *sizeptr)
{
    char   *data;
    size_t  size, have = 0;
    ssize_t n;
    int     desc;

    if (!path || !*path || !dataptr || !sizeptr) {
        errno = EINVAL;
        return 0;
    }

    /* Existing dynamic buffer, or a new buffer? */
    size = *sizeptr;
    if (!size)
        *dataptr = NULL;
    data = *dataptr;

    /* Open pseudofile. */
    desc = openat(dirfd, path, O_RDONLY | O_CLOEXEC | O_NOCTTY);
    if (desc == -1) {
        /* errno set by openat(). */
        return 0;
    }

    while (1) {

        /* Need to resize buffer? */
        if (have >= size) {
            /* For pseudofiles, linear size growth makes most sense. */
            size = (have | 4095) + 4097 - 32;
            data = realloc(data, size);
            if (!data) {
                close(desc);
                errno = ENOMEM;
                return 0;
            }
            *dataptr = data;
            *sizeptr = size;
        }

        n = read(desc, data + have, size - have);
        if (n > 0) {
            have += n;
        } else
        if (n == 0) {
            break;
        } else
        if (n == -1) {
            const int  saved_errno = errno;
            close(desc);
            errno = saved_errno;
            return 0;
        } else {
            close(desc);
            errno = EIO;
            return 0;
        }
    }

    if (close(desc) == -1) {
        /* errno set by close(). */
        return 0;
    }

    /* Append zeroes - we know size > have at this point. */
    if (have + 32 > size)
        memset(data + have, 0, 32);
    else
        memset(data + have, 0, size - have);

    errno = 0;
    return have;
}

int main(void)
{
    char   *data = NULL;
    size_t  size = 0;
    size_t  len;
    int     selfdir;

    selfdir = at_dir(AT_FDCWD, "/proc/self/");
    if (selfdir == -1) {
        fprintf(stderr, "/proc/self/ is not available: %s.\n", strerror(errno));
        exit(EXIT_FAILURE);
    }

    len = read_pseudofile_at(selfdir, "status", &data, &size);
    if (errno) {
        fprintf(stderr, "/proc/self/status: %s.\n", strerror(errno));
        exit(EXIT_FAILURE);
    }
    printf("/proc/self/status: %zu bytes\n%s\n", len, data);

    len = read_pseudofile_at(selfdir, "maps", &data, &size);
    if (errno) {
        fprintf(stderr, "/proc/self/maps: %s.\n", strerror(errno));
        exit(EXIT_FAILURE);
    }
    printf("/proc/self/maps: %zu bytes\n%s\n", len, data);

    close(selfdir);

    free(data); data = NULL; size = 0;

    return EXIT_SUCCESS;
}

The above example program opens a directory descriptor ("atfile handle") to /proc/self. (This way you do not need to concatenate strings to construct paths.)

It then reads the contents of /proc/self/status. If successful, it displays its size (in bytes) and its contents.

Next, it reads the contents of /proc/self/maps, reusing the previous buffer. If successful, it displays its size and contents as well.

Finally, the directory descriptor is closed as it is no longer needed, and the dynamically allocated buffer released.

Note that it is perfectly safe to do free(NULL), and also to discard the dynamic buffer (free(data); data=NULL; size=0;) between the read_pseudofile_at() calls.

Because pseudofiles are typically small, the read_pseudofile_at() uses a linear dynamic buffer growth policy. If there is no previous buffer, it starts with 8160 bytes, and grows it by 4096 bytes afterwards until sufficiently large. Feel free to replace it with whatever growth policy you prefer, this one is just an example, but works quite well in practice without wasting much memory.

Purapurblind answered 17/2, 2021 at 0:43 Comment(6)
Thanks, but having to reallocate the buffer was exactly what I meant when I said that I find it unfortunate. 'needlessly' reallocating kilobytes of data (and subsequently copying fractions of the file around) just personally pain me (which is why I wanted to mmap it in the first place).Skep
@Fl0wless: You need to recalibrate your pain sensors. You see, those "needless" reallocations and copies are exactly what makes the approach robust and reliable. As an example, consider the multiplication of two large matrices, say 256×256. You can optimize the calculation as much as you like, but creating a copy of one of the matrices so that the innermost sum product loop accesses elements consecutive in memory will give an efficiency boost you cannot get any other way. In other words, copies and reallocations are your friend, not your enemy.Differentia
I am well aware of that trick. But unnecessary copies and allocations are the root of all evil.Skep
@Fl0wless: It is not a trick. Algorithmic optimization beats micro-optimizations every single time. The true root of all evil is premature optimization, and optimizing at the wrong scale altogether. Besides, if you actually profile the above code, you'll see that the above code does very few copies; often none. You are just focusing on the completely wrong thing, assuming your intent is to write efficient and reliable C code. If not, well, good luck anyway.Differentia
copying row-major into column-major order is not an algorithmic optimization. It's a question about data acess patterns. (at least that's what I am talking about)Skep
@Fl0wless: It is not algorithmic optimization, but it is a pattern where copies yield superior performance. Copies are not evil, when they allow a robust, efficient implementation. Reading into a repeatedly dynamically reallocated buffer is a robust, efficient implementation: profile it; the reallocation cost vanishes into measurement noise. What you think of as unnecessary copies are not. Thus, the recommendation of recalibrating what you see as painful or evil.Differentia
F
2

They are created on the fly as you read them. Maybe this would help, it is a tutorial showing how a proc file can be implemented:

https://devarea.com/linux-kernel-development-creating-a-proc-file-and-interfacing-with-user-space/

tl;dr: you give it a name and read and write handlers, that's it. Proc files are meant to be very simple to implement from the kernel dev's point of view. They do not behave like full-featured files though.

As for the bonus question, there doesn't seem to be a way to indicate the size of the file, only EOF on reading.

Frimaire answered 16/2, 2021 at 13:54 Comment(1)
Pretty unfortunate ;(Skep
B
1

proc "files" are not really files, they are just streams that can be read/written from, but they contain no pyhsical data in memory you can map to.

https://tldp.org/LDP/Linux-Filesystem-Hierarchy/html/proc.html

Botelho answered 16/2, 2021 at 13:48 Comment(0)
P
0

As already explained by others, /proc and /sys are pseudo-filesystems, consisting of data provided by the kernel, that does not really exist until it is read – the kernel generates the data then and there. Since the size varies, and really is unknown until the file is opened for reading, it is not provided to userspace at all.

It is not "unfortunate", however. The same situation occurs very often, for example with character devices (under /dev), pipes, FIFOs (named pipes), and sockets.

We can trivially write a helper function to read pseudofiles completely, using dynamic memory management. For example:

// SPDX-License-Identifier: CC0-1.0
//
#define  _POSIX_C_SOURCE  200809L
#define  _ATFILE_SOURCE
#define  _GNU_SOURCE
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <string.h>
#include <errno.h>

/* For example main() */
#include <stdio.h>

/* Return a directory handle for a specific relative directory.
   For absolute paths and paths relative to current directory, use dirfd==AT_FDCWD.
*/
int at_dir(const int dirfd, const char *dirpath)
{
    if (dirfd == -1 || !dirpath || !*dirpath) {
        errno = EINVAL;
        return -1;
    }
    return openat(dirfd, dirpath, O_DIRECTORY | O_PATH | O_CLOEXEC);
}

/* Read the (pseudofile) contents to a dynamically allocated buffer.
   For absolute paths and paths relative to current durectory, use dirfd==AT_FDCWD.
   You can safely initialize *dataptr=NULL,*sizeptr=0 for dynamic allocation,
   or reuse the buffer from a previous call or e.g. getline().
   Returns 0 with errno set if an error occurs.  If the file is empty, errno==0.
   In all cases, remember to free (*dataptr) after it is no longer needed.
*/
size_t read_pseudofile_at(const int dirfd, const char *path, char **dataptr, size_t *sizeptr)
{
    char   *data;
    size_t  size, have = 0;
    ssize_t n;
    int     desc;

    if (!path || !*path || !dataptr || !sizeptr) {
        errno = EINVAL;
        return 0;
    }

    /* Existing dynamic buffer, or a new buffer? */
    size = *sizeptr;
    if (!size)
        *dataptr = NULL;
    data = *dataptr;

    /* Open pseudofile. */
    desc = openat(dirfd, path, O_RDONLY | O_CLOEXEC | O_NOCTTY);
    if (desc == -1) {
        /* errno set by openat(). */
        return 0;
    }

    while (1) {

        /* Need to resize buffer? */
        if (have >= size) {
            /* For pseudofiles, linear size growth makes most sense. */
            size = (have | 4095) + 4097 - 32;
            data = realloc(data, size);
            if (!data) {
                close(desc);
                errno = ENOMEM;
                return 0;
            }
            *dataptr = data;
            *sizeptr = size;
        }

        n = read(desc, data + have, size - have);
        if (n > 0) {
            have += n;
        } else
        if (n == 0) {
            break;
        } else
        if (n == -1) {
            const int  saved_errno = errno;
            close(desc);
            errno = saved_errno;
            return 0;
        } else {
            close(desc);
            errno = EIO;
            return 0;
        }
    }

    if (close(desc) == -1) {
        /* errno set by close(). */
        return 0;
    }

    /* Append zeroes - we know size > have at this point. */
    if (have + 32 > size)
        memset(data + have, 0, 32);
    else
        memset(data + have, 0, size - have);

    errno = 0;
    return have;
}

int main(void)
{
    char   *data = NULL;
    size_t  size = 0;
    size_t  len;
    int     selfdir;

    selfdir = at_dir(AT_FDCWD, "/proc/self/");
    if (selfdir == -1) {
        fprintf(stderr, "/proc/self/ is not available: %s.\n", strerror(errno));
        exit(EXIT_FAILURE);
    }

    len = read_pseudofile_at(selfdir, "status", &data, &size);
    if (errno) {
        fprintf(stderr, "/proc/self/status: %s.\n", strerror(errno));
        exit(EXIT_FAILURE);
    }
    printf("/proc/self/status: %zu bytes\n%s\n", len, data);

    len = read_pseudofile_at(selfdir, "maps", &data, &size);
    if (errno) {
        fprintf(stderr, "/proc/self/maps: %s.\n", strerror(errno));
        exit(EXIT_FAILURE);
    }
    printf("/proc/self/maps: %zu bytes\n%s\n", len, data);

    close(selfdir);

    free(data); data = NULL; size = 0;

    return EXIT_SUCCESS;
}

The above example program opens a directory descriptor ("atfile handle") to /proc/self. (This way you do not need to concatenate strings to construct paths.)

It then reads the contents of /proc/self/status. If successful, it displays its size (in bytes) and its contents.

Next, it reads the contents of /proc/self/maps, reusing the previous buffer. If successful, it displays its size and contents as well.

Finally, the directory descriptor is closed as it is no longer needed, and the dynamically allocated buffer released.

Note that it is perfectly safe to do free(NULL), and also to discard the dynamic buffer (free(data); data=NULL; size=0;) between the read_pseudofile_at() calls.

Because pseudofiles are typically small, the read_pseudofile_at() uses a linear dynamic buffer growth policy. If there is no previous buffer, it starts with 8160 bytes, and grows it by 4096 bytes afterwards until sufficiently large. Feel free to replace it with whatever growth policy you prefer, this one is just an example, but works quite well in practice without wasting much memory.

Purapurblind answered 17/2, 2021 at 0:43 Comment(6)
Thanks, but having to reallocate the buffer was exactly what I meant when I said that I find it unfortunate. 'needlessly' reallocating kilobytes of data (and subsequently copying fractions of the file around) just personally pain me (which is why I wanted to mmap it in the first place).Skep
@Fl0wless: You need to recalibrate your pain sensors. You see, those "needless" reallocations and copies are exactly what makes the approach robust and reliable. As an example, consider the multiplication of two large matrices, say 256×256. You can optimize the calculation as much as you like, but creating a copy of one of the matrices so that the innermost sum product loop accesses elements consecutive in memory will give an efficiency boost you cannot get any other way. In other words, copies and reallocations are your friend, not your enemy.Differentia
I am well aware of that trick. But unnecessary copies and allocations are the root of all evil.Skep
@Fl0wless: It is not a trick. Algorithmic optimization beats micro-optimizations every single time. The true root of all evil is premature optimization, and optimizing at the wrong scale altogether. Besides, if you actually profile the above code, you'll see that the above code does very few copies; often none. You are just focusing on the completely wrong thing, assuming your intent is to write efficient and reliable C code. If not, well, good luck anyway.Differentia
copying row-major into column-major order is not an algorithmic optimization. It's a question about data acess patterns. (at least that's what I am talking about)Skep
@Fl0wless: It is not algorithmic optimization, but it is a pattern where copies yield superior performance. Copies are not evil, when they allow a robust, efficient implementation. Reading into a repeatedly dynamically reallocated buffer is a robust, efficient implementation: profile it; the reallocation cost vanishes into measurement noise. What you think of as unnecessary copies are not. Thus, the recommendation of recalibrating what you see as painful or evil.Differentia

© 2022 - 2024 — McMap. All rights reserved.