Where to find the complete definition of off_t type?

Question

I am sending file from client to server using TCP. To mark the end of the file I like to send file size before the actual data. So I use stat system call to find the size of the file. This is of type off_t. I like to know how many bytes it occupies so that I can read it properly on the server side. It is defined in the <sys/types.h>. But I do not understand the definition. It just defines __off_t or _off64_t to be off_t. Where to look for __off_t? Also is it convention that __ is prefixed for most of the things in header files and scares me when I read header files to understand better. How to read a header file better?

#ifndef __off_t_defined
# ifndef __USE_FILE_OFFSET64
typedef __off_t off_t;
# else
typedef __off64_t off_t;
# endif
# define __off_t_defined
#endif 

Klute

Anything starting with __ is reserved for use by the implementation (unless the standard defines a meaning for it, as in __func__ or __FILE__). The level of indirection lets the implementation define its own type __off_t without interfering with anything you can legitimately do. The platform-specific bits of the headers can then be better hidden (so a single copy of the source code can handle 32-bit and 64-bit compilations on a single machine, for example). Reading standard headers is a major chore because there are so many interlinked definitions. – Godthaab

Answer 1

Since this answer still gets voted up, I want to point out that you should almost never need to look in the header files. If you want to write reliable code, you're much better served by looking in the standard. So, the answer to "where can I find the complete definition of off_t" is "in a standard, rather than a header file". Following the standard means that your code will work today and tomorrow, on any machine.

In this case, off_t isn't defined by the C standard. It's part of the POSIX standard, which you can browse here.

Unfortunately, off_t isn't very rigorously defined. All I could find to define it is on the page on sys/types.h:

blkcnt_t and off_t shall be signed integer types.

This means that you can't be sure how big it is. If you're using GNU C, you can use the instructions in the answer below to ensure that it's 64 bits. Or better, you can convert to a standards defined size before putting it on the wire. This is how projects like Google's Protocol Buffers work (although that is a C++ project).

---

For completeness here's the answer to "which header file defines off_t?":

On my machine (and most machines using glibc) you'll find the definition in bits/types.h (as a comment says at the top, never directly include this file), but it's obscured a bit in a bunch of macros. An alternative to trying to unravel them is to look at the preprocessor output:

#include <stdio.h>
#include <sys/types.h>

int main(void) {
  off_t blah;

  return 0;
}

And then:

$ gcc -E sizes.c  | grep __off_t
typedef long int __off_t;
....

However, if you want to know the size of something, you can always use the sizeof() operator.

Edit: Just saw the part of your question about the __. This answer has a good discussion. The key point is that names starting with __ are reserved for the implementation (so you shouldn't start your own definitions with __).

Kreg

I use size of on off_t. On my machine(client) it is 4 bytes. So maximum file size I can represent is 2^32 bytes. Does that have to match with off_t size on my server in this particular case. It should not I believe. – Klute

It's true that off_t can be different sizes on different machines (or compilers). Note that in GCC you can use -D_FILE_OFFSET_BITS=64 to get 8 byte off_t and size_t definitions. – Kreg

Ok.So what should the server look for first 4 or first 8 bytes in order to get the length of the file. – Klute

Ok I got it. I should use larger value like long long which guaranteed to be same on both the machine then up cast to it to send it to the server. – Klute

long long isn't guaranteed to be the same size on two machines - the exact size isn't standardised. – Kreg

However, if you use gcc and -D_FILE_OFFSET_BITS=64 when compiling both client and server, then you know that off_t is 8 bytes on both client and server. – Kreg

@FourOfAKind, off_t is signed so for a 4 byte int, it is +/- 2gig, not 0...4gig – Helga

“shall be signed integer types”. If they represent the size of files, then why use “signed”? – Mark

Using the search box of the linked website doesn't return any results for off_t. But it is there - the direct URL is pubs.opengroup.org/onlinepubs/9699919799/basedefs/… – Komsomol

Answer 2

As the "GNU C Library Reference Manual" says

off_t
    This is a signed integer type used to represent file sizes. 
    In the GNU C Library, this type is no narrower than int.
    If the source is compiled with _FILE_OFFSET_BITS == 64 this 
    type is transparently replaced by off64_t.

and

off64_t
    This type is used similar to off_t. The difference is that 
    even on 32 bit machines, where the off_t type would have 32 bits,
    off64_t has 64 bits and so is able to address files up to 2^63 bytes
    in length. When compiling with _FILE_OFFSET_BITS == 64 this type 
    is available under the name off_t.

Thus if you want reliable way of representing file size between client and server, you can:

1. Use off64_t type and stat64() function accordingly (as it fills structure stat64, which contains off64_t type itself). Type off64_t guaranties the same size on 32 and 64 bit machines.
2. As was mentioned before compile your code with -D_FILE_OFFSET_BITS == 64 and use usual off_t and stat().
3. Convert off_t to type int64_t with fixed size (C99 standard). Note: (my book 'C in a Nutshell' says that it is C99 standard, but optional in implementation). The newest C11 standard says:

7.20.1.1 Exact-width integer types

    1 The typedef name intN_t designates a signed integer type with width N ,
    no padding bits, and a two’s complement representation. Thus, int8_t 
    denotes such a signed integer type with a width of exactly 8 bits.
    without mentioning.

And about implementation:

7.20 Integer types <stdint.h>

    ... An implementation shall provide those types described as ‘‘required’’,
    but need not provide any of the others (described as ‘‘optional’’).
    ...
    The following types are required:
    int_least8_t  uint_least8_t
    int_least16_t uint_least16_t
    int_least32_t uint_least32_t
    int_least64_t uint_least64_t
    All other types of this form are optional.

Thus, in general, C standard can't guarantee types with fixed sizes. But most compilers (including gcc) support this feature.

Rookie

“This is a signed integer type used to represent file sizes”. If they represent the size of files, then why use “signed”? – Mark

(1) Because it's convenient to return negative values to indicate errors. (2) Because the sentence should say "This is a signed integer type used to represent file offsets", i.e., seeking forwards or backwards. – Geryon

Answer 3

If you are having trouble tracing the definitions, you can use the preprocessed output of the compiler which will tell you all you need to know. E.g.

$ cat test.c
#include <stdio.h>
$ cc -E test.c | grep off_t
typedef long int __off_t;
typedef __off64_t __loff_t;
  __off_t __pos;
  __off_t _old_offset;
typedef __off_t off_t;
extern int fseeko (FILE *__stream, __off_t __off, int __whence);
extern __off_t ftello (FILE *__stream) ;

If you look at the complete output you can even see the exact header file location and line number where it was defined:

# 132 "/usr/include/bits/types.h" 2 3 4


typedef unsigned long int __dev_t;
typedef unsigned int __uid_t;
typedef unsigned int __gid_t;
typedef unsigned long int __ino_t;
typedef unsigned long int __ino64_t;
typedef unsigned int __mode_t;
typedef unsigned long int __nlink_t;
typedef long int __off_t;
typedef long int __off64_t;

...

# 91 "/usr/include/stdio.h" 3 4
typedef __off_t off_t;

Kleon

Answer 4

If you are writing portable code, the answer is "you can't tell", the good news is that you don't need to. Your protocol should involve writing the size as (eg) "8 octets, big-endian format" (Ideally with a check that the actual size fits in 8 octets.)

Ampliate