Quick-fixing 32-bit (2GB limited) fseek/ftell on freebsd 7
Asked Answered
T

1

6

I have old 32-bit C/C++ program on FreeBSD, which is used remotely by hundreds of users, and author of which will not fix it. It was written in unsafe way, all file offset are stored internally as unsigned 32-bit offsets, and ftell/fseek functions where used. In FreeBSD 7 (the host platform for software), it means that ftell and fseek uses 32-bit signed long:

 int fseek(FILE *stream, long offset, int whence);

 long ftell(FILE *stream);

I need to do quick fix of the program, because some internal data files suddenly hit 2^31 file size (2 147 483 7yy bytes) after 13 years of collecting data, and internal fseek/ftell assert fails now for any request.

In FreeBSD7 world there is fseeko/ftello hack for 2GB+ files.

 int
 fseeko(FILE *stream, off_t offset, int whence);

 off_t
 ftello(FILE *stream);

The off_t type here is not well-defined; all I know now, that it has 8-byte size and looks like long long OR unsigned long long (I don't know which one).

Is it enough (to work with up to 4 GB files) and safe to search-and-replace all ftell to ftello, and all fseek to fseeko (sed -i 's/ftell/ftello', same for seek), if possible usages of them are:

 unsigned long offset1,offset2; //32bit
 offset1 = (compute + it) * in + some - arithmetic;
 fseek(file, 0, SEEK_END);
 fseek(file, 4, SEEK_END); // or other small int constant

 offset2 = ftell(file);
 fseek(file, offset1, SEEK_SET);  // No usage of SEEK_CUR

and combinations of such calls.

What is the signedness of off_t? It is safe to assign 64-bit off_t into unsigned 32-bit offset? Will it work for bytes in range from 2 GB up to 4 GB?

Which functions may be used for working with offset besides ftell/fseek?

Trierarch answered 29/6, 2014 at 1:18 Comment(8)
So... you have all the source code? If so, rather than a superficial fix, why not do a through update? A patch (as you suggest in the question) will be problematic; and may potentially cause data corruption.Durham
I need quick fix (users will wake up in several hours...), and I don't understand all the code (20 KLoC). Which code may corrupt data? I have lot of greps here, and I see no other usages of fseek/ftell. Programs do lot of offset calculations, so I can't switch to fgetpos/fsetpos. Some file offsets are also stored inside files, and there are 20+ interconnected files with data. Migration to 64-bit offset will break file format and will need file conversion.Trierarch
Understood. (been there, done that...) Just cautioning... make sure the existing 2GB of data is preserved so that it can be restored if needed. How is the code licensed? Is this code 'open-source'?Durham
It is not open-source, but stolen and reuploaded to github. All data must be saved, and it will be accessed by users (up to 5-10 req/s sometimes); so I can't split the database in old 2GB and new one without very major rewrite of software. Already have 2 copies from previous day, and two from this day (after reaching 2 GB); all are saved on healthy HW RAID6.Trierarch
Can you give the URL for the code on github?Durham
The problem here isn't just all the ftell/fseek calls, but any vars that might be used for those calculations as well.Colombes
Joe, as I see, unsigned 32-bit int is used, and sometimes offsets are calculated. How can I check, are they calculated correctly? Can I replace all fseeks with some my_fseek macro to check correctnes of calculations?Trierarch
osgx, see the addendum in my answer. Yes, you can replace all fseek()/ftell() calls with your own functions. With GCC, inline functions are as fast as macros. As to offset calculations, use helper functions to do the arithmetic.Matchbook
M
12

FreeBSD fseeko() and ftello() is documented as POSIX.1-2001 compatible, which means off_t is a signed integer type.

On FreeBSD 7, you can safely do:

off_t          actual_offset;
unsigned long  stored_offset;

if (actual_offset >= (off_t)0 && actual_offset < (off_t)4294967296.0)
    stored_offset = (unsigned long)actual_offset;
else
    some_fatal_error("Unsupportable file offset!");

(On LP64 architectures, the above would be silly, as off_t and long would both be 64-bit signed integers. It would be safe even then; just silly, since all possible file offsets could be supported.)

The thing that people do get bitten by often with this, is that the offset calculations must be done using off_t. That is, it is not enough to cast the result to off_t, you must cast the values used in the arithmetic to off_t. (Technically, you only need to make sure each arithmetic operation is done at off_t precision, but I find it easier to remember the rules if I just punt and cast all the operands.) For example:

off_t          offset;
unsigned long  some, values, used;

offset = (off_t)some * (off_t)value + (off_t)used;
fseeko(file, offset, SEEK_SET);

Usually the offset calculations are used to find a field in a specific record; the arithmetic tends to stay the same. I truly recommend you move the seek operations to a helper function, if possible:

int fseek_to(FILE *const file,
             const unsigned long some,
             const unsigned long values,
             const unsigned long used)
{
    const off_t  offset = (off_t)some * (off_t)value + (off_t)used;
    if (offset < (off_t)0 || offset >= (off_t)4294967296.0)
        fatal_error("Offset exceeds 4GB; I must abort!");
    return fseeko(file, offset, SEEK_SET);
}

Now, if you happen to be in a lucky position where you know all your offsets are aligned (to some integer, say 4), you can give yourself a couple of years of more time to rewrite the application, by using an extension of the above:

#define BIG_N 4

int fseek_to(FILE *const file,
             const unsigned long some,
             const unsigned long values,
             const unsigned long used)
{
    const off_t  offset = (off_t)some * (off_t)value + (off_t)used;
    if (offset < (off_t)0)
        fatal_error("Offset is negative; I must abort!");
    if (offset >= (off_t)(BIG_N * 2147483648.0))
        fatal_error("Offset is too large; I must abort!");
    if ((offset % BIG_N) && (offset >= (off_t)2147483648.0))
        fatal_error("Offset is not a multiple of BIG_N; I must abort!");
    return fseeko(file, offset, SEEK_SET);
}

int fseek_big(FILE *const file, const unsigned long position)
{
    off_t  offset;
    if (position >= 2147483648UL)
        offset = (off_t)2147483648UL
               + (off_t)BIG_N * (off_t)(position - 2147483648UL);
    else
        offset = (off_t)position;
    return fseeko(file, offset, SEEK_SET);
}

unsigned long ftell_big(FILE *const file)
{
    off_t  offset;
    offset = ftello(file);
    if (offset < (off_t)0)
        fatal_error("Offset is negative; I must abort!");
    if (offset < (off_t)2147483648UL)
        return (unsigned long)offset;
    if (offset % BIG_N)
        fatal_error("Offset is not a multiple of BIG_N; I must abort!");
    if (offset >= (off_t)(BIG_N * 2147483648.0))
        fatal_error("Offset is too large; I must abort!");
    return (unsigned long)2147483648UL
         + (unsigned long)((offset - (off_t)2147483648UL) / (off_t)BIG_N);
}

The logic is simple: If offset is less than 231, it is used as-is. Otherwise, it is represented by value 231 + BIG_N × (offset - 231). The only requirement is that offset 231 and above are always multiples of BIG_N.

Obviously, you them must use only the above three functions -- plus whatever variants of fseek_to() you need, as long as they do the same checks, just use different parameters and formula for the offset calculation --, you can support file sizes of up to 2147483648 + BIG_N × 2147483647. For BIG_N==4, that is 10 GiB (less 4 bytes; 10,737,418,236 bytes to be exact).

Questions?


Edited to clarify:

Start with replacing your fseek(file, position, SEEK_SET) with calls to fseek_pos(file, position),

static inline void fseek_pos(FILE *const file, const unsigned long position)
{
    if (fseeko(file, (off_t)position, SEEK_SET))
        fatal_error("Cannot set file position!");
}

and fseek(file, position, SEEK_END) with calls to fseek_end(file, position) (for symmetry -- I'm assuming the position for this one is usually a literal integer constant),

static inline void fseek_end(FILE *const file, const off_t relative)
{
    if (fseeko(file, relative, SEEK_END))
        fatal_error("Cannot set file position!");
}

and finally, ftell(file) with calls to ftell_pos(file):

static inline unsigned long ftell_pos(FILE *const file)
{
    off_t position;
    position = ftello(file);
    if (position == (off_t)-1)
        fatal_error("Lost file position!");
    if (position < (off_t)0 || position >= (off_t)4294967296.0)
        fatal_error("File position outside the 4GB range!");
    return (unsigned long)position;
}

Since on your architecture and OS unsigned long is a 32-bit unsigned integer type and off_t is a 64-bit signed integer type, this gives you the full 4GB range.

For the offset calculations, define one or more functions similar to

static inline void fseek_to(FILE *const file, const off_t term1,
                                              const off_t term2,
                                              const off_t term3)
{
    const off_t position = term1 * term2 + term3;

    if (position < (off_t)0 || position >= (off_t)4294967296.0)
        fatal_error("File position outside the 4GB range!");
    if (fseeko(file, position, SEEK_SET))
        fatal_error("Cannot set file position!");
}

For each offset calculation algorithm, define one fseek_to variant. Name the parameters so that the arithmetic makes sense. Make the parameters const off_t, as above, so you don't need extra casts in the arithmetic. Only the parameters and the const off_t position = line defining the calculation algorithm vary between the variant functions.

Questions?

Matchbook answered 2/7, 2014 at 2:26 Comment(9)
+100 , nearly flawless victory. Was about to post similar but you beat me to it. For my part, I looked at the FXR (the source code is the only truth), so found that off_t is typedef'ed from __off_t in /sys/types.h, line 219. __off_t, in turn, is typedef'ed from __int64_t in /sys/_types.h, line 53.Transpacific
@IwillnotexistIdonotexist: You are right, I should have checked the sources. I was lazy, and since my Google-Fu didn't hit on FXR -- I had forgotten it even exists! --, but did find the man page, I just went with that.. Although off_t has to be signed for negative seeks from SEEK_CUR or SEEK_SET to make any sense (wrt. C standards' rules). I was mostly worried about failing to do the arithmetic correctly using off_t; that was my main point in my answer.Matchbook
What about ftell/ftello? Internally all offset variables are unsigned 32-bit. Some of them are computed and some are stored/loaded in database files. Thank you for code samples!Trierarch
@osgx: First snippet applies. Given unsigned long store_offset; off_t actual_offset; do actual_offset = ftello(file); and if (actual_offset >= (off_t)0 && actual_offset < (off_t)4294967296.0) store_offset = (unsigned long)actual_offset; but otherwise you're out of the 4GB range. I take it the offsets are not aligned in any way, then?Matchbook
Why did you use floating constants like 2147483648.0?Trierarch
@osgx: I did not, actually. (off_t)2147483648.0 and (off_t)4294967296.0 are literal constants of the off_t type. The compiler will do the conversion at compile time. I use such when I don't want to worry about what kind of suffix (none, L, or LL) the compiler requires for large integer constants, and I know the constant can be represented exactly both as a double and as a off_t (or whatever the target type is). Is there a reason you're not accepting my answer?Matchbook
I will accept answer after updating the program according to your advises. Now it is running with find-and-replace of all fseek/ftell calls to fseeko/ftello (there was internal macro for fseek/ftell already, so I redefined it).Trierarch
Why do you use floating point literals in your example? I mean instead of (off_t) 4294967296.0 you can simply write: (off_t) 4294967296lluRole
@maxschlepzig: Look at my previous comment. Here, the question refers to an old C/C++ program. The unsigned long long type was standardized in C99, so if the OP needs weird compiler settings (or wants it to be portable to pre-C99 compilers), I wanted to avoid any issues. On all FreeBSD architectures (and basically all current hardware architectures), double is IEEE 754 binary64 (or binary32), so 4294967296.0 is exact. So, in short, (off_t)4294967296.0 and (off_t)4294967296LLU are both the exact same value, of off_t type; I just thought the former better for this particular case.Matchbook

© 2022 - 2024 — McMap. All rights reserved.