Fread on Lion does not read when length > 2G
Asked Answered
G

3

3

Since Macosx Lion fread does not read file with length > 2G (int size, 2'147'483'648 bytes). It worked for years with macosx snow leopard.

I wrote a program to test it :

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char *argv[])
{
FILE   *fin = NULL, *fout = NULL;
char   *ptr = NULL;
size_t len;
fpos_t flen;

if (!(fin = fopen(argv[1], "rb")))
{
    printf("The input file: %s could not be opened\n", argv[1]);
    return -1;
}
if ((fout = fopen(argv[2], "rb")))
{
    printf("The output file %s already exist\n", argv[2]);
    fclose(fin);
    return -1;
}
if (!(fout = fopen(argv[2],"wb")))
{
    printf("Cannot write on output file %s\n", argv[2]);
    fclose(fin);
    return -1;
}

fseek(fin, 0, SEEK_END);
fgetpos(fin, &flen);
len = flen;
printf("Input file length : %zd\n", len);
fseek(fin, 0, SEEK_SET);

if (!(ptr = malloc(len))) 
{
    printf("Canot allocate %zd bytes\n", len);
    fclose(fin);
    fclose(fout);
    return -1;
}
if (fread(ptr, sizeof(char), len, fin) != len)
{
    printf("Cannot read file\n");
    fclose(fin);
    fclose(fout);
    free(ptr);
    return -1;
}
fclose(fin);
if (fwrite(ptr, sizeof(char), len, fout) != len) 
{
    printf("Cannot write file\n");
    fclose(fout);
    free(ptr);
    return -1;
}
free(ptr);
fclose(fout);

return 1;
}

just run :

  • ./pgm inputfile outputfile
  • openssl sha inputfile
  • openssl sha outputfile

There is no error. The length of the 2 files are the same. The two fingerprints are not the same. (The pointer is well allocated and write in the outputfile) Its only with fread, not fwrite.

i don't understand the problem.

I just see this program (i don't know if apple use this one on Lion) and r variable is defined as int. http://www.opensource.apple.com/source/Libc/Libc-186/stdio.subproj/fread.c

Thanks for answers

Gnomic answered 28/10, 2011 at 11:46 Comment(1)
Is OSX 32-bit or 64-bit? If it's 32-bit, there's no reason to expect to be able to read more than 2GB or even to be able to create an object that large in memory.Dilly
H
1

Sounds like you're not compiling in 64 bit mode. Look for a command line argument or an option to whatever compiler you're using. To make sure you're compiling in the right mode, printf("%d\n", sizeof(int)); and see if it shows you what you expected.

Halda answered 29/10, 2011 at 6:29 Comment(0)
S
1

Works fine for me on MacOS X Lion (10.7.2) with XCode 4.2. The executable is a 64-bit program. You should ensure yours is too.

$ make 2gb
/usr/bin/gcc -g -std=c99 -Wall -Wextra 2gb.c -o 2gb
2gb.c:5: warning: unused parameter ‘argc’
$ file 2gb
2gb: Mach-O 64-bit executable x86_64
$ dd if=/dev/zero of=input bs=1m count=3072
./2g3072+0 records in
3072+0 records out
3221225472 bytes transferred in 42.940363 secs (75016261 bytes/sec)
$ ls -l input
./2gb -rw-r--r--  1 jleffler  staff  3221225472 Oct 29 00:48 input
$ ./2gb input output
Input file length : 3221225472
$ openssl sha input
SHA(input)= c93bf6713a90e34554311f0a9e43cfd1f153475a
$ openssl sha output
SHA(output)= c93bf6713a90e34554311f0a9e43cfd1f153475a
$ ls -l input output
-rw-r--r--  1 jleffler  staff  3221225472 Oct 29 00:48 input
-rw-r--r--  1 jleffler  staff  3221225472 Oct 29 00:49 output
$ rm input output
$ /usr/bin/gcc --version
i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)
Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$

And, when forced into 32-bit compilation:

$ rm 2gb
$ make CC='/usr/bin/gcc -m32' 2gb
/usr/bin/gcc -m32 -g -std=c99 -Wall -Wextra 2gb.c -o 2gb
2gb.c:5: warning: unused parameter ‘argc’
$ dd if=/dev/zero of=input bs=1m count=3072
3072+0 records in
3072+0 records out
3221225472 bytes transferred in 38.326753 secs (84046397 bytes/sec)
$ ./2gb input output
Input file length : 0
$ ls -l input
-rw-r--r--  1 jleffler  staff  3221225472 Oct 29 00:57 input
$ 
Shammer answered 29/10, 2011 at 7:54 Comment(1)
/dev/zero was a bad choice for creating the test file here. Since very large malloc calls will pull in zero pages from the kernel, if fread was broken you'd be testing a file of zeros against a memory space of zeros.Icosahedron
G
0

printf("%d\n", sizeof(int)); gcc -Wall Typ.c -o Typ warning: format ‘%d’ expects type ‘int’, but argument 2 has type ‘long unsigned int’ warning: format ‘%d’ expects type ‘int’, but argument 2 has type ‘long --> 4

Thank you Jonathan.

i am on 10.7.2 and xcode 4.2

gcc --version i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.1.00)

i do gcc TestFile.c -o TestFile.o and even exactly as you did : /usr/bin/gcc -g -std=c99 -Wall -Wextra 2gb.c -o 2gb and i add -m64 after

and i have again 2 differents fingerprint.

xcode 3.6.2 was removed using sudo /Library/uninstall-devtools --mode=all and xcode 4.2 install after.

i run the pgm on another user newly created (cause of $path maybe) and its the same.

i don't understand.


Apple team answer after reporting this bug

Hello Stéphane,

This is a follow up to Bug ID# 10376104. After further investigation it has been determined that this is a known issue, which is currently being investigated by engineering. This issue has been filed in our bug database under the original Bug ID# 6434977. The original bug number being used to track this duplicate issue can be found in the Related Problem section of your bug report's Problem Detail view.

Thank you for submitting this bug report. We truly appreciate your assistance in helping us discover and isolate bugs.

Best Regards,

Developer Bug Reporting Team Apple Worldwide Developer Relations

Gnomic answered 29/10, 2011 at 10:0 Comment(3)
I created a new partition, install Lion, Update Lion (10.7.2), Install Xcode (4.2). and .. its the same problem again. (incredible)...so i'll send a bug to Apple .. my machine is macpro 6 cores Sept 2010.Gnomic
Apple team answer after reporting this bug- see text above (more than 556 char)Gnomic
Its Fixed !! Yes they did it ! It just takes more than 2 years but this problem is solved now.(on lastest Mavericks)Gnomic

© 2022 - 2024 — McMap. All rights reserved.