strtok on 64 bit machines

Question

The following code works differently on 64 bit and on 32 bit which is causing me trouble to port my code.

char * tmp = "How are you?";
printf("size of char * = %ld and size of strtok return val = %ld \n",sizeof(char *),sizeof(strtok(tmp," ")));

Following is the output:

32 bit: 
size of char * = 4 and size of strtok return val = 4 

64 bit:

size of char * = 8 and size of strtok return val = 4

The man page of strtok says:

   #include <string.h>

   char *strtok(char *str, const char *delim);

RETURN VALUE
       The strtok() and strtok_r() functions return a pointer to the next token, or NULL if there are no more tokens.

The char* on a 64 bit machine is supposed to be 8 bytes as printed. So why is strtok returning a 4 bytes char pointer on a 64 bit machine??

Thanks

Peter

Which compiler are you using to get these results? – Medievalist

Have you forgotten to include <string.h>? Then the compiler might be in traditional mood and assume return type int for functions it doesn't know. – Bootlace

Confirmed on gcc-4.5.real (Ubuntu/Linaro 4.5.2-8ubuntu4) 4.5.2. Crazy. – Vigilante

@Daniel: excellent! that's it exactly. – Vigilante

@Daniel: I have included string.h – Peter

gcc versions: 4.6.1 on 64 bit and 4.6.2 on 32 bit machine – Peter

Can't reproduce, gcc-4.5.1 says 8 for both here with <string.h>, 8 and 4 without. – Bootlace

Answer 1

You forgot to #include <string.h>.

This is causing the default return type of int to be inferred by the compiler. By #including the right header file, the correct prototype is pulled into scope.

This solves the problem for me on gcc. If it doesn't for you, what compiler are you using?

Gadgeteer

Your compiler should at least warn you if you try to call an undeclared function. If it doesn't, you need to find out how to increase its warning level. (The implicit int rule was removed from the language by the 1999 ISO C standard.) And in a comment, the OP says he included string.h; if so, that's not the problem. – Torosian

Sorry for the trouble. I had a copy of string.h in my file hierarchy. I deleted the string.h and let gcc take the default path. It works now. @Daniel,Clark: Thanks for the hint !! – Peter

@ntalli: The include directive needs to be #include <string.h>, not #include "string.h"; that should avoid picking up a string.h header from your own file hierarchy. And you definitely should fix the printf format string, as I described in my answer. Compile with gcc -std=c99 -pedantic -Wall -Wextra to get better warnings. (Replace -std=c99 with -ansi if you want C89/C90 code rather than C99.) – Torosian

I will definitely remember to add -std=c99 or -ansi from now on. Although I tested the first comment here by Keith. I completely removed #include<string.h> from the code and recompiled it. It compiles!!! The compiler does not report undefined, undeclared function for strtok. Any idea why this might be happening? – Peter

Answer 2

Calling strtok(tmp, " ") would cause undefined behavior, since it would attempt to modify the string literal that tmp points to -- but since the operand of sizeof is not evaluated (with one exception that doesn't apply here), that's not an issue.

The real problem is that you're trying to print size_t values with a "%ld" format, which requires an unsigned long argument.

If your implementation supports it, the correct format for a size_t argument is "%zu" (added in C99):

printf("size of char * = %zu and size of strtok return val = %zu\n",
       sizeof(char *), sizeof(strtok(tmp," ")));

Otherwise, explicitly convert the arguments to the appropriate size. I'd use "%lu", since size_t is an unsigned type.

printf("size of char * = %lu and size of strtok return val = %lu\n",
       (unsigned long)sizeof(char *), (unsigned long)sizeof(strtok(tmp," ")));

Here's a complete self-contained program that should produce the expected results on any C89 or later implementation:

#include <stdio.h>
#include <string.h>
int main(void) {
    char * tmp = "How are you?";
    printf("size of char * = %lu and size of strtok return val = %lu\n",
           (unsigned long)sizeof(char *),
           (unsigned long)sizeof(strtok(tmp," ")));
    return 0;
}

EDIT : The OP's comment on the other answer indicates that the string.h header was the problem; apparently he had

#include "string.h"

rather than

#include <string.h>

I'm going to leave this answer here because it describes another problem that needs to be fixed in the OP's code, though not the one that caused the observed symptom. and the compiler picked up the wrong string.h header file.

Torosian

I'd love to know that exception to sizeof() that doesn't apply here. :) – Vigilante

I can't bring myself to believe that an incorrect format specifier would cause a value of 8 to be printed as 4 when both, expected an passed type are 64-bit integers. – Bootlace

@Vigilante Variable length arrays? – Bootlace

@sarnold: The operand to sizeof is evaluated if its type is a VLA (variable length array). For example, sizeof (int[func()]) needs to call func() to determine the size of the type. – Torosian

@DanielFischer: The printf format is certainly a problem that should be fixed. I see your point that it's unlikely to cause the specific symptom the OP mentioned -- but the OP said in a comment that he did include string.h (he didn't say <string.h>, but ...). I'd like to see what result he gets if he adds #include <string.h> to his program. For that matter, I'd like to see his whole program rather than the fragment he's shown us. – Torosian

Oh yes, should definitely be fixed. UB is never good to have. According to the comment on the other answer, OP had a stray string.h that was picked up instead of the system header, which caused it. – Bootlace

@DanielFischer: But omitting #include <string.h> does explain the reported symptoms. – Torosian

I had a lot of files in my project and had modified the LD_LIBRARY_PATH to include my project hierarchy. By mistake I overlooked the old stray files when I copied the project to a 64 bit machine. So basically, I did do a #include<string.h> which it took from the LD_LIBRARY_PATH rather than from the gcc default. Maybe this explains the situation better. The code snippet here is not a part of my project. This was purely to debug what was going wrong and I was thrilled when I found the return size of strtok to be 4 bytes !! – Peter

$LD_LIBRARY_PATH affects the dynamic loader/linker; it has no effect on the compiler's handling of #include directives. Could you have set $CPATH or $C_INCLUDE_PATH? – Torosian

I am not sure. Last week I had tweaked my libraries and include paths to run a benchmark. To confirm I just tried a gcc -v and I got #include <...> search starts here: /usr/lib/gcc/x86_64-linux-gnu/4.6.1/include /usr/local/include /usr/lib/gcc/x86_64-linux-gnu/4.6.1/include-fixed /usr/include/x86_64-linux-gnu /usr/include /home/mtprojectpath I am not sure which env variable is updated here. I will find and let you know. – Peter