strtok() - Why you have to pass the NULL pointer in order to get the next token in the string?

Question

This is the explanation of strtok().

#include <string.h>
char* strtok( char* s1, 
              const char* s2 );*

The first call to strtok() returns a pointer to the first token in the string pointed to by s1. Subsequent calls to strtok() must pass a NULL pointer as the first argument, in order to get the next token in the string.

But I don't know, why you have to pass the NULL pointer in order to get the next token in the string. I searched about 15 minutes, but didn't find an explanation in the internet.

Wein

If code did not pass NULL, what should code pass to continue tokenizing the same string? – Simaroubaceous

I guess it's just so you can opt out of tokenizing the whole string at once while not keeping track of where to start the next token yourself. If this was designed in C++ right now, it would most likely be different. – Involute

You did not see cplusplus.com/reference/cstring/strtok in your search? – Genethlialogy

Becaus that's the way it was designed 40 years ago. Poorly. Single-threaded and single-usage (not nestable), instead of providing a context object. – Demavend

The word I was searching for was 'non-re entrant'. – Demavend

Answer 1

strtok() keeps some data inside of itself by using static variables. This way, strtok() can continue searching from the point it left off at during the previous call. To signal strtok() that you want to keep searching the same string, you pass a NULL pointer as its first argument. strtok() checks whether the first argument is NULL and if it is, it uses its currently stored data. If the first parameter is not null, it is treated as a new search and all internal data is reset.

Maybe the best thing you can do is to search for an actual implementation of the strtok() function. I've found one small enough to post it here, so you get an idea of how to handle this NULL parameter:

/* Copyright (c) Microsoft Corporation. All rights reserved. */

#include <string.h>

/* ISO/IEC 9899 7.11.5.8 strtok. DEPRECATED.
 * Split string into tokens, and return one at a time while retaining state
 * internally.
 *
 * WARNING: Only one set of state is held and this means that the
 * WARNING: function is not thread-safe nor safe for multiple uses within
 * WARNING: one thread.
 *
 * NOTE: No library may call this function.
 */

char * __cdecl strtok(char *s1, const char *delimit)
{
    static char *lastToken = NULL; /* UNSAFE SHARED STATE! */
    char *tmp;

    /* Skip leading delimiters if new string. */
    if ( s1 == NULL ) {
        s1 = lastToken;
        if (s1 == NULL)         /* End of story? */
            return NULL;
    } else {
        s1 += strspn(s1, delimit);
    }

    /* Find end of segment */
    tmp = strpbrk(s1, delimit);
    if (tmp) {
        /* Found another delimiter, split string and save state. */
        *tmp = '\0';
        lastToken = tmp + 1;
    } else {
        /* Last segment, remember that. */
        lastToken = NULL;
    }

    return s1;
}

Duroc

* WARNING: Only one set of state is held and this means that the * WARNING: function is not thread-safe nor safe for multiple uses within * WARNING: one thread. What does this mean? – Ecumenism

@MinhTran It means this function should not be used in multi thread call because static variable maybe changed by other thread. See /* UNSAFE SHARED STATE! */ – Swaggering

Answer 2

If you pass a non-NULL value, you are asking it to start tokenizing a different string.

If you pass a NULL value, you are asking to continue tokenizing the same thing as before.

Abessive

But, how does strtok know, that it is the same string? Now you pass NULL, but that is not the address of the string. – Wein

@ezph1, It handles only one string at a time and keeps internal state, which causes all sorts of problems. – Involute