Using strtok() in nested loops in C?

Question

I am trying to use strtok() in nested loops but this is not giving me desired results, possibly because they are using the same memory location. My code is of the form:-

char *token1 = strtok(Str1, "%");
while (token1 != NULL)
{
    char *token2 = strtok(Str2, "%");
    while (token2 != NULL)
    {
        //Do something
        token2 = strtok(NULL, "%");
    }
    // Do something more
    token1 = strtok(NULL, "%");
}

Ordzhonikidze

So, now you know why people don't use strtok() if they can avoid it. It is hard to use correctly. – Amuck

Answer 1

Yes, strtok(), indeed, uses some static memory to save its context between invocations. Use a reentrant version of strtok(), strtok_r() instead, or strtok_s() if you are using VS (identical to strtok_r()).

It has an additional context argument, and you can use different contexts in different loops.

char *tok, *saved;
for (tok = strtok_r(str, "%", &saved); tok; tok = strtok_r(NULL, "%", &saved))
{
    /* Do something with "tok" */
}

Northrup

In case you don't edit in the reason why strtok is behaving this way, here's some more information about strtok_r: mkssoftware.com/docs/man3/strtok_r.3.asp – Blackguardly

Could someone please explain how the above loop works? – Turnery

@MortalMan: that loop on its own can use strtok() or strtok_r() (or Microsoft strtok_s()) with minimal changes for the different calling convention of strtok(). It simply steps through the tokens in str. What's different about using strtok_r() is that you could have an inner loop also using strtok_r() and a different variable in place of saved and the inner loop would not interfere with the outer loop. That cannot be done with strtok(). – Amuck

Answer 2

strtok is using a static buffer. In your case you should use strtok_r. This function is using a buffer provided by the user.

Carioca

Answer 3

WayneAKing posted an alternative in the Microsoft Developer Center.

Citing him:

Go here

http://cpp.snippets.org/code/

and download this file

stptok.c Improved tokenizing function

You can also download the needed header files from the same site.

This is a modified version of strtok which places the parsed tokens (substrings) in a separate buffer. You should be able to modify it to accommodate your needs.

• Wayne

P.S. - Note that these files may be in *nix format with respect to end-of-lines. i.e. - 0x0A only and not 0x0D 0x0A

This is an alternative if you don't have the Microsoft libraries in your environment.

Collazo