Nested strtok function problem in C [duplicate]

Question

I have a string like this:

a;b;c;d;e
f;g;h;i;j
1;2;3;4;5

and i want to parse it element by element. I used nested strtok function but it just splits first line and makes null the token pointer. How can i overcome this? Here is the code:

token = strtok(str, "\n");

while(token != NULL && *token != EOF)
{
    char a[128], b[128];
    strcpy(a,token);
    strcpy(b,a);
    printf("a:%s\n",a);
    char *token2 = strtok(a,";");
    while(token2 != NULL)
    {
        printf("token2 %s\n",token2);
        token2 = strtok(NULL,";");
    }
    strcpy(token,b);
    token = strtok(NULL, "\n");
    if(token == NULL)
    {
        printf("its null");
    }
}

Output:

token 2 a
token 2 b
token 2 c
token 2 d
token 2 e

Mikimikihisa

Are you sure that you've read all lines? – Downes

Yes i am sure i read them in str. – Mikimikihisa

Answer 1

You cannot do that with strtok(); use strtok_r() from POSIX or strtok_s() from Microsoft if they are available, or rethink your design.

char *strtok_r(char *restrict s, const char *restrict sep,
               char **restrict lasts);
char *strtok_s(char *strToken, const char *strDelimit, char **context); 

These two functions are interchangeable.

Note that a variant strtok_s() is specified in an optional part of C11 (Annex K in ISO/IEC 9899:2011). However, few suppliers other than Microsoft have implemented the interfaces in that section of the standard. The version of strtok_s() specified in Annex K has a different interface from Microsoft's strtok_s() — similar problems bedevil a number of the other functions specified in Annex K.

With strtok_r()

#include <string.h>
#include <stdio.h>

int main(void)
{
    char str[] = "a;b;c;d;e\nf;g;h;i;j\n1;2;3;4;5\n";
    char *end_str;
    char *token = strtok_r(str, "\n", &end_str);

    while (token != NULL)
    {
        char *end_token;
        printf("a = %s\n", token);
        char *token2 = strtok_r(token, ";", &end_token);
        while (token2 != NULL)
        {
            printf("b = %s\n", token2);
            token2 = strtok_r(NULL, ";", &end_token);
        }
        token = strtok_r(NULL, "\n", &end_str);
    }

    return 0;
}

Results

a = a;b;c;d;e
b = a
b = b
b = c
b = d
b = e
a = f;g;h;i;j
b = f
b = g
b = h
b = i
b = j
a = 1;2;3;4;5
b = 1
b = 2
b = 3
b = 4
b = 5

Without strtok_r()

This works in context - provided that the data ends with a newline.

#include <string.h>
#include <stdio.h>

int main(void)
{
    char data[] = "a;b;c;d;e\nf;g;h;i;j\n1;2;3;4;5\n";
    char *string = data;
    char *token  = strchr(string, '\n');

    while (token != NULL)
    {
        /* String to scan is in string..token */
        *token++ = '\0';
        printf("a = %s\n", string);
        char *token2 = strtok(string, ";");
        while (token2 != NULL)
        {
            printf("b = %s\n", token2);
            token2 = strtok(NULL, ";");
        }
        string = token;
        token = strchr(string, '\n');
    }

    return 0;
}

Output

a = a;b;c;d;e
b = a
b = b
b = c
b = d
b = e
a = f;g;h;i;j
b = f
b = g
b = h
b = i
b = j
a = 1;2;3;4;5
b = 1
b = 2
b = 3
b = 4
b = 5

Braun

It is not available for me. Any other solution? – Mikimikihisa

@mausmust: Then you're out of luck -- you'll have to code your own solution to tokenize your strings since strtok only works on one string at a time. – Stebbins

Answer 2

strtok_r is the best and safest solution, but there is also a way to do it with strtok:

#include <string.h>
#include <stdio.h>

int main ()
{
  char str[] = "a;b;c;d;e\nf;g;h;i;j\n1;2;3;4;5\n";
  char *line;
  char *token;
  char buf[256];

  for (line = strtok (str, "\n"); line != NULL;
       line = strtok (line + strlen (line) + 1, "\n"))
    {
      strncpy (buf, line, sizeof (buf));
      printf ("Line: %s\n", buf);
      for (token = strtok (buf, ";"); token != NULL;
       token = strtok (token + strlen (token) + 1, ";"))
    {
      printf ("\tToken: %s\n", token);
    }
    }

  return 0;
}

Output:

Line: a;b;c;d;e
    Token: a
    Token: b
    Token: c
    Token: d
    Token: e
Line: f;g;h;i;j
    Token: f
    Token: g
    Token: h
    Token: i
    Token: j
Line: 1;2;3;4;5
    Token: 1
    Token: 2
    Token: 3
    Token: 4
    Token: 5

Benner

+1: Devious...the for loop is essentially using strtok() as a combination of strchr() plus write NUL '\0' over where it was found. It only works because you make and dissect a copy of the line; otherwise, you'd have to call strlen(line) at the top of the loop and save that length to add it to line in the third part of the for loop control. – Braun

strtok_r is not the best, because it's NOT C standard (not C89 and not C99) – Vervet

@user411313: it is the best if it is available :-) – Benner