Is there a way to count tokens in C?

Question

I'm using strtok to split a string into tokens. Does anyone know any function which actually counts the number of tokens?

I have a command string and I need to split it and pass the arguments to execve() .

Thanks!

Edit

execve takes arguments as char**, so I need to allocate an array of pointers. I don't know how many to allocate without knowing how many tokens are there.

Stabilize

strtok() and increment a counter? – Counterpressure

And realloc should solve the problem of not knowing the size in advance. – Wagers

Answer 1

One approach would be to simply use strtok with a counter. However, that will modify the original string.

Another approach is to use strchr in a loop, like so:

int count = 0;
char *ptr = s;
while((ptr = strchr(ptr, ' ')) != NULL) {
    count++;
    ptr++;
}

If you have multiple delimiters, use strpbrk:

while((ptr = strpbrk(ptr, " \t")) != NULL) ...

Bettis

strchr() gets cumbersome when there are multiple field delimiters, for example punctuation. – Brolly

You can use strpbrk in that case. – Bettis

OK, I have amended my answer to include strpbrk. Thanks for your feedback. – Bettis

Sorry to barge in like this but is this seemed like the best answer I've found. Is there a way to get the lengths of each segment between the delimiter. Thanks again for this answer. Cheers @Bettis – Weatherbeaten

You can just keep track of the previous value of ptr and take the difference between the two - the pointers point to different parts of the same string. – Bettis

Answer 2

As number of tokens is nothing but one more than the frequency of occurrence of the delimiter used. So your question boils down to find no. of times of occurrence of a character in a string

say the delimiter used in strtok function in c is ' '

int count =0,i;
char str[20] = "some string here";

for(i=0;i<strlen(str);i++){
    if(str[i] == ' ')
        count++;
}

No. of tokens would be same as count+1

Genera

Does the job but gets cumbersome if more than one delimiter is allowed. – Feverwort

I guess we can use the same procedure and should take care of the corner cases – Genera

This would count two spaces in a row as two separate word-separators. Doesn't strtok() treat consecutive delimiters as a single delimiter? – Flamenco

Answer 3

Here is a version based on strtok which does not modify the original string, but a temporal copy of it. This version works for any combination of tabs and space characters used as tokens separators. The function is

unsigned long int getNofTokens(const char* string){
  char* stringCopy;
  unsigned long int stringLength;
  unsigned long int count = 0;

  stringLength = (unsigned)strlen(string);
  stringCopy = malloc((stringLength+1)*sizeof(char));
  strcpy(stringCopy,string);

  if( strtok(stringCopy, " \t") != NULL){
    count++;
    while( strtok(NULL," \t") != NULL )
        count++;
  }

  free(stringCopy);
  return count;
}

A function call could be

char stringExample[]=" wordA 25.4 \t 5.6e-3\t\twordB 4.5e005\t ";
printf("number of elements in stringExample is %lu",getNofTokens(stringExample));

Output is

number of elements in stringExample is 5

Brandtr