How to use strtok in C properly so there is no memory leak?

Question

I am somewhat confused by what happens when you call strtok on a char pointer in C. I know that it modifies the contents of the string, so if I call strtok on a variable named 'line', its content will change. Assume I follow the bellow approach:

void function myFunc(char* line) {

    // get a pointer to the original memory block
    char* garbageLine = line;

    // Do some work
    // Call strtok on 'line' multiple times until it returns NULL
    // Do more work

    free(garbageLine);
}

Further assume that 'line' is malloced before it is passed to myFunc. Am I supposed to free the original string after using strtok or does it do the job for us? Also, what happens if 'line' is not malloced and I attempt to use the function above? Is it safer to do the following instead? (Assume the programmer won't call free if he knows the line is not malloced)

Invocation

char* garbageLine = line;
myFunc(line);
free(garbageLine);

Function definition

void function myFunc(char* line) {
    // Do some work
    // Call strtok on 'line' multiple times until it returns NULL
    // Do more work
}

Sanferd

Answer 1

strtok() will not free anything, as it has no knowledge of where the string is stored. It could be on the stack or the heap, it doesn't know or care! :)

Is it safer to do the following instead?

Your second example is much better, as it simplifies myFunc(), and makes it useful in more situations as the function does not need to know where the string is allocated. By removing the call to free() from myFunc() you are able to use the function to parse strings from the stack or the heap. The caller allocates the memory, the caller frees the memory!

Further reading: strtok()

Bayly

Note that the second approach can be simplified to myFunc(line); free(line); - the temporary is unnecessary since the pointer line is passed by value to myFunc(). – Calipee

@Calipee You're assuming, of course, that the OP actually malloced line, but that's not evident from the code. Given the evident confusion about memory allocation and deallocation, the assumption may not be warranted. – Kazak

Answer 2

It's worth explaining that strtok does its job by:

1. returning pointers that point INTO the original string; and

2. replacing each separator character that it finds with NULL.

Thus, everything is in-place, and it does not need to allocate any memory.

Priory

Answer 3

In the comment in your question, you say that you "Call strtok on 'line' multiple times until it returns NULL". This sounds as if you may be using strtok incorrectly. The first time you call it, you should call it with 'line' as an argument; on subsequent calls, you should pass it NULL. Take the following as an example:

void function myFunc(char* line) {
  char *segment; // This will point at each delimited substring in turn.

  segment = strtok(line, " ");

  // Do something with segment.

  segment = strtok(NULL, " ");

  // Do something with the new segment.

  free(line);
}

As DrTwox said, though, your second example is better - 'line' should be freed by the same context that malloced it (or not), so the call to free() doesn't belong in this function. And you're better off looping it - something like:

void function myFunc(char* line) {
  char *segment;

  segment = strtok(line, " ");

  while (segment != NULL) {
    // Do something with segment.

    segment = strtok(NULL, " ");
  }
}

Invocation is like this:

char *line = malloc(20*sizeof(char));

// Check that malloc succeeded here.
// Put some data into 'line'.

myFunc(line);

free(line);

// No 'garbageLine' required.

The way that strtok works is a little complex to explain, but you've got the important parts - it doesn't allocate or free any memory. Instead, it works by modifying the string you passed to it.

Corabella

+1 for mentioning that strtok(LINE, "<token>") doesn't work unless all subsequent calls to strtok receive NULL as an argument: strtok(NULL, "<token>"); – Surfactant

Answer 4

What does this have to do with strtok()? If you allocate memory, you need to free it. Where your application decides to allocate and free the memory is up to you. But if you pass the memory to strtok(), that makes no difference as far as if or when the memory is allocated or freed.

Gagnon

Is the way I deallocate memory in both of those approaches correct? All I am wondering is if strtok implicitly deallocotes memory. Keep in mind that I am not an expert in the language. – Sanferd

@user246392: As I said in my answer, calling strtok() has no effect whatsoever on allocated memory. It does not allocate or free any memory. It is not relevant to the question of allocating or freeing memory. Often, it makes sense to free memory from the same routine that allocates it, but if that doesn't fit your app, all that really matters is that it gets freed. – Gagnon

@Sanferd It's not possible to determine whether you are deallocating memory correctly because we have no way of knowing how you allocated it. I can say though that you are almost certainly doing it incorrectly. – Kazak

Answer 5

strtok no more frees memory than strlen does. Why would you expect it to? What memory would it free? Perhaps you think strtok needs to free memory because it stores a NUL, but the content of memory is irrelevant. When you allocate memory, the allocator tracks the size of the block you allocated, and the entire block is freed when you free it.

Kazak