What's the difference between strtok_r and strtok_s in C?

Question

I'm trying to use this function in a C program that needs to be able to compile in Linux and Windows. At first I tried using strtok_r, but then when I compiled on windows, it complained about the function not existing and said it would assume it's an extern function, but then failed. I then used strtok_s and it compiled! Then I tried on Linux but now it's complaining that there is an "undefined reference to 'strtok_s'".

Is one a windows only function and the other a linux function??? What can I do to make it compile on both?

Sublingual

strtok_r() is POSIX; strtok_s() is Windows. Use plain strtok() which is available in all implementations (supporting C89). – Gabrielegabriell

The problem is I need to strtok a string while strtok-ing another string, which is why I need this one since it saves state. – Sublingual

@Gabrielegabriell - unless you need thread or buffer overrun safety – Britishism

Well ... then I refer to R..'s answer. – Gabrielegabriell

@MartinBeckett Wait what? Where does it save the state? There are only two arguments, one is NULL (on the second token), the other is what character you want to find. I want to be able to strtok a string, find something in that string and strtok that one, then go back to the original string and continue to strtok it. – Sublingual

@Gabrielegabriell - sorry was thinking of something else, my mistake! – Britishism

@user390608: The problem you're encountering is exactly why strtok is such a horrible interface. It is stateful and non-reentrant. – Aviculture

This is an ancient question by now, but I felt the need to point out that strtok_s is C11, not just Windows. – Prizewinner

@kyrias close - it's C11 Annex K, which Ulrich Drepper has fought tooth-and-nail to keep from libc users. No, really, that's why linux devs can't use it. One guy. – Delphadelphi

on linux / unix, in glibc, there is strsep. see also What are the differences between strtok and strsep in C – Turgid

Answer 1

Both of these functions are really ugly, unintuitive idioms for parsing strings, and usually fail to meet your particular application's requirements in subtle ways. Even moreso for the plain strtok in standard C. Just throw them out and write your own code to iterate over the char array and break it up as needed. strchr, strspn, and strcspn can be helpful in doing this or you can just work from scratch on the array.

Aviculture

Can you explain a little more about why these functions are bad? It seems more intuitive to me than going through character by character and searching for things. – Sublingual

(1) the separator char is overwritten/lost, so unless you only have one possible separator, you're losing information. (2) multiple consecutive separators are treated as one, which makes sense if your separator is whitespace but not if it's a comma, for instance.. – Aviculture

Please note that surely in some situations, the strtok semantics are just what you want and the only issue is the statefulness (which strtok_r and strtok_s fix). I just find that every time I tried to use them, there's at least one aspect of my parsing needs they don't support, and it ends up being easier to just write the code directly. – Aviculture

-1: For parsing a string that's been read into a temporary buffer, where you want to throw away the delimeters and you only want the strings valid until you're done parsing, strtok is precisely what you want and strtok_r/strtok_s is much better as it is reentrant. – Nekton

"write your own code" -- see also How do I split a string without strtok? and String separator, similar to strtok and glibc/string/strtok_r.c – Turgid

Answer 2

strtok_s is simply the Windows version of strtok_r which is standard everywhere else.

One (common I would think) way to make a program portable when it comes to functions like strtok_s/strtok_r is to use the preprocessor:

#if defined(_WIN32) || defined(_WIN64)
/* We are on Windows */
# define strtok_r strtok_s
#endif

As the prototypes and functionality is the same, you can now use only strtok_r.

Gerri

This would probably break if compiling on cygwin which reports itself as windows but has posix interfaces like strtok_r already defined. Using something like #ifndef HAVE_STRTOK_R (and detecting it in the build scripts) would be better. – Aviculture

@R..: cygwin does not in fact define _WIN32 (or _WIN64) for precisely this reason -- its commonly used to identify code that is non-unix like. Cygwin defines __CYGWIN__ and __unix__ instead – Nekton

It may be good to know that there is an other strtok_s in C11 with a completely different signature: en.cppreference.com/w/c/string/byte/strtok – Lati

@ChrisDodd There are still problems on MinGW. MinGW predefines _WIN32 but it supports strtok_r. I think it is better to check the _MSC_VER macro. – Iambic

Answer 3

I don't have enough reputation to comment on other answers, so I'll have to provide my own.

1) To address this statement:

"strtok_s is a buffer overrun safe version of strtok on Windows. The standard strtok on windows is thread safe..."

This is not true. strtok_s is the thread safe version for the MSVC compiler. strtok is not thread safe!

2) To address this statement:

"This would probably break if compiling on Cygwin which reports itself as windows but has POSIX interfaces like strtok_r already defined."

Again, not true. The difference is which compiler you use. When using Microsoft's Visual C++ compiler, MSVC, the function is strtok_s. Another compiler, such as the GNU Compiler Collection, GCC, may use a different standard library implementation such as strtok_r. Think compiler, not target platform, when identifying which function to use.

In my opinion, Joachim Pileborg's answer is the best one on this page. However, it needs a small edit:

#if defined(_WIN32) /* || defined(_WIN64) */
#define strtok_r strtok_s
#endif

Both _WIN32 and _WIN64 are predefined macros provided by the MSVC compiler. _WIN64 is defined when compiling a 64 bit target. _WIN32 is defined for both 32 and 64 bit targets. This is a compromise that Microsoft made for backwards compatibility. _WIN32 was created to specify the Win32 API. Now you should consider _WIN32 to specify Windows API -- it is not specific to a 32 bit target.

Guardhouse

You are wrong about the first point: strtok is thread-safe on Windows, since two distinct threads can safely use that function within the same process, even if strtok have other problems as explained in another answer by @JameyKirby. – Archive

MinGW predefines _WIN32 but it supports strtok_r, so it is better to check the _MSC_VER macro. – Iambic

Answer 4

Just to clarify. strtok is thread safe in Windows. strtok uses a TLS variable to maintain the last pointer for each thread. However, you can not use strtok to interleave access to more than one token string per thread. strtok_r and strtok_s both address this interleaving problem by allowing the user to maintain the context via the third parameter. Hope this helps.

Hyperbolic

Although not helpful for the OP, who is looking for a portable solution, this was very helpful for me to find out why my legacy code crashes on Linux but not on Windows. As I thought I knew how strtok behaves, I didn't care to read the Windows strtok documentation thoroughly... – Expose

Answer 5

strtok_r is a thread safe version of strtok on POSIX systems

strtok_s is a buffer overrun safe version of strtok on Windows. The standard strtok on windows is thread safe, so strtok_s should be.

Britishism

strtok_r is part of POSIX and has been for years (decades?) – Aviculture

I still think the answer isn't terribly useful since it just covers what OP (mostly) already knew, not how to solve the problem. – Aviculture

@R - the solution is you can use strok_r on Posix and strtok_s on Windows, Joachim's solution describes how. OR you can program windows using only the Posix subsystem – Britishism

@MartinBeckett it's C11 Annex K standard, not just Windows :) – Delphadelphi

@AlexanderRiccio Microsoft strtok_s and C11 strtok_s are completely different! Microsoft strtok_s has only 3 parameters, while C11 strtok_s has 4 parameters. The prototype of Microsoft strtok_s is char* strtok_s(char* str, const char* delimiters, char** context); while the prototype of C11 strtok_s is char *strtok_s(char *restrict str, rsize_t *restrict strmax, const char *restrict delim, char **restrict ptr);. – Iambic

Answer 6

MinGW also predefines _WIN32 but it supports strtok_r, so I don't think it is a good idea to check the _WIN32 macro. It is better to check the _MSC_VER macro, which is the macro for Microsoft Visual Studio.

#ifdef _MSC_VER
#define strtok_r strtok_s
#endif

WARNING: Microsoft strtok_s and C11 strtok_s are completely different! Microsoft strtok_s has only 3 parameters, while C11 strtok_s has 4 parameters, so it can be a compatible issue in the future.

The prototype of Microsoft strtok_s is

char* strtok_s(char* str, const char* delimiters, char** context);

The prototype of C11 strtok_s is

char *strtok_s(char *restrict str, rsize_t *restrict strmax, const char *restrict delim, char **restrict ptr);

Iambic