C strtok function

The strtok function in C is a String method to parse or tokenize a given string using a delimiter. The syntax of this strtok function is

void *strtok(char *str, const char *delimiter);

C strtok function Example

The strtok function is useful to tokenize the given string based on the delimiter we gave. This program helps you to understand the strtok() method with multiple examples using a while loop.

TIP: You must include the C Programming #include<string.h> header before using this String Function in a program.

#include <stdio.h> 
#include<string.h>

int main()
{
    char str[] =  "This is xyz working in abc company";
    char *res;
    
    res = strtok(str, " ");
    
    while(res != NULL)
    {
        printf("%s \n", res);
        res = strtok(NULL, " ");
    }
    
    return 0;
}
C strtok to tokenize string example

C strtok multiple delimiters

Apart from the above-mentioned single delimiter, the strtok() function supports multiple delimiters. We can use the second argument to specify all delimiter characters as a group.

In the following example, we declared a string with different fruit items, and each fruit was separated by either a comma, semicolon, or whitespace. So, we used the strtok() function’s second argument as a comma, a semicolon, and a space. It means the strtok() method splits the string if it finds any of the mentioned separators (“,; “).

NOTE: It does not consider “,; ” as a group. Instead, the strtok() function treats them as individual delimiters and splits if there is a match with any of these delimiters.

#include <stdio.h>
#include <string.h>
int main(void) {
    char str[] = "apple,kiw;banana,orange,cherry grape pineapple";  
    char *token = strtok(str, ",; ");
    
    while (token != NULL) {
        printf("%s\n", token);
        token = strtok(NULL, ",; ");
    }
    return 0;
}
apple
kiw
banana
orange
cherry
grape
pineapple

Split a string by the newline character or tab space using strtok in C

Similar to the other delimiters, we can use the strtok() function to split the string by the newline character (\n) or a tab space (\t). In the following example, we declared a string with \t and \n characters. Next, we passed those characters as the strtok() second argument, so that it considers \t and \n as delimiters to split the string.

#include <stdio.h>
#include <string.h>

int main(void) {
    char str[] = "Hello\tGuys\n\nWelcome to Coding!";

    char *res = strtok(str, "\t\n");

    while (res != NULL) {
        printf("%s\n", res);
        res = strtok(NULL, "\t\n");
    }

    return 0;
}
Hello
Guys
Welcome to Coding!

What does strtok return when there are no more tokens?

The C strtok() function returns NULL if there are no more tokens. In the following example, the string has two characters separated by a space. When we use the strtok() function with a whitespace delimiter, on the first call, it returns a. The second call returns b, whereas the third call returns NULL.

As it is not possible to show NULL as the output, we used the if condition. However, in real programs, we use a while loop and check whether the token is NULL. If it is not null, print the split string from that token.

#include <stdio.h>
#include <string.h>

int main(void) {
    char str[] = "a b";

    char *token;

    token = strtok(str, " ");
    printf("1st call: %s\n", token);

    token = strtok(NULL, " ");
    printf("2nd call: %s\n", token);

    token = strtok(NULL, " ");
    if (token == NULL)
        printf("3rd call: NULL\n");
    return 0;
}
1st call: a
2nd call: b
3rd call: NULL

How to split a string into an array of strings in C?

The most common way to split the original string into an array of strings using a delimiter is by using the C strtok() function.

In the following example, there are four fruits in our original string, and we used the strtok() to split them based on a whitespace. The while loop iterates over the string based on the strtok() method results, and inside it, we add each word to the fruits string array.

The last for loop iterates over the string array and prints each fruit based on the index position.

#include <stdio.h>
#include <string.h>

int main(void) {
    char str[] = "Apple Kiwi Banana Cherry";

    char *fruits[10];
    int total = 0;

    char *sub = strtok(str, " ");

    while (sub != NULL && total < 10) {
        fruits[total++] = sub;
        sub = strtok(NULL, " ");
    }

    for (int i = 0; i < total; i++) {
        printf("fruits[%d] = %s\n", i, fruits[i]);
    }
    return 0;
}
fruits[0] = Apple
fruits[1] = Kiwi
fruits[2] = Banana
fruits[3] = Cherry

What is the difference between C strtok and strtok_r?

strtok_r is more advanced, and r represents reentrant. We can call strtok_r as the thread-safe version of the strtok() method. Similar to the strtok(), the strtok_r function splits the given string into tokens based on the specified delimiters.

The major difference between strtok and strtok_r comes where the tokenizer state is stored.

  • strtok: It uses an internal static variable to keep the state. If we use strtok() in multi-threading or nested parsing, we lose data.
  • strtok_r: It stores the state in a given variable. Because of this reentrant, strtok_r is thread safe, and we can simultaneously tokenize multiple strings (multi-threading). It also works well in nested tokenization.

The syntax of the C strtok_r is as shown below

char *strtok_r(char *str, const char *delim, char **saveptr);

Parameters

  • str: It is the original string that we must split. We must pass the string on the first call only, and for the subsequent calls, use NULL.
  • Delim: Please pass a single or multiple delimiters that the strtok_r function should use to split the string.
  • saveptr:  A pointer to a variable. This variable is used internally to store the state.

Return Value: Both the C strtok() and strtok_t() functions return a pointer to the next token. If there are no tokens, they return NULL.

#include <stdio.h>
#include <string.h>

int main(void) {
    char str[] = "Apple Kiwi Banana Cherry";
    char *saveptr;
    char *token = strtok_r(str, ",", &saveptr);

    while (token != NULL) {
        printf("%s\n", token);
        token = strtok_r(NULL, ",", &saveptr);
    }
    return 0;
}
Apple Kiwi Banana Cherry