Split String to tokens: StringTokenizer, String split, and Scanner

The sample code below compares the output and time used for StringTokenizer, String split, and Scanner to break the same string to string segments using the same delimiter. The StringTokenizer is the fastest and does not generate empty tokens. Both String split and Scanner produce empty tokens. Among these two, the String split is faster. However, StringTokenizer can only use a String as the delimiter, String split and Scanner can take a regular expression pattern as the delimiter.

The String.split("<delimiter>") method ignores empty ending token. If you want the empty token to be included in the result, use String.split("<delimiter>", -1)

import java.util.Scanner;
import java.util.StringTokenizer;

public class StringToken {
    public static void main(String[] args) {
        String str = "00002 A000    1 Cholera due to Vibrio cholerae 01, biovar cholerae           Cholera due to Vibrio cholerae 01, biovar cholerae";
        System.out.println("----Tokenizer----");
        long time = System.currentTimeMillis();
        for (int i = 0; i < 100; i++) {
            System.out.println();
            StringTokenizer st = new StringTokenizer(str, " ");
            while(st.hasMoreTokens()){
                System.out.print(st.nextToken()+",");
            }
        }
        long tokenTime = System.currentTimeMillis()-time;
        System.out.println();
        System.out.println("----split----");
        time = System.currentTimeMillis();
        for (int i = 0; i < 100; i++) {
            System.out.println();
            String[] st = str.split(" ");
            for (String s : st) {
                System.out.print(s + ",");
            }
        }
        long splitTime = System.currentTimeMillis() - time;
        System.out.println();
        System.out.println("----Scanner----");
        time = System.currentTimeMillis();
        for (int i = 0; i < 100; i++) {
            System.out.println();
            Scanner sc = new Scanner(str).useDelimiter(" ");
            while (sc.hasNext()) {
                System.out.print(sc.next() + ",");
            }
        }
        long scanTime = System.currentTimeMillis() - time;
        System.out.println();
        System.out.println("StringTokenizer: "+tokenTime);
        System.out.println("Split: "+splitTime);
        System.out.println("Scanner: " + scanTime);
    }
}

Output:
----Tokenizer----
00002,A000,1,Cholera,due,to,Vibrio,cholerae,01,,biovar,cholerae,Cholera,due,to,Vibrio,cholerae,01,,biovar,cholerae,
. . . . . .

----split----
00002,A000,,,,1,Cholera,due,to,Vibrio,cholerae,01,,biovar,cholerae,,,,,,,,,,,Cholera,due,to,Vibrio,cholerae,01,,biovar,cholerae,
. . . . . .

----Scanner----
00002,A000,,,,1,Cholera,due,to,Vibrio,cholerae,01,,biovar,cholerae,,,,,,,,,,,Cholera,due,to,Vibrio,cholerae,01,,biovar,cholerae,
. . . . . .

StringTokenizer: 27
Split: 38
Scanner: 84

---------------------------------------------------------------------------------------------------

                        

If you have ever asked yourself these questions, this is the book for you. What is the meaning of life? Why do people suffer? What is in control of my life? Why is life the way it is? How can I stop suffering and be happy? How can I have a successful life? How can I have a life I like to have? How can I be the person I like to be? How can I be wiser and smarter? How can I have good and harmonious relations with others? Why do people meditate to achieve enlightenment? What is the true meaning of spiritual practice? Why all beings are one? Read the book free here.