Gea-Suan Lin 1de46569e8 Add a simple test case for tokenizer.

2024-02-16 20:59:15 +08:00

..

tokenize_test.go

Add a simple test case for tokenizer.

2024-02-16 20:59:15 +08:00

tokenize.go

Rename tokenize to tokenizer.

2024-02-09 11:47:13 +08:00