4 Commits

Author	SHA1	Message	Date
Gea-Suan Lin	14e54f2393	Fix test function naming.	2024-02-19 10:41:38 +08:00
Gea-Suan Lin	3dcd171227	Rename.	2024-02-16 21:02:40 +08:00
Gea-Suan Lin	1de46569e8	Add a simple test case for tokenizer.	2024-02-16 20:59:15 +08:00
Gea-Suan Lin	18fbfa7292	Rename tokenize to tokenizer.	2024-02-09 11:47:13 +08:00