3 Commits

Author	SHA1	Message	Date
Cyrus Leung	5ec9c0fb3c	[Core] Factor out input preprocessing to a separate class (#7329)	2024-09-13 02:56:13 +00:00
Cyrus Leung	5ae5ed1e60	[Core] Consolidate prompt arguments to LLM engines (#4328) ... Co-authored-by: Roger Wang <ywang@roblox.com>	2024-05-28 13:29:31 -07:00
GeauxEric	a37d815b83	Make initialization of tokenizer and detokenizer optional (#3748) ... Co-authored-by: Yun Ding <yunding@nvidia.com> Co-authored-by: Roger Wang <ywang@roblox.com>	2024-04-21 22:06:46 +00:00