2 Commits

Author	SHA1	Message	Date
Parth Sareen	0682dae027	sample: improve ollama engine sampler performance (#9374) ... This change bring in various interface cleanups along with greatly improving the performance of the sampler. Tested with llama3.2 on local machine. Improves performance from ~ 70 tokens/s -> 135 tokens/s with topK(40) enabled. Without topK performance is ~ 110 tokens/s	2025-03-07 12:37:48 -08:00
Parth Sareen	0b7e1676eb	sample: add sampling package for new engine (#8410)	2025-02-24 17:19:01 -08:00