youngkingdom/vllm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
52 Commits 157 Branches 93 Tags
331fa0b042b5cbcdadb03f25dc8592aba9c12b53
Go to file
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE
Woosuk Kwon 331fa0b042 Implement scheduler.step & Add a threshold for batch size
2023-02-23 07:54:20 +00:00
cacheflow
Implement scheduler.step & Add a threshold for batch size
2023-02-23 07:54:20 +00:00
csrc
Add reshape_and_cache op
2023-02-18 19:22:57 +00:00
tests
Add tests for kernels
2023-02-18 19:23:07 +00:00
.gitignore
Add gitignore
2023-02-16 07:47:21 +00:00
README.md
Initial commit
2023-02-09 11:24:15 +00:00
setup.py
cache_kernel -> cache_kernels
2023-02-16 20:05:45 +00:00

README.md

CacheFlow

Description
A high-throughput and memory-efficient inference and serving engine for LLMs
amdblackwellcudadeepseekdeepseek-v3gptgpt-ossinferencekimillamallmllm-servingmodel-servingmoeopenaipytorchqwenqwen3tputransformer
Readme Apache-2.0 687 MiB
Languages
Python 86.1%
Cuda 8.1%
C++ 4.2%
Shell 0.7%
C 0.4%
Other 0.4%