youngkingdom/vllm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[Misc] Support FP8 MoE for compressed-tensors (#8588)

Browse Source
This commit is contained in:
Michael Goin
2024-09-25 12:43:36 -04:00
committed by GitHub
parent 64840dfae4
commit 873edda6cf
5 changed files with 226 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
tests/weight_loading/models-large.txt
View File

@ -1,4 +1,5 @@
compressed-tensors, nm-testing/Mixtral-8x7B-Instruct-v0.1-W4A16-quantized, main
compressed-tensors, nm-testing/Mixtral-8x7B-Instruct-v0.1-W4A16-channel-quantized, main
compressed-tensors, nm-testing/Mixtral-8x7B-Instruct-v0.1-W8A16-quantized, main
compressed-tensors, mgoin/DeepSeek-Coder-V2-Lite-Instruct-FP8, main
gptq_marlin, TheBloke/Mixtral-8x7B-v0.1-GPTQ, main