tp1 working multistream tp > 1 broken

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
2025-05-21 19:41:11 +00:00
parent 2259b47951
commit 9c60a6299d
5 changed files with 168 additions and 122 deletions
--- a/examples/basic-ub.py
+++ b/examples/basic-ub.py
@ -40,7 +40,7 @@ def main():
              max_model_len=1024,
              #load_format="dummy",
              ###############
-              tensor_parallel_size=2,
+              tensor_parallel_size=1,
              #data_parallel_size=2,
              enable_expert_parallel=False,
              ###############