vllm/offline_inference at 37bdf9f324d665d4b8b08f99ccfdf1807ded0eb4 - vllm

basic

manually manage stream

2025-05-22 20:51:36 +00:00

disaggregated-prefill-v1

Improve examples rendering in docs and GitHub (#18203 )

2025-05-15 15:57:49 +00:00

openai_batch

Improve examples rendering in docs and GitHub (#18203 )

2025-05-15 15:57:49 +00:00

profiling_tpu

Update deprecated Python 3.8 typing (#13971 )

2025-03-02 17:34:51 -08:00

qwen2_5_omni

[Misc] extract parser.parse_args() (#18323 )

2025-05-19 04:06:26 +00:00

audio_language.py

[Model] Add Granite Speech Support (#16246 )

2025-04-28 10:05:00 +00:00

batch_llm_inference.py

[Ray] Improve documentation on batch inference (#16609 )

2025-04-16 22:19:26 -07:00

chat_with_tools.py

fix: typos (#18151 )

2025-05-15 02:16:15 -07:00

data_parallel.py

support more args in dp example

2025-05-22 20:51:35 +00:00

disaggregated_prefill.py

Construct KVTransferConfig properly from Python instead of using JSON blobs without CLI (#17994 )

2025-05-12 11:25:33 -07:00

eagle.py

[V1][Spec Decode] Share input embedding of target model with EAGLE draft model to free ~1GB for llama 3 model (#17326 )

2025-05-14 12:31:46 -07:00

embed_jina_embeddings_v3.py

[Misc] refactor argument parsing in examples (#16635 )

2025-04-15 08:05:30 +00:00

embed_matryoshka_fy.py

[Misc] refactor argument parsing in examples (#16635 )

2025-04-15 08:05:30 +00:00

encoder_decoder_multimodal.py

[Bugfix] Update Florence-2 tokenizer to make grounding tasks work (#16734 )

2025-04-17 04:17:39 +00:00

encoder_decoder.py

[Misc] refactor argument parsing in examples (#16635 )

2025-04-15 08:05:30 +00:00

llm_engine_example.py

[Misc] refactor examples series (#16708 )

2025-04-16 10:16:36 +00:00

load_sharded_state.py

[Bugfix][V1] Fix bug from putting llm_engine.model_executor in a background process (#15367 )

2025-04-03 07:32:10 +00:00

lora_with_quantization_inference.py

[Misc] Remove qlora_adapter_name_or_path (#17699 )

2025-05-06 23:10:37 -07:00

mistral-small.py

[VLM] Clean up models (#16873 )

2025-04-19 12:13:06 +00:00

mlpspeculator.py

[Misc] refactor argument parsing in examples (#16635 )

2025-04-15 08:05:30 +00:00

multilora_inference.py

[Misc] format and refactor some examples (#16252 )

2025-04-08 10:42:32 +00:00

neuron_eagle.py

Add NeuronxDistributedInference support, Speculative Decoding, Dynamic on-device sampling (#16357 )

2025-05-07 00:07:30 -07:00

neuron_int8_quantization.py

[Misc] format and refactor some examples (#16252 )

2025-04-08 10:42:32 +00:00

neuron_speculation.py

Add NeuronxDistributedInference support, Speculative Decoding, Dynamic on-device sampling (#16357 )

2025-05-07 00:07:30 -07:00

neuron.py

[Misc] format and refactor some examples (#16252 )

2025-04-08 10:42:32 +00:00

prefix_caching.py

[Misc] format and refactor some examples (#16252 )

2025-04-08 10:42:32 +00:00

prithvi_geospatial_mae.py

[Misc] refactor argument parsing in examples (#16635 )

2025-04-15 08:05:30 +00:00

profiling.py

Fix broken example: examples/offline_inference/profiling at scheduler_config (#18117 )

2025-05-13 23:19:14 -07:00

prompt_embed_inference.py

[Misc] refactor prompt embedding examples (#18405 )

2025-05-20 15:26:12 +00:00

qwen_1m.py

Implements dual-chunk-flash-attn backend for dual chunk attention with sparse attention support (#11844 )

2025-05-12 19:52:47 -07:00

reproducibility.py

[Doc] Fix a typo in the file name (#17836 )

2025-05-08 18:04:18 +08:00

rlhf_colocate.py

[RLHF] use worker_extension_cls for compatibility with V0 and V1 (#14185 )

2025-03-07 00:32:46 +08:00

rlhf_utils.py

[RLHF] use worker_extension_cls for compatibility with V0 and V1 (#14185 )

2025-03-07 00:32:46 +08:00

rlhf.py

[Misc] format and refactor some examples (#16252 )

2025-04-08 10:42:32 +00:00

save_sharded_state.py

[Misc] refactor argument parsing in examples (#16635 )

2025-04-15 08:05:30 +00:00

simple_profiling.py

[Misc] refactor argument parsing in examples (#16635 )

2025-04-15 08:05:30 +00:00

structured_outputs.py

[Misc] refactor Structured Outputs example (#16322 )

2025-04-09 23:32:42 +00:00

torchrun_example.py

[Feature] Support Pipeline Parallism in torchrun SPMD offline inference for V1 (#17827 )

2025-05-15 22:28:27 -07:00

tpu.py

[TPU] Increase block size and reset block shapes (#16458 )

2025-05-06 13:55:04 -04:00

vision_language_embedding.py

[Misc] refactor argument parsing in examples (#16635 )

2025-04-15 08:05:30 +00:00

vision_language_multi_image.py

[Model] Broadcast Ovis2 implementation to fit Ovis1.6 (#17861 )

2025-05-11 17:56:30 -07:00

vision_language.py

[Model] Broadcast Ovis2 implementation to fit Ovis1.6 (#17861 )

2025-05-11 17:56:30 -07:00