Deploy vLLM as an OpenAI-Compatible Server

Run an open-weight model behind an OpenAI-compatible /v1/chat/completions endpoint so Hermes, coding agents, and eval harnesses can swap models without rewriting clients.

• A local or server GPU process serving chat completions
• A base URL your agents can target (http://host:8000/v1)
• A path to run tool-eval-bench style suites against your stack

1. Create an environment dedicated to serving (do not mix with random notebooks).
2. Install vLLM following the current upstream install matrix for your CUDA version.
3. Launch the server with model id/path, host, and port. Prefer explicit max model length and GPU memory utilization flags so you do not OOM silently.
4. Smoke test with a curl chat completion using model equal to the served id.
5. Point an agent client at base_url + API key stub if required by the client library.

Example shape (adjust to current vLLM CLI):

# Illustrative — check `vllm serve --help` for your installed version
vllm serve <model-id-or-path> --host 0.0.0.0 --port 8000

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"<model-id>","messages":[{"role":"user","content":"ping"}]}'

• Health endpoint or successful chat completion
• Token generation latency acceptable for your agent loop
• Tool-call path tested if you will use function calling

• OOM: lower max context, tensor parallel carefully, or pick a smaller quant
• Client 404 model: the model field must match the served model name
• Slow first token: expected for cold load; warm with a short prompt

vLLM flags change between releases. Treat this recipe as the workflow, not a frozen flag list — always re-read upstream docs for your version.