Deploy DeepSeek-V3 with vLLM behind a LangChain agent

To deploy DeepSeek-V3 with vLLM behind a LangChain agent, you stand up a vLLM inference server that mimics the OpenAI chat protocol, then point LangChain’s ChatOpenAI client at that local endpoint. This walkthrough covers the full path from GPU provisioning to a running tool-calling agent, using only publicly available packages and the official model weights.

Step 1: Provision hardware and install vLLM

DeepSeek-V3 is a 671B-parameter Mixture-of-Experts model. A single 80GB GPU will not hold it; plan for 8× A100/H100 80GB with tensor parallelism, or use FP8 weights if your hardware supports them. The model weights require roughly 1.3TB of disk for full precision and about 700GB for FP8 checkpoints. Start with a clean Python 3.10+ environment on a CUDA 12.1 host.

pip install vllm==0.6.3  # or latest stable
pip install torch --index-url https://download.pytorch.org/whl/cu121

If you prefer containers, the official image bundles the server:

docker pull vllm/vllm-openai:latest

Verify CUDA visibility before downloading weights:

nvidia-smi -L

The model is hosted on Hugging Face as deepseek-ai/DeepSeek-V3. If you are on a restricted network, set HF_HOME and use a local mirror.

Step 2: Start the vLLM OpenAI-compatible server

vLLM ships an API server that speaks the OpenAI /v1/chat/completions contract. Launch it with tensor parallelism sized to your GPU count. Run from the host or inside the container with --gpus all.

python -m vllm.entrypoints.openai.api_server \
  --model deepseek-ai/DeepSeek-V3 \
  --served-model-name deepseek-v3 \
  --tensor-parallel-size 8 \
  --port 8000 \
  --max-model-len 8192 \
  --quantization fp8 \
  --gpu-memory-utilization 0.9

The server blocks the terminal. In production, wrap it in systemd or a Kubernetes Deployment. The --served-model-name flag decouples the external model ID from the Hugging Face path, which matters when you later configure LangChain. If you need tool calling, add --enable-auto-tool-choice and --tool-call-parser hermes only if your vLLM build supports it for this model; otherwise skip and use a ReAct prompt (see Step 7 note).

Step 3: Verify the server independently

Before involving LangChain, confirm the endpoint responds. List models with curl:

curl http://localhost:8000/v1/models

You should see JSON listing deepseek-v3. Then run a minimal completion:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-v3",
    "messages": [{"role": "user", "content": "What is 2+2?"}],
    "max_tokens": 50
  }'

A valid response object with choices confirms the stack works. If you get 503, the model is still loading; watch the server log for Started engine. You can also test with the OpenAI Python client:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
print(client.chat.completions.create(model="deepseek-v3", messages=[{"role":"user","content":"hi"}], max_tokens=10).choices[0].message.content)

Step 4: Install LangChain packages

In a separate Python environment (or the same), install the OpenAI integration and core langchain.

pip install langchain-openai langchain-core langchain

We use langchain-openai because vLLM mirrors the OpenAI schema exactly. No custom client needed. Pin versions if you target reproducibility:

pip install langchain-openai==0.2.0 langchain-core==0.3.0

Step 5: Point ChatOpenAI at the local endpoint

LangChain’s ChatOpenAI accepts a base_url and any API key. vLLM ignores auth, so pass a placeholder.

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="deepseek-v3",
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
    temperature=0.2,
    max_tokens=1024,
)

Call it directly to confirm the wiring:

resp = llm.invoke("Explain tensor parallelism in one sentence.")
print(resp.content)

If this prints coherent text, you have successfully managed to deploy DeepSeek-V3 with vLLM behind a LangChain agent-ready interface. For streaming, use llm.stream("…") and iterate chunks.

Step 6: Define tools for the agent

A LangChain agent needs callable tools. Define a trivial one to demonstrate routing:

from langchain_core.tools import tool

@tool
def get_server_load() -> str:
    """Return current fake server load percentage."""
    return "42%"

@tool
def multiply(a: int, b: int) -> int:
    """Multiply two integers."""
    return a * b

Tools must have accurate docstrings; the model uses them to decide invocation. Type hints are forwarded as JSON schema. Keep signatures simple—DeepSeek-V3 can mishandle deeply nested objects.

Step 7: Build the tool-calling agent

Use the modern create_tool_calling_agent constructor with a prompt that includes a agent_scratchpad placeholder.

from langchain.agents import AgentExecutor, create_tool_calling_agent
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a helpful ops assistant with access to tools."),
    ("human", "{input}"),
    MessagesPlaceholder(variable_name="agent_scratchpad"),
])

tools = [get_server_load, multiply]
agent = create_tool_calling_agent(llm, tools, prompt)
executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

Note: DeepSeek-V3 is not trained on OpenAI function-call tokens. vLLM will forward the tools schema, but the model may return a JSON blob inside content rather than a structured tool_calls field. If you observe that, switch to create_react_agent with a ReAct prompt that instructs the model to emit Action: and Action Input: strings. The rest of the loop stays identical.

Step 8: Run and verify the agent loop

Invoke with a query that forces tool use:

result = executor.invoke({"input": "What is the server load and what is 7 times 6?"})
print(result["output"])

Expected console output shows the agent calling get_server_load and multiply, then synthesizing a final answer. If verbose=True, you see the intermediate steps. That is the proof your deploy DeepSeek-V3 vLLM LangChain agent pipeline is functional.

For async serving:

import asyncio
async def run():
    return await executor.ainvoke({"input": "Multiply 12 by 9"})
print(asyncio.run(run())["output"])

Step 9: Handle concurrency and timeouts

vLLM processes requests asynchronously but has a max batch size. Set --max-num-seqs 256 to match expected QPS. In LangChain, wrap calls with a timeout:

from langchain_core.runnables import RunnableConfig

async def run_safe():
    try:
        return await executor.ainvoke(
            {"input": "What is 8*7?"},
            config=RunnableConfig(timeout=30)
        )
    except asyncio.TimeoutError:
        return {"output": "Agent timed out"}

print(asyncio.run(run_safe())["output"])

If you see RuntimeError: CUDA out of memory, reduce --gpu-memory-utilization or lower --max-model-len.

Step 10: Production notes and alternatives

For a single node, export VLLM_WORKER_MULTIPROC_METHOD=spawn if you hit fork issues on Python 3.10. Use Prometheus metrics at http://localhost:8000/metrics to track token throughput. Log X-Request-Id from the vLLM response for tracing through LangChain.

If operating multi-GPU vLLM clusters is not viable for your team, an OpenAI-compatible gateway such as n4n.ai exposes DeepSeek-V3 with automatic fallback when a provider is degraded, letting you keep the same LangChain client code without owning the GPUs. Self-hosting wins on data residency and latency control.

Verify success checklist

curl /v1/models returns deepseek-v3.
Direct ChatOpenAI call returns text.
Agent executor invokes at least one tool and returns a merged answer.
No CUDA out of memory errors in server log.

Any of these failing points to a misconfigured --tensor-parallel-size or wrong base_url. The pattern above is the minimal skeleton; extend tools and prompt as needed for your domain.