W&B Weave for LLM tracking solves a specific problem: most teams instrument LLM calls with print statements and a CSV, then lose prompt lineage the moment they tweak a template. This guide lays out an ordered path to adopt Weave properly—from op-level tracing to prompt versioning and experiment comparison—without turning your codebase into a telemetry dump.
1. Install and initialize a project
Weave is a Python package (with TS support, but Python is ahead). Pin a version; the API has shifted across minor releases.
pip install weave==0.51.0Initialize once per process. The project name scopes your objects in the W&B UI.
import weave
weave.init("llm-experiments")If you run this outside a script (e.g., a notebook), the first call creates the project. In CI or serving, set WEAVE_PROJECT env var and call weave.init() with no args.
2. Wrap LLM calls with @weave.op
The core primitive is the op. Any function decorated with @weave.op gets traced: inputs, outputs, latency, and call graph.
import openai
import weave
weave.init("llm-experiments")
@weave.op()
def generate(prompt: str, model: str = "gpt-4o-mini") -> str:
client = openai.OpenAI()
resp = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
)
return resp.choices[0].message.contentCall it normally. Weave captures the call asynchronously; you’ll see it in the Weave tab of your W&B project.
Pitfall: do not pass huge objects (full document corpora, base64 PDFs) as op inputs. Weave serializes inputs to its object store. Use weave.ref or log only hashes/summaries.
@weave.op()
def generate_from_doc(doc_id: str, prompt: str) -> str:
# fetch doc outside the logged input
text = fetch_doc(doc_id)
return generate(prompt + "\n" + text[:2000])3. Version prompts with weave.Model
A prompt template is code, but it deserves versioning independent of your Git SHA. Subclass weave.Model and declare the template as an attribute. Publishing creates an immutable version.
import weave
class Summarizer(weave.Model):
prompt_template: str
model_name: str = "gpt-4o-mini"
@weave.op()
def predict(self, text: str) -> str:
prompt = self.prompt_template.format(text=text)
return generate(prompt, self.model_name)
# version 1
v1 = Summarizer(prompt_template="Summarize in one sentence: {text}")
weave.publish(v1, name="summarizer")
# later, tweak and republish
v2 = Summarizer(prompt_template="You are an editor. Condense: {text}")
weave.publish(v2, name="summarizer")In the UI you get a diff between versions and can pin a specific one for inference. Tradeoff: models are snapshots. If you mutate the attribute at runtime without republishing, Weave logs the mutated value but won’t auto-version.
4. Run experiments across model variants
Weave doesn’t require its own eval framework to be useful. Wrap a loop and let ops accumulate.
models = ["gpt-4o-mini", "gpt-4o", "claude-3-5-sonnet"]
texts = load_eval_set()
for m in models:
for t in texts:
out = generate(f"Classify: {t}", model=m)
# optionally scoreFor structured comparison, publish a dataset and use weave.Evaluation (available in recent versions). But the manual loop is enough to spot regressions in tone or latency.
Common mistake: logging every token of a 100k-call production stream to Weave. Sample 5–10% in prod; run full sweeps in offline eval.
5. Capture token usage and latency
Weave’s OpenAI integration auto-patches the client and records usage metadata. Use the integration instead of raw SDK when possible.
from weave.integrations.openai import OpenAI
import weave
weave.init("llm-experiments")
client = OpenAI() # patched
@weave.op()
def generate(prompt: str, model: str):
resp = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
)
return resp.choices[0].message.content, resp.usageThe resp.usage appears in the op’s output panel. You can then group by model in the Weave board to compare cost per successful call. Latency is captured by the op wrapper itself—no extra code.
If you use a non-OpenAI provider, return usage explicitly as a dict and Weave will store it; just don’t expect automatic cost conversion.
6. Pitfalls and tradeoffs
Serialization overhead. Every op input/output is serialized via pydantic and sent to the Weave server. For high-QPS services, this adds 1–5ms per call. Run the Weave client in a background thread or disable tracing in hot paths with weave.trace_disabled().
Prompt drift. Publishing a weave.Model is a manual step. If your team edits templates in a JSON file but forgets to republish, experiments will reference stale versions. Enforce publishing in CI when prompt files change.
No real-time alerting. Weave is for post-hoc analysis. Don’t poll it for latency SLO breaches; use Prometheus for that. Weave answers “which prompt version caused the drop in quality,” not “is the site up.”
Object store cost. Long-text inputs stored indefinitely can inflate W&B storage bills. Set a retention policy or use weave.ref to externalize large blobs.
7. Multi-provider routing through a gateway
When you route across vendors, tracking per-call metadata gets fragmented. If you front models with a gateway like n4n.ai—one OpenAI-compatible endpoint covering 240+ models with automatic fallback on provider degradation—you keep a single call surface. Wrap that call in an op and Weave stays consistent across backends.
from weave.integrations.openai import OpenAI
import weave
weave.init("llm-experiments")
# point at gateway, still OpenAI-compatible
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")
@weave.op()
def routed_generate(prompt: str, model: str):
resp = client.chat.completions.create(
model=model, # e.g. "anthropic/claude-3-5-sonnet"
messages=[{"role": "user", "content": prompt}],
)
return resp.choices[0].message.contentThe gateway handles fallback; Weave records which model string you requested and the latency observed. You lose provider-internal token breakdowns unless the gateway returns them in usage (n4n.ai meters per-token usage and forwards it). That’s enough to attribute cost by experiment.
8. What to do next
Start with one critical prompt path. Wrap it, publish the model, and run a 50-example eval set weekly. Resist the urge to trace every helper function—only ops that change outputs matter.
Once you have two prompt versions and a score, build a Weave board filtering by model_name and prompt_template version. That view is where W&B Weave for LLM tracking pays off: you can show exactly which template edit moved the accuracy number, and roll back by pinning the prior published model.
For deeper workflows, add weave.dataset versioning and automated eval in CI. But the ordered path above is sufficient to stop losing prompt lineage by Friday.