langsmith-trace

Two main topics: adding tracing to your application, and querying traces for debugging and analysis.

Setup

Environment Variables

LANGSMITH_API_KEY=lsv2_pt_your_api_key_here          # Required
LANGSMITH_PROJECT=your-project-name                   # Optional: default project
LANGSMITH_WORKSPACE_ID=your-workspace-id              # Optional: for org-scoped keys

Dependencies

pip install langsmith click rich python-dotenv

Adding Tracing to Your Application

LangChain/LangGraph Apps

Just set environment variables — tracing is automatic:

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY=<your-api-key>
export OPENAI_API_KEY=<your-openai-api-key>  # or your LLM provider's key

Optional variables:

• LANGSMITH_PROJECT - specify project name (defaults to "default")
• LANGCHAIN_CALLBACKS_BACKGROUND=false - use for serverless to ensure traces complete before function exit

Non-LangChain/LangGraph Apps

Check the codebase first: If using OpenTelemetry, prefer the OTel integration (https://docs.langchain.com/langsmith/trace-with-opentelemetry). For Vercel AI SDK, LlamaIndex, Instructor, DSPy, or LiteLLM, see native integrations at https://docs.langchain.com/langsmith/integrations.

If not using an integration, use the @traceable decorator and wrap your LLM client:

Python:

from langsmith import traceable
from langsmith.wrappers import wrap_openai
from openai import OpenAI

client = wrap_openai(OpenAI())

@traceable
def my_llm_pipeline(question: str) -> str:
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": question}],
    )
    return resp.choices[0].message.content

Traces automatically appear in your LangSmith workspace.

Best Practices

• Apply @traceable to all nested functions you want visible in LangSmith. Only decorated functions appear as separate spans in the trace hierarchy.
• Wrapped clients auto-trace all calls — wrap_openai() automatically records every LLM call without additional decorators.
• Name your traces for easier filtering: @traceable(name="retrieve_docs") or traceable(myFunc, { name: "retrieve_docs" })
• Add metadata for searchability: @traceable(metadata={"user_id": "123", "feature": "chat"})

# Example: nested tracing
@traceable
def rag_pipeline(question: str) -> str:
    docs = retrieve_docs(question)  # traced if @traceable applied
    return generate_answer(question, docs)  # traced if @traceable applied

@traceable(name="retrieve_docs")
def retrieve_docs(query: str) -> list[str]:
    # retrieval logic
    return docs

@traceable(name="generate_answer")
def generate_answer(question: str, docs: list[str]) -> str:
    # LLM calls via wrapped client are auto-traced
    return client.chat.completions.create(...)

---

Querying Traces

Use the scripts below to query, analyze, and export traces from LangSmith.

Navigate to skills/langsmith-trace/scripts/ to run commands.

Quick Reference

# Show recent traces
python query_traces.py recent --limit 10 --project my-project

# Show with metadata (timing, tokens, costs)
python query_traces.py recent --limit 10 --include-metadata

# Filter by time
python query_traces.py recent --last-n-minutes 60
python query_traces.py recent --since 2025-01-20T10:00:00Z

# Get specific trace details
python query_traces.py trace <trace-id> --show-hierarchy

# Export traces to directory (recommended for bulk collection)
python query_traces.py export ./traces --limit 50 --include-metadata
python query_traces.py export ./traces --limit 20 --include-io    # With inputs/outputs
python query_traces.py export ./traces --limit 20 --full          # Everything

# Filter by run type
python query_traces.py export ./traces --run-type tool            # Only tool calls
python query_traces.py export ./traces --run-type llm             # Only LLM calls

# Search by name pattern
python query_traces.py search "agent" --project my-project

# Output as JSON
python query_traces.py recent --format json --limit 5

Commands

recent - List recent traces (--limit, --project, --last-n-minutes, --include-metadata, --format)

trace <id> - Get specific trace (--show-hierarchy, --include-metadata, --output)

export <dir> - Bulk export to directory (--limit, --include-metadata, --include-io, --full, --run-type, --max-concurrent)

search <pattern> - Find runs by name (--limit, --last-n-minutes)

Tips

• Use export for bulk data, always specify --project, use /tmp for temp files
• Include --include-metadata for performance/cost analysis
• Increase --max-concurrent 10 for large exports
• Use --format json with jq for analysis

Related skills

• Use langsmith-dataset skill to generate evaluation datasets from traces
• Use langsmith-evaluator skill to create evaluators and measure performance