Skip to main content

📝 LangSmith

Description​

< What is it? >​

  • Definition: LangSmith is a platform for observing, debugging, and evaluating LLM applications. It also provides prompt-management and deployment capabilities. It works with multiple frameworks and model providers; using LangChain is not a requirement. See LangSmith observability.
  • Example: a developer can inspect a RAG request to identify the retrieved documents, model response, tool calls, and the stage responsible for a failure.

Key points​

< Tracing an application >​

  • Trace: a record of an application execution, including its instrumented component runs. Parent–child relationships connect retrieval, model calls, and tool execution to the overall request.
  • Recorded information: depending on instrumentation, runs include inputs, outputs, timings, errors, and metadata. Tracing records what the application exposes; it does not automatically reveal every internal operation.
  • Ecosystem role: dashed arrows in the LangChain–LangGraph–LangSmith diagram represent tracing events. Recording those events is separate from selecting the workflow's next action.

< Evaluating answer quality >​

  • Datasets and experiments: run application versions against a dataset and compare their results. Evaluators can use code, human review, or model-based judgments.
  • Offline evaluation: test selected examples before releasing a change, including previously observed failures.
  • Online evaluation: assess production interactions using configured evaluators. A trace supplies evidence about execution; quality assessment requires evaluation criteria. See the LangSmith evaluation guide.

Comparison​

< Observability vs. evaluation in LangSmith >​

  • Different questions: tracing and evaluation complement each other.

    CapabilityQuestionExample
    ObservabilityWhat happened during this request?Retrieval returned D17, and the model answered “60 days”
    EvaluationDid the result satisfy the task?Fail a factual-consistency check because D17 states “30 days”

Examples​

< Turn a failed answer into a regression case >​

  • Illustrative input: “How long do I have to return headphones?” The retrieved policy states “30 days,” but the application answers “60 days.”
  • Inspection: use the trace to check the retrieved policy and the context actually passed to the model. This helps distinguish retrieval errors from answer-generation errors.
  • Evaluation case: save the question, relevant context, and expected fact of 30 days as a test case. Compare the original and revised application using a criterion that checks factual agreement, including paraphrased answers. These are illustrative records, not results from a live LangSmith experiment.

Reference​