Debugging agents is notoriously complex. Traces can extend for hundreds of steps, prompts can comprise thousands of lines, and the source of a problem can be difficult to pinpoint.

Polly: The AI Assistant for Debugging

Polly is an AI assistant designed to simplify this process. It can analyze complex traces, identify critical points, and provide precise guidance on the cause of errors. Polly is now available to all LangSmith users.

What's New

Previously, Polly was only accessible in certain sections of LangSmith. Now it is integrated into all pages and workflows, including projects, runs, threads, experiments, datasets, annotation queues, and evaluators.

Polly maintains conversation memory, allowing you to switch between tasks without losing context. In addition, it can take concrete actions, such as updating prompts, creating datasets from failed runs, filtering project views, developing evaluator code, and comparing experiments.

Key Features

  • Thread Analysis: Polly can analyze entire conversations between users and agents, providing insights into user sentiment, issues encountered, and problem resolution.
  • Evaluator Development: Polly helps develop and improve evaluator logic, generating code for hallucination checking and edge case handling.
  • Experiment Result Analysis: Polly can compare the results of different experiments, providing recommendations based on actual data.

Get Started

Polly is accessible via the Cmd+I (Mac) or Ctrl+I (Windows/Linux) key combination on any LangSmith page. To start using Polly, you need to add an API key for your model provider as a workspace secret.