AI Agents and Applications (with LangChain, LangGraph, and MCP)

Prompt Engineering Qualitydesign lever

The degree to which prompts are deliberately structured with clear persona, context, instructions, output format, tone, and in-context examples (zero-shot through few-shot and chain-of-thought) to guide the LLM toward accurate and appropriate outputs for a given task. High quality reflects well-crafted, task-specific prompt templates with appropriate examples and safety instructions such as instructing the model to admit ignorance rather than hallucinate.

Indexing Strategy Sophisticationdesign lever

The extent to which the content ingestion pipeline employs advanced techniques beyond naive fixed-size chunking, including appropriate splitting strategies (HTML header, markdown header, recursive character, sentence-based), multi-vector embedding schemes (child chunk embeddings, summary embeddings, hypothetical question embeddings), chunk expansion with adjacent context, and metadata enrichment. Higher sophistication means the vector store indexes each document in multiple semantically rich ways.

Query Transformation Sophisticationdesign lever

The degree to which the system applies pre-retrieval query reformulation techniques including rewrite-retrieve-read, multiple query generation, step-back abstraction, hypothetical document embeddings (HyDE), and single-step or multi-step decomposition of complex questions. Higher sophistication means the system does not pass raw user queries directly to the vector store but instead reformulates them to maximize semantic alignment with indexed content.

Retrieval Architecture Breadthdesign lever

The extent to which the application draws on multiple heterogeneous content stores beyond a single vector database, including relational SQL databases, graph databases, and metadata-filtered vector search, with appropriate routing logic to direct each query to the most relevant store and post-processing techniques such as reciprocal rank fusion to rerank results from multiple sources.

Agent Orchestration Design Qualitydesign lever

The quality of the agent architecture as measured by adoption of explicit typed state management (TypedDict state schemas), appropriate use of LangGraph's conditional edges and node functions, clear tool descriptions and system prompts that guide LLM tool selection, use of proven patterns such as ReAct and supervisor, and avoidance of brittle linear chains for tasks requiring dynamic decision-making. Encompasses both single-agent and multi-agent system design.

Observability and Tracing Practicedesign lever

The extent to which the development team instruments the LLM application with end-to-end tracing using tools such as LangSmith, enabling inspection of every LLM call, tool invocation, retrieved document, and message flow. High observability practice means traces are captured for all environments, examined during development, and used to diagnose issues in production rather than relying on surface-level outputs alone.

MCP Integration Adoptiondesign lever

The degree to which the agent application leverages the Model Context Protocol to discover and consume remote tools published by MCP servers, rather than maintaining bespoke per-service tool wrappers. Higher adoption means external capabilities are accessed through standardized MCP clients (FastMCP, LangChain MultiServerMCPClient) and the agent tool registry is populated dynamically from MCP server tool catalogs.

Retrieval Relevancepsychological state

The degree to which the document chunks returned by the retrieval stage of a RAG pipeline are semantically pertinent to the user's question and contain sufficient information for the LLM to synthesize a correct and complete answer. High retrieval relevance means the top-k results consistently contain the key facts needed, with minimal noise or off-topic content, and that context length is appropriate—neither too sparse to answer nor too verbose to confuse the model.

Agent Reasoning and Tool Selection Qualitypsychological state

The quality of the agent's intermediate reasoning steps, including its ability to correctly identify which tools to call, formulate accurate tool arguments, interpret tool outputs, know when to request additional tool calls versus when to synthesize a final answer, and avoid defaulting to internal model knowledge when tools should be used. High quality means the agent's chain of thought is coherent, grounded in retrieved evidence, and converges efficiently to a correct final answer.

Context Grounding Qualitypsychological state

The degree to which the LLM's generated responses are anchored to verified external context provided in the prompt (retrieved chunks, tool outputs, database records) rather than to the model's pre-trained parametric knowledge. High grounding means the model cites or refers to retrieved sources, stays within the boundaries of provided context, and explicitly acknowledges uncertainty rather than fabricating plausible-sounding but unverified information.

Answer Accuracyoutcome metric

The correctness and completeness of the final natural language response delivered to the user, measured against a ground truth reference. High accuracy means the answer correctly addresses all parts of the user's question, contains no factual errors, and does not omit critical information available in the knowledge base. Encompasses both factual precision and response completeness.

Hallucination Rateoutcome metric

The proportion of LLM responses that contain fabricated, unsupported, or factually incorrect information not present in the retrieved context or tool outputs. Lower is better. Hallucinations occur when the model fills gaps with plausible-sounding content drawn from parametric memory rather than grounded context, particularly when the retrieved context does not contain the answer.

System Reliability and Maintainabilityoutcome metric

The degree to which the LLM application or agent system behaves predictably, handles edge cases gracefully, can be debugged and extended without major refactoring, and remains stable as underlying models or data sources change. High reliability reflects modular component design, explicit state management, robust error handling, and the ability to swap LLMs or vector stores without rewriting application logic.

Developer Productivityoutcome metric

The speed and ease with which a software developer can design, implement, test, and iterate on LLM-powered applications and agent systems. High productivity reflects reduced boilerplate through framework abstractions (LangChain, LangGraph, pre-built agent components), faster debugging through observability tooling (LangSmith), and reuse of community resources (LangChain Hub, MCP ecosystem) rather than building every component from scratch.