The directory of AI wrapper tools and LLM middleware
Almost no production application talks to a large language model directly. Between your code and the raw completion endpoint sits a growing stack of wrapper tools — the libraries, gateways, and platforms that make LLMs reliable, observable, affordable, and safe enough to ship. WrapDirectory catalogs that middleware layer in one place, so engineering and platform teams can compare options without wading through a dozen marketing sites. Every listing here is a real, verifiable project or product with a link to its official source.
We organize the landscape into five practical categories. LLM frameworks — such as LangChain, LlamaIndex, DSPy, Haystack, Semantic Kernel, CrewAI, and Pydantic AI — give you the building blocks for chains, retrieval-augmented generation (RAG), and multi-agent orchestration. Routing and gateways — LiteLLM, OpenRouter, Portkey, Kong AI Gateway, vLLM, and Ollama — sit in front of one or many model providers to normalize APIs, balance load, cache responses, control spend, and remove vendor lock-in. Observability tools — Langfuse, LangSmith, Helicone, Phoenix, Weights & Biases, Braintrust, AgentOps, OpenLLMetry, and Promptfoo — capture traces, run evaluations, and track cost and latency so non-deterministic behavior becomes debuggable. Prompt management platforms like PromptLayer and Agenta version prompts and decouple them from application code. And guardrails projects such as Guardrails AI and NVIDIA NeMo Guardrails enforce safety, structure, and topical boundaries on what models say and do.
Why does this middleware matter? A bare model call is a single point of failure: if your provider has an outage, your product goes down; if a prompt drifts, quality silently degrades; if an agent loops, your bill explodes. The tools in this directory address exactly those risks. A routing proxy adds automatic fallbacks and retries across providers. An observability platform turns an opaque chain of model calls into an inspectable trace with token counts and evaluation scores. A guardrails layer blocks prompt injection and PII leakage before it reaches a user. A structured-output library like Instructor, Outlines, or Mirascope guarantees the model returns valid JSON the first time instead of breaking your parser. Adopted together, these wrappers are what separate a weekend demo from a system that survives real traffic.
The market moves quickly, with new open-source projects and commercial platforms appearing every month and existing ones overlapping into adjacent categories — many gateways now bundle observability and guardrails, and many frameworks ship their own tracing. WrapDirectory tracks these shifts and keeps each entry current with a plain-language editorial summary of what the tool does, who builds it, and where it fits in the stack. Use the category filters and search above to narrow the field to your use case, whether you are choosing your first framework, hardening an agent for production, or standardizing tooling across an engineering organization. Note too that the categories are guidance rather than hard walls: a project such as Portkey spans routing, caching, observability, and guardrails at once, while LangChain, LlamaIndex, and Langfuse each blur the line between framework and platform — so it is worth reading the editorial summary for any tool you shortlist rather than relying on its label alone. Many of the strongest production stacks combine several of these tools, pairing a framework for orchestration with a gateway for resilience, an observability platform for evaluation, and a guardrails layer for safety. Browse the full listing below and follow the official links to dig deeper.
Featured
The most widely used open-source framework for building LLM applications — chains, agents, RAG, and integrations.
LangChain is the most widely adopted open-source framework for building applications on top of large language models, with libraries in both Python and JavaScript/TypeScript. It pioneered the abstraction of "chains" that compose prompts, models, memory, and tools into reusable pipelines, and it ships hundreds of integrations spanning vector stores, document loaders, and model providers. Maintained by LangChain, Inc. (which also builds LangSmith and LangGraph), it remains a default starting point for RAG systems and agentic workflows.
LLM Frameworks
⭐ 138,040 stars
python javascript framework open-source
Featured
Data framework for LLM apps — connect, structure, and query private data with agentic RAG workflows.
LlamaIndex is a data framework purpose-built for connecting large language models to private and domain-specific data. It provides ingestion connectors, indexing structures, and query engines that make retrieval-augmented generation reliable at scale, and it has expanded into agentic workflows through LlamaAgents and the managed LlamaCloud parsing service. Created by Jerry Liu and maintained by the LlamaIndex team, it is available in both Python and TypeScript and is one of the most popular RAG-focused libraries in production use today.
LLM Frameworks
⭐ 49,775 stars
python typescript rag data-indexing
Unified API layer for 100+ LLMs — call any model with OpenAI-compatible syntax; load balancing and fallbacks.
LiteLLM, built by BerriAI, is an open-source unified interface that lets developers call more than 100 large language model providers using the familiar OpenAI completion syntax. It ships both as a lightweight Python SDK and as a self-hostable proxy server (gateway) that adds load balancing, automatic fallbacks, retry logic, spend tracking, and virtual API keys for teams. By normalizing inputs and outputs across providers such as OpenAI, Anthropic, Azure, Bedrock, and Vertex AI, it removes vendor lock-in and is widely used as the routing layer in production LLM stacks.
Routing & Gateways
⭐ 48,755 stars
proxy multi-model openai-compatible open-source
Stanford framework for programming — not prompting — LLMs, with automated prompt optimization.
DSPy is a framework from the Stanford NLP group for programming language models rather than hand-crafting brittle prompts. Developers declare the input and output behavior they want using typed signatures and composable modules, and DSPy's optimizers (such as MIPROv2 and BootstrapFewShot) automatically compile and tune the underlying prompts and few-shot examples against a metric. This shifts prompt engineering from manual trial-and-error to a systematic, measurable optimization process, and the project has become a leading approach for building self-improving LLM pipelines.
LLM Frameworks
⭐ 34,733 stars
python stanford prompt-optimization open-source
Microsoft-originated library to constrain and control LLM generation with a programmatic templating language.
Guidance is an open-source library, originally created at Microsoft and now stewarded by the guidance-ai organization, that gives developers fine-grained control over text generation from large language models. Instead of issuing a single prompt and parsing the result, Guidance interleaves generation with control structures — loops, conditionals, regex constraints, and selections — so the model is steered to produce exactly the structure required. This constrained-decoding approach reduces wasted tokens, improves reliability for structured output, and works across both local and hosted models.
LLM Frameworks
⭐ 21,484 stars
python constrained-generation microsoft open-source
Structured text generation library — constrain LLM outputs to regex, JSON Schema, and grammars.
Outlines, developed by the team at .txt (dottxt), is a Python library for guaranteed structured generation from language models. It compiles regular expressions, JSON Schemas, Pydantic models, and context-free grammars into finite-state machines that mask the model's token logits during decoding, so the output is forced to conform to the requested format without retries or parsing failures. Because the constraints operate at the sampling level, Outlines adds negligible overhead and works with local open-weight models served through vLLM, transformers, and similar backends.
LLM Frameworks
⭐ 13,905 stars
python structured-generation regex json-schema
Get reliable, type-safe structured outputs from LLMs using Pydantic models.
Instructor, created by Jason Liu, is a popular Python library that makes structured extraction from large language models simple and reliable by building on Pydantic. Developers define the desired schema as a Pydantic model, and Instructor patches the LLM client so responses are validated, coerced into typed objects, and automatically re-asked when validation fails. It works on top of function calling and JSON modes across providers including OpenAI, Anthropic, and Gemini, and has been ported to TypeScript, Go, Ruby, and other languages, making it a de facto standard for data extraction.
LLM Frameworks
⭐ 13,069 stars
python structured-output pydantic open-source
ML experiment tracking and LLM observability — W&B Weave for trace logging, evals, and datasets.
Weights & Biases is a well-established MLOps platform whose Weave product extends its experiment-tracking heritage into the LLM era. Weave automatically logs traces of LLM and agent calls, captures inputs, outputs, latencies, and costs, and provides a framework for running and comparing evaluations against datasets. Engineering teams use it to debug non-deterministic AI behavior, version prompts and models, and build a feedback loop between production data and offline testing. W&B is widely deployed across research labs and enterprises building generative AI features.
Observability
⭐ 11,093 stars
mlops experiment-tracking eval fine-tuning
Open-source LLM observability — traces, evals, and a prompt playground built on OpenTelemetry.
Phoenix is the open-source LLM observability and evaluation platform from Arize AI. It captures traces of LLM, retrieval, and agent calls using the OpenInference standard built on OpenTelemetry, then surfaces them in an interactive UI for debugging RAG pipelines, spotting hallucinations, and analyzing token usage. Phoenix bundles a library of LLM-as-a-judge evaluators and a prompt playground so teams can iterate without leaving the tool, and it can run locally in a notebook or be self-hosted, complementing Arize's enterprise monitoring product.
Observability
⭐ 9,918 stars
observability tracing evals opentelemetry
Open-source framework for adding input/output guardrails and validators to LLM applications.
Guardrails AI is an open-source Python framework for adding programmable safety and reliability checks around large language model inputs and outputs. Developers attach validators — drawn from the community Guardrails Hub — to detect and remediate issues such as PII leakage, toxic language, hallucinated facts, off-topic responses, and malformed structure, with the option to re-ask, fix, or block on failure. By treating validation as a composable layer rather than ad-hoc prompt instructions, it gives teams enforceable controls for deploying LLMs in regulated and customer-facing settings.
Guardrails
⭐ 6,944 stars
guardrails validation python open-source
Open-source LLM observability — one-line proxy integration for logging, caching, and cost tracking.
Helicone is an open-source LLM observability platform that integrates through a single line of configuration by routing requests through its proxy or async logging SDK. Once connected, it captures every request and response with full token counts, latency, and cost, and adds production features such as response caching, rate limiting, retries, and user-level analytics. Helicone's gateway model makes it provider-agnostic, and teams use it to monitor spend, debug prompts, and segment usage by customer. It is available both as a hosted service and a self-hosted deployment.
Observability
⭐ 5,755 stars
observability logging caching open-source
Enterprise AI evaluation and observability — run evals, trace LLM calls, and monitor production quality.
Braintrust is an end-to-end platform for evaluating and shipping AI products, used by engineering teams to bring rigor to non-deterministic LLM behavior. It centers on evals — scoring model outputs against datasets with code-based or LLM-as-a-judge scorers — alongside logging and tracing of production calls so regressions are caught before release. Its Playground lets engineers and domain experts compare prompts and models side by side, and the Loop assistant helps author scorers. Braintrust is offered as a hosted product with options for on-premise and hybrid deployment.
Observability
eval observability tracing enterprise
Unified API for 300+ AI models from one endpoint — route by price, speed, or quality with automatic fallbacks.
OpenRouter is a hosted inference marketplace that exposes hundreds of models from dozens of providers behind a single OpenAI-compatible API endpoint. Developers send one request and OpenRouter handles authentication, billing, and routing — automatically falling back to alternate providers when one is down and letting users prioritize by price, latency, or throughput. It aggregates models from OpenAI, Anthropic, Google, Meta, Mistral, and many open-weight hosts, provides unified usage analytics, and removes the need to hold separate accounts and keys with each model vendor.
Routing & Gateways
proxy multi-model api routing
AI gateway with routing, caching, observability, and guardrails — drop-in middleware for LLM API calls.
Portkey is an AI gateway and control panel that sits between applications and language model providers. Its open-source gateway routes traffic across 250-plus models with load balancing, conditional routing, semantic caching, automatic retries, and fallbacks, while the observability suite logs every request with cost, latency, and tracing. Portkey also layers in guardrails, prompt management, and budget controls, giving platform teams a single place to enforce reliability and governance policies. It is available as a hosted service and as a self-hostable, enterprise-deployable component.
Routing & Gateways
proxy multi-model observability caching
Prompt management and LLM request logging — version prompts, track costs, and replay requests.
PromptLayer is a prompt engineering and management platform that decouples prompts from application code so non-engineers can iterate on them. It maintains a versioned registry where prompts can be edited, A/B tested, and deployed without redeploying the app, and it logs every LLM request for cost tracking, search, and replay. Teams use its evaluation pipelines and visual editor to collaborate across product, engineering, and subject-matter experts. PromptLayer integrates with major model providers and frameworks, acting as the system of record for an organization's prompts.
Prompt Management
prompt-management logging versioning collaboration
Low-level orchestration framework for building stateful, multi-actor LLM agents as graphs.
LangGraph is a low-level orchestration framework from LangChain, Inc. for building stateful, controllable agents. It models agent logic as a graph of nodes and edges with built-in persistence, so workflows can include cycles, branching, human-in-the-loop checkpoints, and durable memory that survives interruptions. Unlike higher-level abstractions, it gives developers explicit control over each step, making it suitable for long-running and multi-agent systems. The companion LangGraph Platform offers managed deployment, and it integrates tightly with LangSmith for tracing and evaluation.
LLM Frameworks
python javascript agents orchestration
Observability and evaluation platform for LLM applications, framework-agnostic and built by LangChain.
LangSmith is the commercial observability and evaluation platform built by LangChain, Inc., though it works with any stack regardless of whether LangChain is used. It captures detailed traces of LLM, retrieval, and tool calls so developers can inspect the full execution path of agents, then layers on dataset-driven evaluations, online scoring of production traffic, and prompt management with versioning. By unifying debugging, testing, and monitoring in one workspace, LangSmith helps teams move LLM features from prototype to reliable production deployment.
Observability
observability tracing evals prompt-management
Open-source orchestration framework by deepset for production RAG and agent pipelines.
Haystack is an open-source AI orchestration framework created by deepset and first released in 2020. It lets Python developers assemble modular, serializable pipelines from components — retrievers, rankers, prompt builders, generators, and tools — to build production-grade retrieval-augmented generation, semantic search, and agentic applications. Pipelines are cloud-agnostic and Kubernetes-ready with logging and monitoring hooks, and the framework supports hybrid retrieval and self-correction loops. deepset also offers a commercial enterprise platform and Studio on top of the open-source core for teams scaling AI systems.
LLM Frameworks
python rag agents orchestration
Open-source LLM engineering platform — observability, evals, prompt management, and a playground.
Langfuse is an open-source LLM engineering platform (YC W23) that brings observability, evaluation, prompt management, and a testing playground into one self-hostable tool. It ingests traces via OpenTelemetry and integrates with LangChain, the OpenAI SDK, LiteLLM, and others, so teams can debug agent runs, track token cost and latency, and manage versioned prompts with low-latency caching. Its evaluation features support LLM-as-a-judge scoring and dataset experiments. Langfuse offers a generous free tier on its managed cloud and full source for on-premise deployment.
Observability
observability tracing evals prompt-management
Microsoft's open-source SDK for integrating LLMs into apps across .NET, Python, and Java.
Semantic Kernel is Microsoft's open-source SDK for embedding large language models into enterprise applications, with first-class support across .NET, Python, and Java. It introduces plugins, planners, and a memory abstraction that let developers orchestrate AI functions alongside native code, and it provides connectors to OpenAI, Azure OpenAI, and Hugging Face models. Production-ready since its 1.0 release, it is backed by Microsoft and is converging with AutoGen into the unified Microsoft Agent Framework, making it a common choice for organizations standardizing on the Microsoft stack.
LLM Frameworks
dotnet python java microsoft
Microsoft Research framework for building event-driven, multi-agent LLM applications.
AutoGen is an open-source framework from Microsoft Research's AI Frontiers Lab for building multi-agent LLM applications. It models solutions as conversations between specialized agents — such as an assistant and a code executor — that collaborate, call tools, and hand off tasks under an event-driven, distributed runtime. AutoGen supports multiple models and human-in-the-loop participation, and ships AutoGen Studio, a low-code interface for prototyping agent teams. The project is converging with Semantic Kernel into Microsoft's unified Agent Framework and is widely used in agent research and experimentation.
LLM Frameworks
python dotnet microsoft multi-agent
Standalone Python framework for orchestrating role-playing, collaborative AI agent crews.
CrewAI is a lean, standalone Python framework for orchestrating teams of autonomous AI agents. Built independently of LangChain, it lets developers define agents with roles, goals, and backstories, then coordinate them as a "crew" that delegates tasks and shares context, or as deterministic "flows" for event-driven control. It emphasizes speed and simplicity for production use, supports a wide range of model providers and tools, and is complemented by CrewAI Enterprise for deployment, monitoring, and management. It has become one of the most popular multi-agent frameworks.
LLM Frameworks
python multi-agent orchestration open-source
High-throughput, memory-efficient inference and serving engine for LLMs.
vLLM is a high-throughput, memory-efficient inference and serving engine for large language models, originally developed at UC Berkeley's Sky Computing Lab and now a community-driven project. Its core innovation, PagedAttention, manages the attention key-value cache like virtual memory, dramatically improving GPU utilization and enabling continuous batching of requests. vLLM serves open-weight models through an OpenAI-compatible API, supports quantization, tensor parallelism, and speculative decoding, and has become the de facto open-source backend for self-hosting LLMs at scale in production environments.
Routing & Gateways
inference serving open-source openai-compatible
Run open-weight LLMs locally with a single command and an OpenAI-compatible API.
Ollama is an open-source tool that makes running open-weight large language models on local hardware as simple as a single command. It packages model weights, configuration, and a runtime into portable bundles, manages downloads from its model library, and exposes both a CLI and a local REST API with an OpenAI-compatible endpoint so existing tools can point at it unchanged. Built on the llama.cpp engine, Ollama runs on macOS, Linux, and Windows, and is widely used by developers for private, offline experimentation and local-first application development.
Routing & Gateways
local inference open-source openai-compatible
Lightweight, provider-agnostic Python toolkit for building with LLMs — the "LLM anti-framework."
Mirascope is a lightweight Python toolkit that bills itself as the "LLM anti-framework," favoring small, composable building blocks over heavy abstractions. Using simple decorators, developers write prompts, tools, structured outputs, streaming, and async calls that work across a single unified interface for providers including OpenAI, Anthropic, Mistral, Gemini, Groq, and Vertex AI. It is type-safe and leans on Pydantic for validation, integrates with observability tools such as Logfire and Langfuse, and supports MCP workflows, giving engineers low-level control without locking them into a monolithic framework.
LLM Frameworks
python structured-output provider-agnostic open-source
Observability and monitoring SDK purpose-built for AI agents and multi-agent systems.
AgentOps is a developer platform focused specifically on observability for AI agents, where conventional LLM logging falls short. Its Python SDK instruments agent runs to capture step-by-step session replays, tool calls, token costs, latency, and errors, with visual dashboards for debugging complex multi-agent behavior. It integrates out of the box with frameworks including CrewAI, AutoGen, the OpenAI Agents SDK, LangChain, and Camel, and tracks more than 400 LLMs. Teams use AgentOps to benchmark, monitor, and control the cost of agentic applications as they move into production.
Observability
observability agents cost-tracking open-source
NVIDIA's open-source toolkit for adding programmable rails to conversational LLM systems.
NeMo Guardrails is an open-source toolkit from NVIDIA for adding programmable safety and topical rails to LLM-based conversational applications. Developers define rails using Colang, a modeling language for conversation flows, to control what the assistant may discuss, block jailbreaks and toxic content, enforce factual grounding, and gate access to third-party tools and actions. The rails run as a layer between the user and the model, can be combined with community and third-party checks, and let enterprises deploy chatbots and assistants with predictable, governed behavior.
Guardrails
guardrails nvidia safety python
Agent framework from the Pydantic team bringing type-safe, validated outputs to LLM apps.
Pydantic AI is an agent framework built by the team behind Pydantic, the validation library used inside most Python LLM SDKs. It brings the same type-safe, ergonomic developer experience to building agents, with structured and validated outputs, dependency injection, tool calling, and streaming, all defined in plain Python. It is model-agnostic across OpenAI, Anthropic, Gemini, and others, and integrates with Pydantic Logfire for observability. By leaning on familiar Pydantic patterns, it lowers the barrier to building reliable, production-grade agentic applications.
LLM Frameworks
python agents structured-output type-safe
Open-source OpenTelemetry-based observability SDK for LLM and agent applications.
OpenLLMetry is an open-source set of extensions to OpenTelemetry, built by Traceloop, that adds standardized instrumentation for LLM and vector-database calls. Because it builds on the OpenTelemetry standard, traces flow into any compatible backend — Datadog, Grafana, Honeycomb, or Traceloop's own platform — without locking teams into a proprietary tool. It automatically instruments popular providers and frameworks such as OpenAI, Anthropic, LangChain, and LlamaIndex, capturing prompts, completions, costs, and latency so engineers can monitor and debug AI systems alongside the rest of their infrastructure.
Observability
observability opentelemetry tracing open-source
Open-source CLI and library for testing, evaluating, and red-teaming LLM prompts and apps.
Promptfoo is an open-source, developer-first tool for testing and evaluating LLM applications from the command line or CI. Engineers declare test cases and assertions in simple config files, then run side-by-side comparisons across prompts, models, and providers to catch regressions before shipping. Beyond accuracy evals, Promptfoo includes a security and red-teaming suite that probes applications for prompt injection, jailbreaks, and data leakage. It runs locally without sending data to a third party, integrates into automated pipelines, and is widely adopted for systematic prompt quality assurance.
Observability
eval testing red-teaming open-source
AI gateway built on Kong's API platform for routing, securing, and governing LLM traffic.
Kong AI Gateway extends Kong's widely deployed open-source API gateway with a set of plugins purpose-built for large language model traffic. It gives platform teams a single control point to route requests across multiple model providers, normalize their APIs, enforce semantic caching, apply rate limiting and token-based cost controls, and add prompt guarding and PII redaction. Because it builds on the proven Kong proxy, it slots into existing API management and observability workflows, making it a natural choice for enterprises governing LLM usage at scale.
Routing & Gateways
gateway proxy multi-model enterprise
Open-source LLMOps platform for prompt management, evaluation, and observability.
Agenta is an open-source, end-to-end LLMOps platform that unifies prompt engineering, evaluation, and observability in a single workbench. Its prompt playground lets engineers and domain experts compare prompts and model configurations side by side and deploy versioned prompts via a registry without code changes. Agenta's evaluation tools support automated and human scoring against datasets, while OpenTelemetry-based tracing captures production behavior. It integrates with major model providers and frameworks and can be self-hosted, giving teams a collaborative, vendor-neutral hub for shipping reliable LLM applications.
Prompt Management
prompt-management eval observability open-source