AI Tools & Frameworks

Machine Learning Frameworks

LLM Development Frameworks

LangChain / LangGraph

Comprehensive framework for building LLM applications with chains, agents, and memory. LangGraph adds stateful workflow orchestration for complex agent systems.

• Chain composition for multi-step reasoning
• Memory systems for conversation context
• Tool integration for function calling
• Vector store abstractions for RAG

LlamaIndex

Specialized framework for data ingestion and retrieval-augmented generation (RAG). Focuses on connecting LLMs to private data sources with intelligent indexing.

Semantic Kernel (Microsoft)

Enterprise-focused SDK for AI orchestration with first-class support for .NET and Python. Emphasizes planning, plugins, and integration with Microsoft ecosystem.

Haystack

End-to-end framework for building search systems and NLP pipelines. Strong focus on RAG, document processing, and question answering at scale.

Inference Servers and Deployment

Deployment Platforms

• Modal: Serverless GPU functions, pay-per-use pricing
• Replicate: API-first model hosting with simple scaling
• BentoML: Package and deploy ML models as APIs
• Ray Serve: Scalable model serving on Ray clusters

Orchestration & Agent Frameworks

Agent Frameworks

Vector Databases

Essential for RAG implementations and semantic search:

• Chroma: Lightweight, embedded vector store
• Pinecone: Managed vector database with low latency
• Weaviate: Open-source with hybrid search capabilities
• Qdrant: High-performance with filtering and payload support
• pgvector: PostgreSQL extension for vector operations — ideal if you already run Postgres (one fewer system)

💡 When NOT to use a framework: for a simple chat or a single RAG flow, plain OpenAI-client calls plus a vector-DB client are often simpler and far easier to debug than LangChain-style abstractions. Orchestration frameworks earn their complexity with multi-step pipelines, many data sources, or agent workflows — add them when the problem demands it, not by default. (Full stack map: the local LLM software stack.)

Fine-Tuning Toolchain

The layer this page most often gets asked about — adapting open models on your own hardware:

• PEFT / LoRA / QLoRA: the core technique family — train small adapter matrices on a frozen (optionally 4-bit) base; a 70B fine-tunes on a single 48GB GPU.
• Axolotl: config-driven fine-tuning harness wrapping the best practices; the community default for serious runs.
• Unsloth: heavily optimized LoRA training — up to ~2× faster and lighter on VRAM; ideal on consumer cards.
• LLaMA-Factory: GUI-friendly alternative covering many model families and training modes.
• sentence-transformers: embedding models (and rerankers) for the retrieval side — self-hostable, a private assistant should not call a cloud embedding API.

When to fine-tune at all (vs RAG vs prompting) is its own decision — see RAG vs fine-tuning.

Monitoring & Observability

Production LLM deployments require specialized monitoring for costs, latency, and quality.

LLM Observability Tools

• LangSmith: Debugging and monitoring for LangChain applications
• Weights & Biases: Experiment tracking and model registry
• MLflow: End-to-end ML lifecycle management
• Arize AI: ML observability with drift detection
• Phoenix (Arize): Open-source LLM evaluation and tracing

Key Metrics to Track

Tool Selection Guide

By Use Case

💡 Start Simple: Begin with managed APIs (OpenAI) and LangChain. Move to self-hosted inference (Ollama, vLLM) as requirements crystallize. Add observability early to understand real-world performance.

Resources and Further Reading

On AI-Radar

• Latest framework news and updates
• Self-hosting tools and infrastructure
• LLM models and deployment

Recent Framework Articles