Why I Built ai-chat-toolkit (and What I Learned)

Every AI-powered product I have worked on involved writing the same code: a provider abstraction layer, a streaming handler, a history management system, and a tool-calling loop. The implementations looked different on the surface but solved identical problems underneath.

After the third time, I stopped and extracted the pattern into ai-chat-toolkit.

The problem it solves

When you integrate an LLM into a product, you face these recurring decisions:

1. How do I abstract over multiple providers so I can switch without rewriting?
2. How do I handle streaming in a way that works across HTTP, WebSocket, and SSE?
3. How do I manage conversation history without bloating the context window?
4. How do I implement tool calling in a way that is testable and maintainable?

Most tutorials give you a single-provider script. Most frameworks give you too much. I wanted something in between.

What the toolkit does

import { ChatEngine } from 'ai-chat-toolkit';

const engine = new ChatEngine({
  provider: 'anthropic',
  model: 'claude-sonnet-4-6',
  tools: [orderTool, productTool],
  maxHistory: 20,
});

const response = await engine.chat('What is the status of order #1234?');

Under the hood it handles the tool-call loop, trims history when it approaches the context limit, and normalises streaming events across providers.

What I learned building it

Streaming is the hard part. Handling partial JSON in tool call arguments across providers with different chunking behaviour took more time than everything else combined.

History management is under-discussed. Most examples ignore that context windows fill up. The toolkit implements a sliding window with configurable preservation of system messages and recent tool call pairs.

The tool-call loop has edge cases. What happens when a tool throws? When the model calls the same tool in an infinite loop? These cases need explicit handling.

Testing is the real unlock. Because the toolkit wraps provider calls, you can swap in a mock provider for tests.

The packages

• ai-chat-toolkit — core engine, provider abstraction, history management
• ai-chat-toolkit-server — Express adapter exposing a chat API
• ai-chat-toolkit-widget — React widget that connects to the server
• ai-chat-toolkit-rag — RAG extension for document-grounded answers

Where it is headed

The next milestone is a proper evaluation harness — the ability to run your chat system against a test dataset and score it. Evaluation is the gap between "it works in demos" and "I trust it in production".