llamadart library

High-performance Dart and Flutter plugin for local LLM inference.

llamadart allows you to run Large Language Models (LLMs) locally with GGUF models through llama.cpp and .litertlm bundles through LiteRT-LM across native and web platforms.

Core Components

• LlamaEngine: The stateless engine for model loading, tokenization, and generation.
• ChatSession: A high-level, stateful interface for chat-based interactions. It automatically manages conversation history and context window limits.
• LlamaBackend: The platform-agnostic interface for inference.

Custom backends and their test fakes also import package:llamadart/backend.dart. The raw llama.cpp FFI is in package:llamadart/llama_cpp_bindings.dart, outside semantic versioning.

Simple Example

final engine = LlamaEngine(LlamaBackend());
await engine.loadModel('path/to/model.gguf'); // or model.litertlm

final session = ChatSession(engine);
await for (final chunk in session.create([LlamaTextContent('Hello!')])) {
  stdout.write(chunk.text);
}

final reply = await session.send('Say that again, shorter.');
print(reply.text);

await engine.dispose();