toonX

TOON Format Diagram

Dart implementation of TOON (Token-Oriented Object Notation) - a compact, human-readable format for LLMs that uses ~40% fewer tokens than JSON while maintaining better accuracy.

🎯 What is TOON?

TOON combines YAML's indentation-based structure with CSV-style tabular arrays to create the most efficient format for passing structured data to Large Language Models. It's a lossless, drop-in replacement for JSON designed specifically for LLM inputs.

Think: JSON for machines, TOON for AI models.

📦 What This Package Supports

Core Functionality:

  • ✅ encode() - Convert Dart objects to TOON format
  • ✅ decode() - Parse TOON strings back to Dart objects
  • ✅ yamlToToon() - Convert YAML to TOON format
  • ✅ toonToYaml() - Convert TOON to YAML format
  • ✅ Lossless round-trip conversion with full JSON data model support

Encoding Options:

  • ✅ Custom delimiters - comma (,), tab (\t), or pipe (|) for arrays
  • ✅ Length markers - optional # prefix for array validation (e.g., tags[#3])
  • ✅ Custom indentation - configurable spaces per level (default: 2)
  • ✅ Tabular arrays - automatic CSV-style format for uniform objects
  • ✅ Flat map mode - flatten nested structures with custom separators
  • ✅ Smart quoting - minimal quoting, only when necessary

Decoding Options:

  • ✅ Strict mode - validates array lengths, delimiters, and syntax (default)
  • ✅ Lenient mode - best-effort parsing for malformed input
  • ✅ Unflatten support - reconstruct nested structures from flat maps
  • ✅ Escape sequence handling - proper handling of special characters

Additional Features:

  • ✅ CLI tool - command-line interface for JSON/YAML/TOON conversion
  • ✅ Comprehensive tests - 79+ tests including large datasets
  • ✅ Full documentation - detailed API docs for all functions
  • ✅ Type-safe - strongly typed Dart implementation

💡 Why TOON?

Token Efficiency

TOON significantly reduces token usage compared to other formats:

Format Tokens vs TOON
TOON 2,744 baseline
JSON (compact) 3,081 +12%
YAML 3,719 +36%
JSON 4,545 +66%
XML 5,167 +88%

LLM Accuracy

Benchmark results across 4 models on 209 data retrieval questions:

  • TOON: 73.9% accuracy
  • JSON: 69.7% accuracy
  • YAML: 69.0% accuracy
  • XML: 67.1% accuracy

Result: TOON achieves higher accuracy while using fewer tokens.

🚀 Quick Start

Installation

dependencies:
    toonx: ^1.0.0

Basic Usage

import 'package:toonx/toonx.dart';

final data = {
  'users': [
    {'id': 1, 'name': 'Alice', 'role': 'admin'},
    {'id': 2, 'name': 'Bob', 'role': 'user'},
  ]
};

// Encode to TOON
final toon = encode(data);
print(toon);
// users[2]{id,name,role}:
//   1,Alice,admin
//   2,Bob,user

// Decode back to Dart
final decoded = decode(toon);
print(decoded); // Original data restored

💡 Why TOON?

Token Efficiency

TOON significantly reduces token usage compared to other formats:

Format Tokens vs TOON
TOON 2,744 baseline
JSON (compact) 3,081 +12%
YAML 3,719 +36%
JSON 4,545 +66%
XML 5,167 +88%

LLM Accuracy

Benchmark results across 4 models on 209 data retrieval questions:

  • TOON: 73.9% accuracy
  • JSON: 69.7% accuracy
  • YAML: 69.0% accuracy
  • XML: 67.1% accuracy

Result: TOON achieves higher accuracy while using fewer tokens.

🎨 Advanced Features

Custom Delimiters

Use tabs or pipes for maximum efficiency:

// Tab delimiter - best for token efficiency
encode(data, options: EncodeOptions(delimiter: '\t'));

// Pipe delimiter - human-readable
encode(data, options: EncodeOptions(delimiter: '|'));

Flat Map

Flatten deeply nested structures:

final config = {
  'database': {'connection': {'host': 'localhost', 'port': 5432}}
};

// Flatten
final flat = encode(
  config,
  options: EncodeOptions(enforceFlatMap: true, flatMapSeparator: '.'),
);
// database.connection.host: localhost
// database.connection.port: 5432

// Unflatten on decode
final restored = decode(
  flat,
  options: DecodeOptions(enforceFlatMap: true, flatMapSeparator: '.'),
);

Strict vs Lenient Parsing

// Strict mode (default) - validates array lengths and structure
decode(toon, options: DecodeOptions(strict: true));

// Lenient mode - best-effort parsing
decode(toon, options: DecodeOptions(strict: false));

Length Markers

Explicit array size validation:

encode(data, options: EncodeOptions(lengthMarker: '#'));
// tags[#3]: admin,ops,dev

YAML Support

Convert between YAML and TOON formats:

// YAML → TOON
final yamlString = '''
users:
  - id: 1
    name: Alice
    role: admin
  - id: 2
    name: Bob
    role: user
''';

final toon = yamlToToon(yamlString);
// users[2]{id,name,role}:
//   1,Alice,admin
//   2,Bob,user

// TOON → YAML
final yaml = toonToYaml(toon);
print(yaml);

🔧 CLI Tool

Convert JSON, YAML, or TOON files via command line:

# JSON → TOON
dart run toonx input.json

# YAML → TOON
dart run toonx config.yaml

# TOON → JSON (default)
dart run toonx data.toon

# Encode with explicit command
dart run toonx encode input.json

# Decode with explicit command
dart run toonx decode data.toon

# Pipe from stdin
cat data.json | dart run toonx encode
echo '{"name": "Alice"}' | dart run toonx encode

# Show help
dart run toonx --help

Supported file formats:

  • .json → Auto-converts to TOON
  • .yaml, .yml → Auto-converts to TOON
  • .toon → Auto-converts to JSON

📚 When to Use TOON

Perfect for:

  • ✅ Uniform arrays of objects (e-commerce orders, user records, logs)
  • ✅ LLM prompts with structured data
  • ✅ API responses for AI models
  • ✅ Time-series data and analytics
  • ✅ Configuration files for AI systems

Consider alternatives:

  • ❌ Deeply nested, non-uniform structures (JSON may be better)
  • ❌ Pure tabular data (CSV is smaller but less structured)
  • ❌ Binary data or extremely large payloads

📖 Examples

See comprehensive examples in example/example.dart:

  1. Basic encoding and decoding
  2. Tabular arrays for uniform data
  3. Custom delimiters (tab, pipe)
  4. Length markers
  5. Flat map for nested structures
  6. Strict vs lenient parsing
  7. Real-world e-commerce example

🧪 Benchmarks

This package includes extensive tests with real-world data:

  • 100+ employee records
  • E-commerce orders with nested structures
  • GitHub repositories data
  • Analytics time-series
  • Configuration files

Run tests: dart test

🙏 Credits & Reference

This Dart implementation is based on the TOON (Token-Oriented Object Notation) format, an innovative data format designed for efficient LLM communication. Special thanks to the TOON format creators for their excellent work on this specification.

🤝 Contributing

Contributions are welcome! To contribute to this package:

  1. Fork this repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Make your changes and add tests
  4. Ensure all tests pass (dart test)
  5. Commit your changes (git commit -m 'Add amazing feature')
  6. Push to your branch (git push origin feature/amazing-feature)
  7. Open a Pull Request

Please report bugs and feature requests in our issue tracker.

Made with ❤️ for the Dart & Flutter community

Libraries

toonx