realm_flutter_vector_db 1.0.13
realm_flutter_vector_db: ^1.0.13 copied to clipboard
Realm Dart SDK with built-in HNSW vector search - a mobile database with AI/ML vector similarity search capabilities for semantic search.
Note
π Atlas Device Sync Replacement Available!
MongoDB deprecated Atlas Device Sync in September 2024, but we've got you covered!
Introducing Flutter Realm Sync - the open-source, production-ready successor that provides:
- β Real-time bidirectional sync with MongoDB Atlas
- β Offline-first architecture with automatic conflict resolution
- β Self-hosted control - no vendor lock-in
- β Production-ready server included (Node.js + TypeScript)
- β Battle-tested with 1000s of documents in real apps
π Get Started with Flutter Realm Sync β
π¦ GitHub Repository β
Realm is a mobile database that runs directly inside phones, tablets or wearables. This repository holds the source code for the Realm SDK for Flutterβ’ and Dartβ’.
Features #
- Mobile-first: Realm is the first database built from the ground up to run directly inside phones, tablets, and wearables.
- Simple: Realm's object-oriented data model is simple to learn, doesn't need an ORM, and the API lets you write less code to get apps up & running in minutes.
- Modern: Realm supports latest Dart and Flutter versions and is built with sound null-safety.
- Fast: Realm is faster than even raw SQLite on common operations while maintaining an extremely rich feature set.
- Vector Search (HNSW): Built-in support for high-performance vector similarity search using Hierarchical Navigable Small World (HNSW) algorithm. Perfect for AI/ML applications, semantic search, recommendation systems, and RAG (Retrieval-Augmented Generation) patterns.
- π Flutter Realm Sync: Open-source, production-ready replacement for deprecated Atlas Device Sync. Real-time bidirectional sync with MongoDB Atlas, offline-first architecture, automatic conflict resolution, and self-hosted control. Includes a complete Node.js server and is battle-tested in production apps. Get Started β
Getting Started #
-
Import Realm in a dart file
app.dartimport 'package:realm/realm.dart'; // import realm package part 'app.realm.dart'; // declare a part file. @RealmModel() // define a data model class named `_Car`. class _Car { late String make; late String model; int? kilometers = 500; } -
Generate RealmObject class
Carfrom data model class_Car.dart run realm_flutter_vector_db generate -
Open a Realm and add some objects.
var config = Configuration.local([Car.schema]); var realm = Realm(config); var car = Car("Tesla", "Model Y", kilometers: 5); realm.write(() { realm.add(car); }); -
Query objects in Realm.
var cars = realm.all<Car>(); Car myCar = cars[0]; print("My car is ${myCar.make} model ${myCar.model}"); cars = realm.all<Car>().query("make == 'Tesla'"); -
Get stream of result changes for a query.
final cars = realm.all<Car>().query(r'make == $0', ['Tesla']); cars.changes.listen((changes) { print('Inserted indexes: ${changes.inserted}'); print('Deleted indexes: ${changes.deleted}'); print('Modified indexes: ${changes.modified}'); }); realm.write(() => realm.add(Car('VW', 'Polo', kilometers: 22000)));
Vector Search with HNSW #
Realm now supports high-performance vector similarity search using the Hierarchical Navigable Small World (HNSW) algorithm. This enables AI/ML applications including semantic search, recommendation systems, image similarity, and RAG (Retrieval-Augmented Generation) patterns.
Quick Start with Vector Search #
-
Define a model with vector embeddings:
import 'package:realm/realm.dart'; part 'app.realm.dart'; @RealmModel() class _Document { @PrimaryKey() late String id; late String title; late String content; late List<double> embedding; // Vector embeddings } -
Generate the RealmObject class:
dart run realm_flutter_vector_db generate -
Create a vector index and perform similarity search:
var config = Configuration.local([Document.schema]); var realm = Realm(config); // Create HNSW vector index realm.write(() { realm.createVectorIndex<Document>( 'embedding', metric: VectorDistanceMetric.cosine, // or euclidean, dotProduct m: 16, // connections per layer (default: 16) efConstruction: 200, // build quality (default: 200) ); }); // Add documents with embeddings realm.write(() { realm.add(Document( '1', 'AI Technology', 'Machine learning and neural networks', embedding: [0.95, 0.85, 0.05, 0.10, 0.02, 0.08], )); realm.add(Document( '2', 'Nature Guide', 'Forest ecosystems and wildlife', embedding: [0.08, 0.12, 0.95, 0.88, 0.02, 0.05], )); }); // K-Nearest Neighbors (KNN) search final queryVector = [0.9, 0.8, 0.1, 0.1, 0.05, 0.05]; final results = realm.vectorSearchKnn<Document>( 'embedding', queryVector: queryVector, k: 5, // Return top 5 similar documents ); for (var result in results) { print('${result.object.title}: distance=${result.distance}'); } // Radius search (all documents within distance threshold) final radiusResults = realm.vectorSearchRadius<Document>( 'embedding', queryVector: queryVector, maxDistance: 0.5, );
Vector Search Features #
-
Distance Metrics:
VectorDistanceMetric.cosine- Cosine similarity (recommended for normalized vectors)VectorDistanceMetric.euclidean- Euclidean distanceVectorDistanceMetric.dotProduct- Dot product similarity
-
Search Types:
- KNN Search: Find K nearest neighbors (
vectorSearchKnn) - Radius Search: Find all vectors within distance threshold (
vectorSearchRadius)
- KNN Search: Find K nearest neighbors (
-
Index Management:
createVectorIndex()- Create HNSW index on vector propertyremoveVectorIndex()- Remove index (preserves data)hasVectorIndex()- Check if index existsgetVectorIndexStats()- Get index statistics (numVectors, maxLayer)
-
Tuning Parameters:
m(default: 16) - Number of bi-directional links per node. Higher values = better recall, more memoryefConstruction(default: 200) - Build-time quality parameter. Higher values = better index quality, slower indexing
Production Migration Pattern #
When changing vector dimensions (e.g., 4D β 6D), use this safe migration pattern:
realm.write(() {
// 1. Remove existing index (data is preserved!)
if (realm.hasVectorIndex<Document>('embedding')) {
realm.removeVectorIndex<Document>('embedding');
}
// 2. Transform embeddings
for (final doc in realm.all<Document>()) {
final oldValues = List<double>.from(doc.embedding); // Create defensive copy
final newValues = [...oldValues, 0.0, 0.0]; // Add new dimensions
doc.embedding.clear();
doc.embedding.addAll(newValues);
}
// 3. Create new index with updated dimensions
realm.createVectorIndex<Document>(
'embedding',
metric: VectorDistanceMetric.cosine,
m: 16,
efConstruction: 200,
);
});
Key points:
removeVectorIndex()does NOT delete your data- Always create a defensive copy with
List<double>.from()before modifying - This pattern avoids data loss unlike
shouldDeleteIfMigrationNeeded: true
Performance Benchmarks #
Benchmark results with 100 queries (1024-dimensional embeddings):
| Metric | Performance |
|---|---|
| Bulk Insert | 0.90ms per record |
| Index Creation | 125ms (m=16, efConstruction=200) |
| KNN Search (Cold Start) | 2,016ΞΌs |
| KNN Search (Warm) | ~102ΞΌs (9,766 searches/sec) |
| Radius Search | 104-959ΞΌs |
| Filtered Search | 162-629ΞΌs |
| Memory Overhead | ~100% (index size β data size) |
Distance Metrics Comparison (all perform similarly):
- Cosine: 190ms index creation, 155ΞΌs search
- Euclidean: 183ms index creation, 152ΞΌs search
- Dot Product: 178ms index creation, 157ΞΌs search
Parameter Tuning Impact:
- m=8, efConstruction=100: 118ΞΌs search
- m=16, efConstruction=200: 112ΞΌs search
- m=32, efConstruction=400: 104ΞΌs search (fastest)
Higher HNSW parameters yield better search performance at the cost of slightly larger index size and longer index creation time.
Use Cases #
- Semantic Search: Find documents by meaning, not just keywords
- Recommendation Systems: Suggest similar items based on embeddings
- Image Similarity: Find visually similar images using vision model embeddings
- RAG Applications: Retrieve relevant context for AI chatbots and assistants
- Duplicate Detection: Find near-duplicate content
- Clustering & Classification: Group similar items together
For a complete example with 26 comprehensive tests, see example/lib/main.dart. Performance benchmarks are available in the test suite.
Samples #
For complete samples check the Realm Flutter and Dart Samples.
Documentation #
For API documentation go to
Use realm package for Flutter and realm_dart package for Dart applications.
For complete documentation of the SDKs, go to the Realm SDK documentation.
If you are using the Realm SDK for the first time, refer to the Quick Start documentation.
To learn more about using Realm with Atlas App Services and Device Sync, refer to the following Realm SDK documentation:
Realm Flutter SDK #
Realm Flutter package is published to realm.
Environment setup for Realm Flutter #
- Realm Flutter supports the platforms iOS, Android, Windows, MacOS and Linux.
- Flutter 3.10.2 or newer.
- For Flutter Desktop environment setup, see Desktop support for Flutter.
- Cocoapods v1.11 or newer.
- CMake 3.21 or newer.
Usage #
The full contents of catalog.dart is listed after the usage
-
Add
realmpackage to a Flutter application.flutter pub add realm_flutter_vector_db -
For running Flutter widget and unit tests run the following command to install the required native binaries.
dart run realm_flutter_vector_db install -
Import Realm in a dart file (ex.
catalog.dart).import 'package:realm/realm.dart'; -
Declare a part file
catalog.realm.dartin the begining of thecatalog.dartdart file after all imports.import 'dart:io'; part 'catalog.realm.dart'; -
Create a data model class.
It should start with an underscore
_Itemand be annotated with@RealmModel()@RealmModel() class _Item { @PrimaryKey() late int id; late String name; int price = 42; } -
Generate RealmObject class
Itemfrom data model class_Item.On Flutter use
dart run realm_flutter_vector_dbto runrealm_flutter_vector_dbpackage commandsdart run realm_flutter_vector_db generateA new file
catalog.realm.dartwill be created next to thecatalog.dart.*The generated file should be committed to source control
-
Use the RealmObject class
Itemwith Realm.// Create a Configuration object var config = Configuration.local([Item.schema]); // Opean a Realm var realm = Realm(config); var myItem = Item(0, 'Pen', price: 4); // Open a write transaction realm.write(() { realm.add(myItem); var item = realm.add(Item(1, 'Pencil')..price = 20); }); // Objects `myItem` and `item` are now managed and persisted in the realm // Read object properties from realm print(myItem.name); print(myItem.price); // Update object properties realm.write(() { myItem.price = 20; myItem.name = "Special Pencil"; }); // Get objects from the realm // Get all objects of type var items = realm.all<Item>(); // Get object by index var item = items[1]; // Get object by primary key var itemByKey = realm.find<Item>(0); // Filter and sort object var objects = realm.query<Item>("name == 'Special Pencil'"); var name = 'Pen'; objects = realm.query<Item>(r'name == $0', [name]); // Close the realm realm.close();
Full contents of catalog.dart #
import 'package:realm/realm.dart';
part 'catalog.realm.dart';
@RealmModel()
class _Item {
@PrimaryKey()
late int id;
late String name;
int price = 42;
}
// Create a Configuration object
var config = Configuration.local([Item.schema]);
// Open a Realm
var realm = Realm(config);
var myItem = Item(0, 'Pen', price: 4);
// Open a write transaction
realm.write(() {
realm.add(myItem);
var item = realm.add(Item(1, 'Pencil')..price = 20);
});
// Objects `myItem` and `item` are now managed and persisted in the realm
// Read object properties from realm
print(myItem.name);
print(myItem.price);
// Update object properties
realm.write(() {
myItem.price = 20;
myItem.name = "Special Pencil";
});
// Get objects from the realm
// Get all objects of type
var items = realm.all<Item>();
// Get object by index
var item = items[1];
// Get object by primary key
var itemByKey = realm.find<Item>(0);
// Filter and sort object
var objects = realm.query<Item>("name == 'Special Pencil'");
var name = 'Pen';
objects = realm.query<Item>(r'name == $0', [name]);
// Close the realm
realm.close();
Realm Dart Standalone SDK #
Realm Dart package is published to realm_dart.
Environment setup for Realm Dart #
- Realm Dart supports the platforms Windows, Mac and Linux.
- Dart SDK 3.0.2 or newer.
Usage #
-
Add
realm_dartpackage to a Dart application.dart pub add realm_dart -
Install the
realm_dartpackage into the application. This downloads and copies the required native binaries to the app directory.dart run realm_dart install -
Import realm_dart in a dart file (ex.
catalog.dart).import 'package:realm_dart/realm.dart'; -
To generate RealmObject classes with realm_dart use this command.
On Dart use
dart run realm_dartto runrealm_dartpackage commandsdart run realm_dart generateA new file
catalog.realm.dartwill be created next to thecatalog.dart.*The generated file should be committed to source control
-
The usage of the Realm Dart SDK is the same like the Realm Flutter above.
Sync Realm Data with MongoDB Atlas using Flutter Realm Sync #
The Open-Source Replacement for Atlas Device Sync
With Atlas Device Sync deprecated, Flutter Realm Sync is the community-driven, production-ready solution for real-time bidirectional sync between Realm databases and MongoDB Atlas.
Why Flutter Realm Sync? #
| Feature | Atlas Device Sync (Deprecated) |
Flutter Realm Sync (Active & Open Source) |
|---|---|---|
| Real-time Sync | βοΈ | βοΈ Socket.IO powered |
| Offline-First | βοΈ | βοΈ Native Realm integration |
| Open Source | β Closed | βοΈ MIT License |
| Self-Hosted | β | βοΈ Full control |
| Production Ready | β Deprecated | βοΈ Battle-tested |
| Active Development | β | βοΈ Community-driven |
Quick Start Guide #
Step 1: Add Flutter Realm Sync #
dependencies:
realm_flutter_vector_db: ^1.0.11
flutter_realm_sync: ^0.0.1
socket_io_client: ^3.1.2
flutter pub get
Step 2: Define Your Realm Model #
Add required sync fields to your Realm models:
import 'package:realm_flutter_vector_db/realm_vector_db.dart';
part 'models.realm.dart';
@RealmModel()
@MapTo('tasks')
class _Task {
@PrimaryKey()
@MapTo('_id')
late String id;
late String title;
late String status;
late int progressMinutes;
// Required for sync functionality
@MapTo('sync_updated_at')
int? syncUpdatedAt;
@MapTo('sync_update_db')
bool syncUpdateDb = false;
}
Generate the Realm schema:
dart run realm_flutter_vector_db generate
Step 3: Initialize Realm with Sync Models #
import 'package:realm_flutter_vector_db/realm_vector_db.dart';
import 'package:flutter_realm_sync/services/Models/sync_metadata.dart';
import 'package:flutter_realm_sync/services/Models/sync_db_cache.dart';
import 'package:flutter_realm_sync/services/Models/sync_outbox_patch.dart';
// Configure Realm with your models + sync models
final config = Configuration.local([
Task.schema,
SyncMetadata.schema, // Required for sync state
SyncDBCache.schema, // Required for sync caching
SyncOutboxPatch.schema, // Required for sync operations
], schemaVersion: 1);
final realm = Realm(config);
Step 4: Set Up the Sync Server #
Complete backend included! Get the production-ready Node.js server:
git clone https://github.com/mohit67890/realm-sync-server.git
cd realm-sync-server
npm install
# Add your MongoDB Atlas URI to .env
echo "MONGODB_URI=your_mongodb_connection_string" > .env
# Start the sync server
npm run dev # Development mode
npm start # Production mode
The server provides:
- β Socket.IO with room-based isolation
- β MongoDB Atlas integration
- β Automatic change broadcasting
- β Historic sync for offline catch-up
- β Deploy to AWS/GCP/Heroku/DigitalOcean
Server Repository: https://github.com/mohit67890/realm-sync-server
Step 5: Connect to Sync Server #
import 'package:socket_io_client/socket_io_client.dart' as IO;
import 'package:flutter_realm_sync/services/RealmSync.dart';
// Connect to your sync server
final socket = IO.io(
'http://your-server-url:3000',
IO.OptionBuilder()
.setTransports(['websocket'])
.disableAutoConnect()
.build(),
);
socket.onConnect((_) {
print('β
Connected to sync server');
// Join sync room
socket.emitWithAck('sync:join', {'userId': 'user-123'}, ack: (data) {
if (data['success'] == true) {
print('Joined sync room successfully');
}
});
});
socket.connect();
Step 6: Initialize RealmSync #
final realmSync = RealmSync(
realm: realm,
socket: socket,
userId: 'user-123',
configs: [
SyncCollectionConfig<Task>(
collectionName: 'tasks',
results: realm.all<Task>(),
idSelector: (obj) => obj.id,
needsSync: (obj) => obj.syncUpdateDb,
fromServerMap: (map) {
return Task(
map['_id'] as String,
map['title'] as String,
map['status'] as String,
map['progressMinutes'] as int,
syncUpdatedAt: map['sync_updated_at'] as int?,
);
},
),
],
);
// Start real-time sync
realmSync.start();
// Fetch historic changes (data while offline)
realmSync.fetchAllHistoricChanges(applyLocally: true);
Step 7: Write and Sync Data #
import 'package:flutter_realm_sync/services/RealmHelpers/realm_sync_extensions.dart';
// Create and sync a task in one call
final task = Task(
ObjectId().toString(),
'Complete project',
'in_progress',
100,
);
realm.writeWithSync(task, () {
task.syncUpdateDb = true;
realm.add(task);
});
// Sync to MongoDB Atlas and all connected devices
realmSync.syncObject('tasks', task.id);
// Update existing task
realm.writeWithSync(task, () {
task.status = 'completed';
task.progressMinutes = 120;
});
realmSync.syncObject('tasks', task.id);
That's it! Your app now has:
- β Real-time bidirectional sync with MongoDB Atlas
- β Offline-first architecture with automatic conflict resolution
- β Multi-device sync across iOS, Android, macOS, Windows, Linux
- β Automatic reconnection and historic sync
Key Features #
π Bidirectional Real-Time Sync #
Changes flow seamlessly: Device βοΈ MongoDB Atlas βοΈ All Devices
πΎ Offline-First Architecture #
Write locally, sync automatically when online. Zero data loss.
β‘ Intelligent Batching #
Bulk operations with smart debouncing for optimal performance.
π― Automatic Conflict Resolution #
Last-write-wins with millisecond-precision timestamps.
π Production-Ready Server #
Complete Node.js + TypeScript backend included. Deploy anywhere.
π¨ Fully Customizable #
Pre-processors, custom serializers, your business logic.
π Battle-Tested #
Powers production apps with 10,000+ documents, <100ms sync latency.
Advanced Features #
Listen to Sync Events #
final subscription = realmSync.objectChanges.listen((event) {
print('Synced ${event.collectionName}: ${event.id}');
// Access the synced object
print('Object: ${event.object}');
});
// Cancel when done
subscription.cancel();
Custom Pre-Processing #
Modify data before sending to server:
SyncCollectionConfig<Task>(
// ... other config ...
emitPreProcessor: (rawJson) {
// Add metadata
rawJson['clientVersion'] = '2.1.0';
rawJson['deviceId'] = DeviceInfo.id;
rawJson['timestamp'] = DateTime.now().toIso8601String();
return rawJson;
},
)
Historic Sync for Offline Catch-Up #
// Fetch all changes since last sync
realmSync.fetchAllHistoricChanges(applyLocally: true);
// Manual fetch for specific collection
socket.emitWithAck(
'sync:get_changes',
{
'userId': 'user-123',
'collectionName': 'tasks',
'since': lastSyncTimestamp,
},
ack: (response) {
// Process historic changes
},
);
Resources #
- π¦ Flutter Realm Sync Package: https://pub.dev/packages/flutter_realm_sync
- π» GitHub Repository: https://github.com/mohit67890/flutter_realm_sync
- π§ Sync Server Repository: https://github.com/mohit67890/realm-sync-server
- π Full Documentation: See package README for comprehensive guides
- π¬ Example Chat App: Production-ready demo with offline support
Migration from Atlas Device Sync #
Migrating from deprecated Atlas Device Sync? Flutter Realm Sync provides:
- Drop-in replacement - Similar API, familiar concepts
- Self-hosted control - Your infrastructure, your rules
- Cost savings - No vendor lock-in or surprise bills
- Active development - Community-driven with rapid updates
- Production support - Battle-tested with real apps
Get started today: https://pub.dev/packages/flutter_realm_sync
Building the source #
See CONTRIBUTING.md for instructions about building the source.
Code of Conduct #
This project adheres to the MongoDB Code of Conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to [email protected].
License #
Realm Flutter and Dart SDKs and Realm Core are published under the Apache License 2.0.