ModelParams class

Configuration parameters for loading a Llama model.

These parameters affect the initial model loading and context allocation. Most of these cannot be changed once the model is loaded.

Context batching fields in this class mirror llama.cpp semantics: n_batch is the logical max decode batch, and n_ubatch is the physical micro-batch size.

Example:

final params = ModelParams(
  contextSize: 4096,
  gpuLayers: 33, // Offload 33 layers to GPU
  splitMode: ModelSplitMode.none,
  mainGpu: 1, // Use the second GPU device for the full model
);
final engine = await LlamaEngine.load(
  LlamaModel(ModelSource.path('path/to/model.gguf')),
  params: params,
);

Constructors

ModelParams({bool liteRtLmEmbeddingEngine = false, int contextSize = 4096, ComputeDevice device = ComputeDevice.auto, int gpuLayers = maxGpuLayers, GpuBackend preferredBackend = GpuBackend.auto, @Deprecated('Use device, which applies to every runtime. This will be removed in ' '1.0.') LiteRtLmBackendPreference liteRtLmBackend = LiteRtLmBackendPreference.auto, LiteRtLmActivationDataType? liteRtLmActivationDataType, int? liteRtLmPrefillChunkSize, bool? liteRtLmParallelFileSectionLoading, String? liteRtLmDispatchLibDir, String? liteRtLmCacheDir, int? liteRtLmMaxProgramCacheBytes, ModelSplitMode splitMode = ModelSplitMode.layer, int mainGpu = 0, List<LoraAdapterConfig> loras = const [], String? chatTemplate, int numberOfThreads = 0, int numberOfThreadsBatch = 0, int batchSize = 0, int microBatchSize = 0, int maxParallelSequences = 1, int speculativeRollbackTokenMax = 0, bool useMmap = true, bool useMlock = false, bool loadMtp = false, FlashAttention flashAttention = FlashAttention.auto, KvCacheType cacheTypeK = KvCacheType.f16, KvCacheType cacheTypeV = KvCacheType.f16, bool? kvUnified, double? ropeFrequencyBase, double? ropeFrequencyScale, bool? preferMemory64, int? modelBytesHint})
Creates configuration for the model. Use validate to check for llama.cpp-incompatible combinations before passing to a load call.
const

Properties

batchSize → int
Maximum prompt/eval tokens per decode call (n_batch).
final
cacheTypeK → KvCacheType
llama_context_params.type_k. Non-F16 requires flashAttention enabled.
final
cacheTypeV → KvCacheType
llama_context_params.type_v. Non-F16 requires flashAttention enabled.
final
chatTemplate → String?
Optional Jinja chat template that replaces the model's own template.
final
contextSize → int
Context size (n_ctx) in tokens.
final
device → ComputeDevice
Device to run the model on, for every runtime.
final
flashAttention → FlashAttention
llama_context_params.flash_attn_type. User-explicit values override the platform/backend heuristic.
final
gpuLayers → int
Number of model layers to offload to the GPU (n_gpu_layers).
final
hashCode → int
The hash code for this object.
no setterinherited
kvUnified → bool?
llama_context_params.kv_unified. null keeps the current heuristic (auto-enabled when maxParallelSequences > 1).
final
liteRtLmActivationDataType → LiteRtLmActivationDataType?
Native LiteRT-LM activation data type override.
final
liteRtLmBackend → LiteRtLmBackendPreference
Preferred LiteRT-LM runtime backend for .litertlm models.
final
liteRtLmCacheDir → String?
Native LiteRT-LM runtime cache directory.
final
liteRtLmDispatchLibDir → String?
Native LiteRT-LM dispatch library directory for Android NPU deployments.
final
liteRtLmEmbeddingEngine → bool
Selects LiteRT-LM's separate embedding engine for .litertlm bundles.
final
liteRtLmMaxProgramCacheBytes → int?
Size cap in bytes for the native LiteRT-LM GPU program cache files of one cache directory.
final
liteRtLmParallelFileSectionLoading → bool?
Native LiteRT-LM file-section loading override.
final
liteRtLmPrefillChunkSize → int?
Native LiteRT-LM prefill chunk size for CPU dynamic models.
final
loadMtp → bool
Whether llama.cpp should load bundled multi-token prediction tensors.
final
loras → List<LoraAdapterConfig>
Initial LoRA adapters to load along with the model.
final
mainGpu → int
Primary GPU device index for model loading.
final
maxParallelSequences → int
Maximum parallel sequence slots in context memory (n_seq_max).
final
microBatchSize → int
Micro-batch size used by backend schedulers (n_ubatch).
final
modelBytesHint → int?
Web/WebGPU only: approximate model size in bytes, used to decide whether to load the mem64 core up front (instead of waiting for an out-of-memory failure and retrying). Ignored on non-web backends. null when unknown.
final
numberOfThreads → int
Number of threads to use for generation (n_threads).
final
numberOfThreadsBatch → int
Number of threads to use for batch processing (n_threads_batch).
final
preferMemory64 → bool?
Web/WebGPU only: prefer the 64-bit (wasm64/mem64) bridge core.
final
preferredBackend → GpuBackend
Preferred GPU backend for inference.
final
ropeFrequencyBase → double?
llama_context_params.rope_freq_base. null keeps the model's trained value.
final
ropeFrequencyScale → double?
llama_context_params.rope_freq_scale. null keeps the model's trained value.
final
runtimeType → Type
A representation of the runtime type of the object.
no setterinherited
speculativeRollbackTokenMax → int
llama.cpp recurrent-state rollback snapshots per sequence (n_rs_seq).
final
splitMode → ModelSplitMode
Model tensor distribution strategy across GPU devices.
final
useMlock → bool
Whether llama.cpp should lock model weights in memory. Default false.
final
useMmap → bool
Whether llama.cpp should memory-map model weights. Default true.
final

Methods

copyWith({bool? liteRtLmEmbeddingEngine, int? contextSize, ComputeDevice? device, int? gpuLayers, GpuBackend? preferredBackend, LiteRtLmBackendPreference? liteRtLmBackend, LiteRtLmActivationDataType? liteRtLmActivationDataType, bool clearLiteRtLmActivationDataType = false, int? liteRtLmPrefillChunkSize, bool clearLiteRtLmPrefillChunkSize = false, bool? liteRtLmParallelFileSectionLoading, bool clearLiteRtLmParallelFileSectionLoading = false, String? liteRtLmDispatchLibDir, bool clearLiteRtLmDispatchLibDir = false, String? liteRtLmCacheDir, bool clearLiteRtLmCacheDir = false, int? liteRtLmMaxProgramCacheBytes, bool clearLiteRtLmMaxProgramCacheBytes = false, ModelSplitMode? splitMode, int? mainGpu, List<LoraAdapterConfig>? loras, String? chatTemplate, bool clearChatTemplate = false, int? numberOfThreads, int? numberOfThreadsBatch, int? batchSize, int? microBatchSize, int? maxParallelSequences, int? speculativeRollbackTokenMax, bool? useMmap, bool? useMlock, bool? loadMtp, FlashAttention? flashAttention, KvCacheType? cacheTypeK, KvCacheType? cacheTypeV, bool? kvUnified, bool clearKvUnified = false, double? ropeFrequencyBase, bool clearRopeFrequencyBase = false, double? ropeFrequencyScale, bool clearRopeFrequencyScale = false, bool? preferMemory64, bool clearPreferMemory64 = false, int? modelBytesHint, bool clearModelBytesHint = false}) → ModelParams
Creates a copy of this ModelParams with updated fields.
noSuchMethod(Invocation invocation) → dynamic
Invoked when a nonexistent method or property is accessed.
inherited
toString() → String
A string representation of this object.
inherited
validate() → void
Validates the parameter combination. Throws LlamaArgumentException when a value is out of range or the combination is incompatible with llama.cpp (a non-F16 KV cache requires flashAttention != disabled), or when device contradicts another field:

Operators

operator ==(Object other) → bool
The equality operator.
inherited

Constants

defaultBatchSize → const int
Automatic logical batch size for generative contexts.
defaultLiteRtLmMaxProgramCacheBytes → const int
Program cache cap applied when liteRtLmMaxProgramCacheBytes is null: 1 GiB per cache directory.
defaultMicroBatchSize → const int
Automatic physical micro-batch size for generative contexts.
maxGpuLayers → const int
Maximum number of GPU layers to safely offload all layers.
unlimitedLiteRtLmProgramCacheBytes → const int
liteRtLmMaxProgramCacheBytes value that never deletes program caches.