LoadOptions class final
How a model is loaded. The defaults are the settings the package's measurements used: 4096 tokens of context, flash attention on, KV cache f16, every layer on the GPU.
- Annotations
-
- @immutable
Constructors
- LoadOptions({int contextTokens = 4096, int? threads, int? gpuLayers, int? microBatchSize, int? batchSize, bool flashAttention = true, KvCacheQuant kvCacheK = KvCacheQuant.f16, KvCacheQuant kvCacheV = KvCacheQuant.f16, ComputeBackend compute = ComputeBackend.auto})
-
const
Properties
- batchSize → int?
-
Logical batch size (n_batch); null (or 0) keeps the runtime's default.
final
- compute → ComputeBackend
-
The compute backend asked for.
final
- contextTokens → int
-
Context size in tokens (n_ctx).
final
- flashAttention → bool
-
Flash attention; the quantized KV cache types need it.
final
- gpuLayers → int?
-
Layers offloaded to the GPU; null (or a negative value) offloads all of them.
final
- hashCode → int
-
The hash code for this object.
no setteroverride
- kvCacheK → KvCacheQuant
-
Storage type of the KV cache's keys.
final
- kvCacheV → KvCacheQuant
-
Storage type of the KV cache's values.
final
- microBatchSize → int?
-
Prompt tokens per GPU pass (n_ubatch). Null means 128 on iOS, where a hot phone aborted 512-token passes, and
the runtime's default (512) elsewhere; 0 is the runtime's default everywhere.
final
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
- threads → int?
-
CPU threads for generation and for prompt processing; null (or 0) lets the runtime choose.
final
Methods
-
copyWith(
{int? contextTokens, int? threads, int? gpuLayers, int? microBatchSize, int? batchSize, bool? flashAttention, KvCacheQuant? kvCacheK, KvCacheQuant? kvCacheV, ComputeBackend? compute}) → LoadOptions - A copy with the given fields replaced. A null argument keeps the current value.
-
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
toJson(
) → Map< String, Object?> - Every field, enums by name, in declaration order.
-
toString(
) → String -
A string representation of this object.
override
Operators
-
operator ==(
Object other) → bool -
The equality operator.
override