Vocabulary class
Constructors
- Vocabulary({String subwordPrefix = '##'})
Properties
- clsTokenId → int
-
no setter
- hashCode → int
-
The hash code for this object.
no setterinherited
- maskTokenId → int
-
no setter
- padTokenId → int
-
no setter
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
- sepTokenId → int
-
no setter
- size → int
-
no setter
- subwordPrefix → String
-
final
- subwordTrie → Trie
-
no setter
-
tokens
→ List<
String> -
no setter
- trie → Trie
-
no setter
- unkTokenId → int
-
no setter
-
vocabularyMap
→ Map<
String, int> -
no setter
Methods
-
contains(
String token) → bool -
findLongestMatch(
String text, {int startIndex = 0, bool isSubword = false}) → TrieMatch? -
idToToken(
int id) → String -
isSpecialToken(
String token) → bool -
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
tokenToId(
String token) → int -
toString(
) → String -
A string representation of this object.
override
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited
Static Methods
-
fromFile(
String path, {String subwordPrefix = '##'}) → Future< Vocabulary> -
fromFileSync(
String path, {String subwordPrefix = '##'}) → Vocabulary -
fromString(
String content, {String subwordPrefix = '##'}) → Vocabulary -
fromTokens(
List< String> tokens, {String subwordPrefix = '##'}) → Vocabulary