tokenize abstract method

Future<List<int>> tokenize(
  1. int modelHandle,
  2. String text, {
  3. bool addSpecial = true,
})

Encodes the given text into a list of token IDs.

Implementation

Future<List<int>> tokenize(
  int modelHandle,
  String text, {
  bool addSpecial = true,
});