tokenize method

  1. @override
Future<List<int>> tokenize(
  1. String text
)
override

The tokens of text, special tokens added as generate adds them (the vocabulary decides about BOS).

Implementation

@override
Future<List<int>> tokenize(String text) async {
  _checkOpen();
  return _queue.run(() async => List<int>.generate(text.length ~/ 4, (i) => i));
}