tiktoken is a fast BPE tokeniser for use with OpenAI's models.
A fast tokenizer/lexer core that is well tested.
Tokenize JavaScript
A regular expression (RexExp) stream tokenizer.
An HTML5 lexer
Lightweight debugging utility for Node.js and the browser
Lodash modular utilities.
React is a JavaScript library for building user interfaces.
A querystring parser that supports nesting and arrays, with a depth limit
Higher-order functions and common patterns for asynchronous code
Babel compiler core.