tokenkit is a toolkit implementing advanced methods to transfer models and model knowledge across tokenizers. LLMs are bound to the tokenizer they were pretrained with. This limits their adaptabilit… [+6550 chars]...
