The new link is:
https://github.com/chonkie-inc/chonkie

* feat: update lock

* pin transformers

* feat: allow passing validation set explicitly

* fix: typing

* fix: input validation

* add tests, extra check, pre-commit

---------

Co-authored-by: stephantul <[email protected]>

* Added multilingual results

* Updated readme

* Updated readme

* Updated readme

* Updated readme

* Updated readme

* remove multiword warning

* add superbpe tokenizers

* fix issue with mwe

* form

* working version

* first pass

* small fixes, many comments

* fix e5 bug

* Adjust arcane formulae

* fix: logging

* wip

* wip

* wip

* lower complexity

* add lock file

* fix: metaspace pretokenizer

* fix: bug in vocab

* feat: spaces/commas etc.

* turn tokenizer into package

* add annotations

* feat: turn tokenizer into package

* fix: future

* add tokenizer function

* update lockfile

* feat: improve segmentation of unigram

* fix: broken merge

* fix interpunct tokens

* fix tests, make tokenizer changes better

* update lock file

* fix comment, add additional check for pad token

* tests: add a lot of tests

* fix: 3.9 error

* Added new logo

* Added new logo

* Added new logo

* Added new logo

* Added new logo

* wip

* fix: tokenizer bug

* add correct scaling for byte