transformers

The Split class accepts SplitDelimiterBehavior which is really useful. The Punctuation however always uses SplitDelimiterBehavior::Isolated (and Whitespace on the other hand behaves like SplitDelimiterBehavior::Removed).

impl PreTokenizer for Punctuation {
    fn pre_tokenize(&self, pretokenized: &mut PreTokenizedString) -> Result<()> {
        pretokenized.split(|_, s| s.spl

Problem
Since Java 8 was introduced there is no need to use Joda as it has been replaced the native Date-Time API.

Solution
Ideally greping and replacing the text should work (mostly)

Additional context
Need to check if de/serializing will still work.

Hey! Thanks for the work on this.

Wondering how we can use this with mocha? tsconfig-paths has its own tsconfig-paths/register to make this work

https://github.com/dividab/tsconfig-paths#with-mocha-and-ts-node

Basically with mocha we have to run mocha -r ts-node/register -- but that wouldnt have the compiler flag.

Would be worthwhile to have the ability to do it which looks like