PEFT integration

[AngledLuffa]

Integrating PEFT into several different annotators

We integrate PEFT into our training pipeline for several different models. This greatly reduces the size of models with finetuned transformers, letting us make the finetuned versions of those models the default_accurate model.

The biggest gains observed are with the constituency parser and the sentiment classifier.

Previously, the default_accurate package used transformers where the head was trained but the transformer itself was not finetuned.

Model improvements

• POS trained with split optimizer for transformer & non-transformer - unfortunately, did not find settings which consistently improved results Splitting out the optimizer utility + LR scheduler for PoS #1320
• Sentiment trained with peft on the transformer: noticeably improves results for each model. SST scores go from 68 F1 w/ charlm, to 70 F1 w/ transformer, to 74-75 F1 with finetuned or Peft finetuned transformer. Peft sentiment #1335
• NER also trained with peft: unfortunately, no consistent improvements to scores PEFT on Named Entity Recognition #1336
• depparse includes peft: no consistent improvements yet TrainOps + Two Stage Optim for depparse #1337 Depparse peft #1344
• Dynamic oracle for top-down constituent parser scheme. Noticeable improvement in the scores for the topdown parser Con topdown oracle #1341
• Constituency parser uses peft: this produces significant improvements, close to the full benefit of finetuning the entire transformer when training constituencies. Example improvement, 87.01 to 88.11 on ID_ICON dataset. Con peft #1347
• Scripts to build a silver dataset for the constituency parser with filtering of sentences based on model agreement among the sub-models for the ensembles used. Preliminary work indicates an improvement in the benefits of the silver trees, with more work needed to find the optimal parameters used to build the silver dataset. Con silver #1348
• Lemmatizer ignores goeswith words when training: eliminates words which are a single word, labeled with a single lemma, but split into two words in the UD training data. Typical example would be split email addresses in the EWT training set. Lemma goeswith #1346 lemmatizer should skip Typo + GoesWith tags #1345

Features

• Include SpacesAfter annotations on words in the CoNLL output of documents: "SpaceAfter=No" not being included in misc field of Word objects #1315 Spaces after #1322
• Lemmatizer operates in caseless mode if all of its training data was caseless. Most relevant to the UD Latin treebanks. Potentially lowercase the data in a lemmatizer if all of the training… #1331 Latin default package doesn't usually lemmatize words starting with a capital letter #1330
• wandb support for coref Enables weights and biases support for coref #1338
• Coref annotator breaks length ties using POS if available coref not using proper noun #1326 c4c3de5

Bugfixes

• Using a proxy with download_resources_json was broken: Fix bug: Added missing parameter in download_resources_json #1318 An exception occurred when using a proxy to download the model #1317 Thank you @ider-zh
• Fix deprecation warnings for escape sequences: Fix invalid escape sequences #1321 Add support for Python 3.11 (& 3.12) #1293 Thank you @sterliakov
• Coref training rounding error Fixes coref warmup rounding error #1342
• Top-down constituency models were broken for datasets which did not use ROOT as the top level bracket... this was only DA_Arboretum in practice Fix TOP_DOWN parser for da_arboretum, which needs to look at the actu… #1354
• V1 of chopping up some longer texts into shorter texts for the transformers to get around length limits. No idea if this actually produces reasonable results for words after the token limit. Initial attempt to chop up long inputs to a transformer into pieces t… #1350 Tokenizer doesn't respect combined_electra-large's max_length #1294
• Coref prediction off-by-one error for short sentences, was falsely throwing an exception at sentence breaks: Span crossing two sentences? #1333 Coref model predicted a span that crossed two sentences #1339 f1fbaaa
• Clarify error when a language is only partially handled: da01644 Language "si" does not have 'default_processors' and thus Pipeline creation fails with user-unfriendly error. #1310

---

This discussion was created from the release PEFT integration.