Florence finetunes in workflows

[probicheaux]

Description

A few notable changes to florence models:

• Adds in a new (fake) "task_type" unstructured, to allow the user to type whatever they want as a prompt
• Creates a v2 block that remaps model_version to model_id so that the user can search their finetuned florence 2 models in app
• tweaks list of required model files to avoid always redownloading florence2 models
• adds no_repeat_ngram_size=0 to our transformers generation args

Why we set no_repeat_ngram_size=0:

I found a very sneaky bug destroying the ability of the model to generate valid json, where this parameter was set to 3. Having the parameter set to 3 means that once the model has seen a sequence of 3 tokens, it can never repeat that sequence.

Here's the generation for a receipt json before and after making this change:
Generation before:

{'<RECEIPT>': '{"subtotal": "$27.36", "total":{"29.96"}'}

Generation after:

{'<RECEIPT>': '{"subtotal": "$27.36", "total": "$29.96"}'}

Here's why -- here's the tokens the model generated after I fixed the parameter.

['</s>', '<s>', '{"', 'sub', 'total', '":', ' "$', '27', '.', '36', '",', ' "', 'total', '":', ' "$', '29', '.', '96', '"}', '</s>']

Notice the repeated sequence of tokens:

'total', '":', ' "$',

that the model was unable to generate before this change.

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

How has this change been tested, please provide a testcase or example of how you tested the change?

tested and deployed thouroughly on localhost

Any specific deployment considerations

no

Docs

• Docs updated? What were the changes:

[PawelPeczek-Roboflow]

in UI we do not display multiple versions of the same block treating newer versions as the ones shipping super-set of features - in this case (if I understand correctly) - you either choose pre-trained model (v1) or fine-tuned one (v2) - which needs to change

[probicheaux]

v2 can use the pre-trained ones by passing florence-2-base or florence-2-large as the model_id (same as before)! This is analogous to yolov8n-640 for object detection. This should show up in the model selector pop up but isn't -- we tried to do this in the frontend similar to yolov8n-640 or whatever, but its not working right now. Will look into it.

[probicheaux]

Also the field is pre populated with the right pretrained value so ppl should not see a difference unless they try

[probicheaux]

@PawelPeczek-Roboflow good to go?