A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)

streamline the fine-tuning process for multimodal models: PaliGemma, Florence-2, and Qwen2-VL

Code for "Aligning Linguistic Words and Visual Semantic Units for Image Captioning", ACM MM 2019

CapDec: SOTA Zero Shot Image Captioning Using CLIP and GPT2, EMNLP 2022 (findings)

JoyCaption is an image captioning Visual Language Model (VLM) being built from the ground up as a free, open, and uncensored model for the community to use in training Diffusion models.

Audio Captioning datasets for PyTorch.

Fully-Convolutional Point Networks for Large-Scale Point Clouds

VisText is a benchmark dataset for semantically rich chart captioning.

A Tennis dataset and models for event detection & commentary generation

Python code for handling the Clotho dataset.

A Base Tensorflow Project for Medical Report Generation

Medical image captioning using OpenAI's CLIP

What and How Well You Performed? A Multitask Learning Approach to Action Quality Assessment [CVPR 2019]

[CVPR21] Visual Semantic Role Labeling for Video Understanding (https://arxiv.org/abs/2104.00990)

Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluation. CVPR 2023

A gradio based image captioning tool that uses the GPT-4-Vision API to generate detailed descriptions of images.

A Pytorch implementation of Attention on Attention module (both self and guided variants), for Visual Question Answering

Using LLMs and pre-trained caption models for super-human performance on image captioning.

Audio captioning baseline system for DCASE 2020 challenge.

Sample app to display live captioning to a WebRTC video session with the Deepgram API.