publications

2023

1. 

   OtterHD: A High-Resolution Multi-modality Model

   Bo Li* , Peiyuan Zhang*, Jingkang Yang^†, Yuanhan Zhang^†, Fanyi Pu^†, and Ziwei Liu

   2023

   Abs arXiv Bib PDF Code

   

   In this paper, we present OtterHD-8B, an innovative multimodal model evolved from Fuyu-8B, specifically engineered to interpret high-resolution visual inputs with granular precision. Unlike conventional models that are constrained by fixed-size vision encoders, OtterHD-8B boasts the ability to handle flexible input dimensions, ensuring its versatility across various inference requirements. Alongside this model, we introduce MagnifierBench, an evaluation framework designed to scrutinize models’ ability to discern minute details and spatial relationships of small objects. Our comparative analysis reveals that while current leading models falter on this benchmark, OtterHD-8B, particularly when directly processing high-resolution inputs, outperforms its counterparts by a substantial margin. The findings illuminate the structural variances in visual information processing among different models and the influence that the vision encoders’ pre-training resolution disparities have on model effectiveness within such benchmarks. Our study highlights the critical role of flexibility and high-resolution input capabilities in large multimodal models and also exemplifies the potential inherent in the Fuyu architecture’s simplicity for handling complex visual data.

   @misc{li2023otterhd,
     title = {OtterHD: A High-Resolution Multi-modality Model},
     author = {Li*, Bo and Zhang*, Peiyuan and Yang<sup>†</sup>, Jingkang and Zhang<sup>†</sup>, Yuanhan and Pu<sup>†</sup>, Fanyi and Liu, Ziwei},
     year = {2023},
     eprint = {2311.04219},
     archiveprefix = {arXiv},
     primaryclass = {cs.CV},
     google_scholar_id = {u-x6o8ySG0sC},
   }

2. 

   MIMIC-IT: Multi-Modal In-Context Instruction Tuning

   Bo Li*, Yuanhan Zhang*, Liangyu Chen*, Jinghao Wang*, Fanyi Pu*, Jingkang Yang , Chunyuan Li, and Ziwei Liu

   2023

   Abs arXiv Bib PDF Code

   

   High-quality instructions and responses are essential for the zero-shot performance of large language models on interactive natural language tasks. For interactive vision-language tasks involving intricate visual scenes, a large quantity of diverse and creative instruction-response pairs should be imperative to tune vision-language models (VLMs). Nevertheless, the current availability of vision-language instruction-response pairs in terms of quantity, diversity, and creativity remains limited, posing challenges to the generalization of interactive VLMs. Here we present MultI-Modal In-Context Instruction Tuning (MIMIC-IT), a dataset comprising 2.8 million multimodal instruction-response pairs, with 2.2 million unique instructions derived from images and videos. Each pair is accompanied by multi-modal in-context information, forming conversational contexts aimed at empowering VLMs in perception, reasoning, and planning. The instruction-response collection process, dubbed as Syphus, is scaled using an automatic annotation pipeline that combines human expertise with GPT’s capabilities. Using the MIMIC-IT dataset, we train a large VLM named Otter. Based on extensive evaluations conducted on vision-language benchmarks, it has been observed that Otter demonstrates remarkable proficiency in multi-modal perception, reasoning, and in-context learning. Human evaluation reveals it effectively aligns with the user’s intentions. We release the MIMIC-IT dataset, instruction-response collection pipeline, benchmarks, and the Otter model.

   @article{li2023mimicit,
     title = {MIMIC-IT: Multi-Modal In-Context Instruction Tuning},
     author = {Li*, Bo and Zhang*, Yuanhan and Chen*, Liangyu and Wang*, Jinghao and Pu*, Fanyi and Yang, Jingkang and Li, Chunyuan and Liu, Ziwei},
     year = {2023},
     eprint = {2306.05425},
     archiveprefix = {arXiv},
     primaryclass = {cs.CV},
     google_scholar_id = {u5HHmVD_uO8C},
   }