Elman, Your Multimodal Digest: Bridging the Gap Between Pixels and Prose
The convergence of vision and language is rapidly reshaping the landscape of AI. This newsletter delves into the latest breakthroughs in multimodal image and text foundation models, exploring novel approaches to data generation, evaluation benchmarks, and real-world applications. From enhancing visual instruction tuning to achieving latency-free robot control, these advancements highlight the growing power and potential of multimodal AI.
Hybrid Instruction Generation for High-Quality Text-Rich Image Datasets
A High-Quality Text-Rich Image Instruction Tuning Dataset via Hybrid Instruction Generation by Shijie Zhou, Ruiyi Zhang, Yufan Zhou, Changyou Chen https://arxiv.org/abs/2412.16364
Caption: The image illustrates the construction of the LLaVAR-2 dataset for visual instruction tuning. It shows the process of enriching manual captions with details using GPT-4 and OCR, followed by the generation of extractive and self-explain question-answer pairs. Finally, the generated data is filtered using mIFD and FFD scores to ensure high quality.
Multimodal Large Language Models (MLLMs) often struggle to understand images rich in textual content due to a lack of appropriate training data. While self-instruction methods offer a seemingly convenient solution, they often fall short in generating high-quality data because of difficulties in aligning visual and textual information. This paper introduces LLaVAR-2, a new dataset specifically designed for visual instruction tuning that leverages a hybrid approach, combining human expertise with the capabilities of large language models like GPT-4.
LLaVAR-2 consists of two main components: LLaVAR-2-Cap for detailed image captioning and LLaVAR-2-VQA for visual question answering. LLaVAR-2-Cap begins with human-annotated captions from the TRINS dataset. These initial captions are then enriched with fine-grained details and text labels extracted using OCR and GPT-4, providing comprehensive descriptions of both visual and textual elements within the image. LLaVAR-2-VQA comprises extractive question-answer pairs (D<sub>e</sub>:{Q<sub>e</sub>, A<sub>e</sub>}) generated using GPT-4 based on these enriched captions and OCR data. Importantly, each extractive pair is supplemented with a self-explain pair (D<sub>r</sub>:{Q<sub>r</sub>, A<sub>r</sub>}), also generated by GPT-4, that explains the reasoning behind the answer A<sub>e</sub>. This self-explanation component is designed to improve the model's ability to handle complex reasoning tasks and understand the intricate relationships between visual and textual elements.
To ensure the dataset's high quality, the authors introduce two novel filtering metrics: multimodal Instruction-following Difficulty (mIFD) and Fact-Following Difficulty (FFD). mIFD, calculated as mIFD(Q, A, I<sub>i</sub>) = √VFD(D, I<sub>i</sub>) × IFD(Q, A), combines Visual-Following Difficulty (VFD) and Instruction Following Difficulty (IFD) to filter out extractive QA pairs that are either irrelevant to the image or where the answer is unrelated to the question. FFD, calculated as FFD(D, D<sub>r</sub>, I<sub>i</sub>) = S<sub>θ</sub>(D<sub>r</sub>|D, I<sub>i</sub>) / S<sub>θ</sub>(D<sub>r</sub>|I<sub>i</sub>), measures the closeness between the extractive and self-explain pairs, filtering out redundant or unrelated self-explanations. These filtering mechanisms ensure that the dataset is focused on relevant and informative instruction-following data. The resulting LLaVAR-2 dataset contains 42k detail-enriched captions and 382k visual question-answering pairs.
Experiments demonstrate the effectiveness of LLaVAR-2 in improving visual instruction tuning. A model fine-tuned on LLaVAR-2, LLaVAR-2-3.8B, substantially outperforms models trained with self-instruct data on various benchmarks, including LLaVAR-2-Cap, LLaVAR-2-VQA, and established text-rich image understanding datasets. For instance, on LLaVAR-2-VQA, LLaVAR-2-3.8B achieves a 59.6% extract accuracy, significantly higher than other models. On text-rich image captioning tasks, LLaVAR-2 models show marked improvements over baselines in terms of text similarity metrics like BLEU, METEOR, ROUGE, and CIDEr. These results underscore the value of the hybrid instruction generation approach and the crucial role of high-quality, detail-rich data for effective visual instruction tuning.
Quantifying the Realism Gap: A New Benchmark for AI-Generated Images
ANID: How Far Are We? Evaluating the Discrepancies Between AI-synthesized Images and Natural Images through Multimodal Guidance by Renyang Liu, Ziyu Lyu, Wei Zhou, See-Kiong Ng https://arxiv.org/abs/2412.17632
Caption: This figure illustrates the DNAI dataset used in the AI-Natural Image Discrepancy Evaluation Benchmark (ANID) study. It shows examples of naturally generated images alongside their AI-generated counterparts, categorized by prompting method (T2I, TI2I, I2I). The accompanying framework diagram outlines the five dimensions of evaluation used to quantify the discrepancies between AI and natural images.
Despite the remarkable progress in AI image generation, a critical question remains: how realistic are these synthetic images truly? This study introduces the AI-Natural Image Discrepancy Evaluation Benchmark (ANID) to quantify the gap between AI-generated images (AIGIs) and natural images. The researchers have assembled a massive multimodal dataset, the Distinguishing Natural and AI-generated Images (DNAI) dataset, comprising over 440,000 AIGI samples, generated by eight representative models using unimodal (text-to-image, image-to-image) and multimodal (text vs. image-to-image) prompts. This dataset is significantly larger than previous efforts, scaling to 100x the size of prior datasets.
The ANID benchmark utilizes a comprehensive evaluation framework spanning five key dimensions: naive visual feature quality, semantic alignment in multimodal generation, aesthetic appeal, downstream task applicability, and human validation. Naive image quality is assessed at pixel, frame, and content distribution levels using metrics such as SSIM, LPIPS, FID, and Inception Score. Semantic alignment is measured using CLIP Score. Aesthetic appeal is evaluated using NIMA and LAION-AES. Downstream applicability is tested in image recognition (measuring Mismatch Rate) and semantic segmentation (using Intersection over Union - IoU). Crucially, human evaluations provide a layer of subjective assessment across quality, alignment, and aesthetics.
The results reveal substantial discrepancies between AIGIs and natural images across all dimensions. Quantitatively, AIGIs exhibit 10% to 30% lower performance compared to natural images. For instance, AIGIs showed a 20% to 50% reduction in SSIM compared to their natural counterparts. The classification mismatch rate in downstream image recognition tasks ranged from 29.89% to 94.44% for AIGIs. Human evaluations highlighted even larger discrepancies than quantitative metrics, emphasizing the importance of subjective assessments. The study also observed that different prompting methods affect semantic alignment, with text-based prompts (both unimodal and multimodal) demonstrating better alignment than image-based prompts. These findings highlight the ongoing challenges in achieving true realism in AIGIs. While significant progress has been made, this study underscores the need for further research to close the remaining gap. The discrepancies observed across different image categories indicate potential imbalances in training data, emphasizing the need for more balanced datasets and targeted model improvements. The misalignment between quantitative metrics and human perception also necessitates the development of new evaluation measures specifically designed for AIGC. This comprehensive benchmark provides invaluable insights for future research and development in AI image generation, paving the way for more realistic and applicable synthetic imagery.
Speech-Driven Visual Reasoning with SilVar
SilVar: Speech Driven Multimodal Model for Reasoning Visual Question Answering and Object Localization by Tan-Hanh Pham, Hoang-Nam Le, Phu-Vinh Nguyen, Chris Ngo, Truong-Son Hy https://arxiv.org/abs/2412.16771
Caption: This diagram illustrates the architecture of SilVar, a novel multimodal model designed for speech-driven visual reasoning. It shows how SilVar processes speech instructions (converted to text by Whisper) and visual input (encoded by CLIP) using LLaMA to generate responses, such as identifying and localizing Canada Geese in an image. The two-stage training process, involving speech-to-text alignment and multimodal instruction fine-tuning, is also depicted.
While Visual Language Models (VLMs) have demonstrated remarkable capabilities, most rely on text-based instructions, limiting their practicality in human-machine interactions. Furthermore, open-source models capable of effective speech interaction are scarce. This paper introduces SilVar, a novel end-to-end multimodal model designed for reasoning in visual question answering using speech instructions. SilVar tackles object localization and detailed scene description, moving beyond simple object recognition. It leverages existing open-source foundation models, combining CLIP for visual encoding, Whisper for audio encoding, and LLaMA 3.1-8B as the core language model.
The authors explore various reasoning techniques with different levels of complexity, including conversational, simple, and complex speech instructions. To facilitate this research, they've created a new dataset, SilVar-bench, specifically designed for speech-based reasoning tasks in object localization. This dataset consists of images paired with text, speech instructions, and bounding box annotations, generated with the help of GPT-4. SilVar-bench emphasizes human-machine conversation, detailed descriptions, and reasoning instructions and responses, pushing beyond the capabilities of traditional object recognition datasets. The training process involves two stages: speech-to-text alignment using Whisper and subsequent LLM training using the aligned speech and visual data. Existing text-based reasoning datasets like MMMU, LISA, and ScienceQA are also used for pre-training and comparison.
SilVar's performance was evaluated on MMMU, ScienceQA, and SilVar-bench. On MMMU, SilVar achieved a validation score of 31.8 with text instructions and 30.2 with speech instructions, demonstrating competitive performance against state-of-the-art models like LLaVA-1.5 and Qwen-VL-7B-Chat. On ScienceQA, SilVar achieved an average accuracy of 63.21% with speech instructions, showcasing its effective processing of multimodal inputs. Qualitative comparisons with commercial chatbots like GPT-40 and Gemini 1.5 Pro revealed SilVar's superior ability to provide detailed explanations and accurate bounding boxes for localized objects. An ablation study investigated different audio adapter architectures (MLP and Transformer), finding minimal performance differences, suggesting that Whisper's encoder effectively transfers speech signals to the language model. SilVar highlights the potential of speech-driven multimodal models for complex visual reasoning tasks. The results indicate that while text-based instructions still yield slightly better performance, speech-based interaction is a viable and promising alternative, especially in situations where text input is inconvenient. The introduction of the SilVar-bench dataset provides a valuable resource for future research in this emerging field.
Unifying Image and Pixel-Level Vision-Language Alignment with DINOv2
DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment by Cijo Jose, Théo Moutakanni, Dahyun Kang, Federico Baldassarre, Timothée Darcet, Hu Xu, Daniel Li, Marc Szafraniec, Michaël Ramamonjisoa, Maxime Oquab, Oriane Siméoni, Huy V. Vo, Patrick Labatut, Piotr Bojanowski https://arxiv.org/abs/2412.16334
Caption: This diagram illustrates the dino.txt framework, which aligns a frozen DINOv2 visual backbone with a trainable text encoder for open-vocabulary vision-language tasks. It shows the three stages: SSL pretraining of DINOv2, text alignment training using a contrastive loss on image patches and text embeddings, and finally, test-time inference for both classification and segmentation. The framework leverages a combined global and local image representation g for improved performance.
Self-supervised visual foundation models like DINOv2 excel at generating powerful image embeddings, but they lack the inherent language understanding of vision-language models like CLIP. This limits their application in open-vocabulary tasks. This paper introduces dino.txt, a method that bridges this gap by aligning DINOv2's feature space with language, enabling it to handle open-vocabulary recognition tasks without the substantial computational cost of training CLIP from scratch. The method builds upon Locked-image text tuning (LiT), which trains a text encoder to align with a frozen vision model, but addresses LiT's weaknesses in dense prediction tasks.
Instead of relying solely on the [CLS] token, dino.txt uses a combined representation g = [c'; σ([f₁,………, f’])], where c' is the updated [CLS] token, f’ are the patch embeddings after passing through trainable vision blocks, and σ represents average pooling. This combined representation captures both global and local image information, improving alignment for both classification and segmentation. Two trainable transformer blocks are added on top of the frozen DINOv2 backbone, enabling visual features to adapt to the new training data and mitigating the domain gap between pre-training and LiT training data. A novel data curation strategy balances image and text distributions, resulting in more efficient training and enhanced performance.
The method's performance was evaluated on zero-shot classification, image-text retrieval, and open-vocabulary segmentation tasks. Dino.txt achieves state-of-the-art zero-shot classification accuracy, outperforming existing CLIP-like models on ImageNet-v2, ImageNet-A, and iNaturalist2021. On ImageNet-1K, it achieves 81.4% accuracy with a ViT-L model, comparable to state-of-the-art methods, but with significantly less training time (19 hours on 128 A100 GPUs versus 110 hours for CLIP). While its retrieval performance is slightly lower than some specialized models, dino.txt significantly outperforms them on open-vocabulary segmentation, achieving 20.6 mIoU on ADE20K, 32.1 mIoU on Cityscapes, and 62.1 mIoU on PASCAL VOC2012 using a simple inference procedure without task-specific adaptations.
An error analysis on ADE20K revealed limitations in existing semantic segmentation benchmarks for open-vocabulary evaluation. Issues such as overlapping objects and inconsistencies between class names and their everyday language usage were identified, highlighting the need for more robust evaluation metrics and datasets. Analysis of the trained text encoder revealed lower performance compared to CLIP's text encoder on text-based benchmarks, suggesting that freezing the vision encoder might hinder the text encoder's learning. Future research directions include improving the text encoder's quality and developing more appropriate benchmarks for open-vocabulary semantic segmentation.
Kernel-Based Continual Learning: Reaching Joint Training Accuracy
Continual Learning Using a Kernel-Based Method Over Foundation Models by Saleh Momeni, Sahisnu Mazumder, Bing Liu https://arxiv.org/abs/2412.15571
Caption: This figure showcases the performance of the Kernel Linear Discriminant Analysis (KLDA) method for continual learning. The left graph demonstrates KLDA's robustness across varying kernel bandwidths (σ), while the right graph illustrates its performance with different dimensionality (D) of Random Fourier Features. Notably, KLDA consistently achieves accuracy comparable to, or even exceeding, joint training across various text datasets (CLINC, Banking, DBpedia, HWU).
Continual learning (CL), particularly class-incremental learning (CIL), remains a significant challenge. CIL involves training a model on a sequence of tasks, each introducing new classes. The main obstacles are catastrophic forgetting (CF) – where performance on previous tasks degrades when learning new ones – and inter-task class separation (ICS) – the difficulty of distinguishing between classes from different tasks without task information during testing. Existing methods struggle with these issues despite employing various strategies like regularization, replay, and architectural modifications.
This paper introduces Kernel Linear Discriminant Analysis (KLDA), a novel CIL method that leverages the power of foundation models (FMs) without updating their parameters, thus preventing CF. KLDA utilizes the rich features extracted from a pre-trained FM. However, instead of using these features directly, it employs a kernel-based approach to enhance them. Specifically, the Radial Basis Function (RBF) kernel and its Random Fourier Features (RFF) approximation are used to map the features into a higher-dimensional space for improved linear separability, addressing the ICS challenge. For each new task, KLDA computes the mean of the kernelized features for each class and updates a shared covariance matrix for all classes seen so far. Classification is then performed using Linear Discriminant Analysis (LDA). An ensemble variant, KLDA-E, further boosts performance by averaging predictions from multiple KLDA models initialized with different RFF parameters.
The authors evaluated KLDA on several text and image classification datasets. On text datasets using BART-base as the FM, KLDA-E consistently matched or even exceeded the accuracy of joint training – the upper bound for CIL performance, achieved by training on all classes/tasks simultaneously. This is a remarkable achievement, as existing CIL methods typically fall considerably short of this upper bound. For example, on the CLINC dataset, KLDA-E achieved 96.62% accuracy compared to 95.33% for joint training. Similar results were observed on Banking (93.03% vs. 91.36%), DBpedia (94.53% vs. 94.83%), and HWU (89.78% vs. 88.60%) datasets. KLDA's robustness across different LFMs was also demonstrated, achieving comparable performance to joint training with models like MiniLM, BERT-base, RoBERTa-large, T5-3b, and Mistral-7b. On image datasets, KLDA performed competitively with joint training, although a small performance gap suggests that current vision FMs may not possess the same level of generalizability as their language counterparts.
KLDA also offers practical advantages in efficiency and memory usage. As it only computes class means and updates the covariance matrix, training time is significantly shorter compared to fine-tuning based methods. The memory footprint, while larger than NCM due to the kernel transformation and covariance matrix, remains manageable and doesn't scale with the number of classes in the same way as methods requiring replay buffers or generative components. These results strongly suggest that KLDA offers a promising new direction for CIL, effectively addressing the challenges of catastrophic forgetting and inter-task class separation while achieving near-optimal performance. Reaching joint training accuracy without replay data is a significant step forward for practical applications of continual learning.
A Comprehensive Survey of Multimodal Datasets and Applications
Survey of Large Multimodal Model Datasets, Application Categories and Taxonomy by Priyaranjan Pattnayak, Hitesh Laxmichand Patel, Bhargava Kumar, Amit Agarwal, Ishan Banerjee, Srikant Panda, Tejaswini Kumar https://arxiv.org/abs/2412.17759
This survey provides a comprehensive overview of the current state of multimodal datasets, their application categories, and a taxonomy for understanding the burgeoning field of Multimodal Large Language Models (MLLMs). Driven by the increasing sophistication of LLMs, multimodal learning, which focuses on integrating and processing diverse data types like text, images, and audio, is rapidly evolving. The paper emphasizes the crucial role of datasets in training and evaluating these models, highlighting the shift from single-modality methods to the more nuanced and comprehensive approach of multimodal learning.
The survey categorizes multimodal datasets into three primary types: training-specific, task-specific, and domain-specific. Training-specific datasets are further subdivided into Multimodal Pre-Training (MM-PT) and Multimodal Instruction Tuning (MM-IT). MM-PT datasets, such as LAION-5B and MS-COCO, focus on aligning different modalities like text and images. MM-IT datasets, including LLaVA-Instruct and SVIT, leverage instruction-formatted data to improve zero-shot performance through techniques like Supervised Fine-Tuning (SFT) and Reinforcement Learning with Human Feedback (RLHF). Task-specific datasets, like SlideVQA and ImageNet, are designed for specific tasks such as visual question answering and object recognition, pushing the boundaries of model capabilities in those areas. Domain-specific datasets, like MIMIC-CXR (medical imaging) and KITTI (autonomous driving), cater to the unique challenges and data requirements of specific fields, enabling the development of tailored multimodal solutions.
The paper underscores the opportunities and challenges presented by MLLMs. By integrating diverse modalities, these models offer transformative potential across various domains, from healthcare and education to scientific research. However, challenges persist, including the scarcity of large-scale, high-quality multimodal datasets, the computational demands of training and deploying these models, and the need to ensure their reliability, interpretability, and ethical alignment. Addressing these challenges requires developing novel model architectures, efficient training strategies, and robust evaluation frameworks.
The survey explores the characteristics and limitations of existing multimodal datasets. While datasets like MS-COCO offer extensive real-world data distributions, challenges related to data biases, imbalances, and limited task diversity remain. Privacy and security concerns also necessitate responsible data usage. The paper highlights the importance of thoughtful dataset design and curation to overcome these limitations and promote the development of robust and ethical multimodal learning systems. Finally, the survey identifies emerging trends and future directions in multimodal dataset development, including the integration of diverse sensory inputs like tactile and olfactory data, a focus on geographically and linguistically diverse datasets to improve generalization and mitigate biases, and the creation of datasets that capture complex intermodal interactions for real-world applications. The emphasis on standardized documentation and benchmarking practices is crucial for ensuring transparency, reproducibility, and ethical considerations in this rapidly evolving field.
Latency-Free Multimodal Control for Quadruped Robots
QUART-Online: Latency-Free Large Multimodal Language Model for Quadruped Robot Learning by Xinyang Tong, Pengxiang Ding, Donglin Wang, Wenjie Zhang, Can Cui, Mingyang Sun, Yiguo Fan, Han Zhao, Hongyin Zhang, Yonghao Dang, Siteng Huang, Shangke Lyu https://arxiv.org/abs/2412.15576
Caption: This diagram illustrates the QUART-Online architecture for quadruped robot control. The top pathway shows the slower, 2Hz QUART model, while the bottom pathway depicts the 50Hz QUART-Online model, which uses compressed action tokens and a decoder to achieve latency-free performance. This allows the robot to react in real-time to instructions like "go avoid the red rectangle tunnel," as shown by the robot successfully navigating around the obstacle.
Deploying large multimodal language models (MLLMs) for real-time robot control presents a significant challenge due to inference latency. Traditional methods like parameter reduction, while improving inference speed, often compromise the model's performance, especially its ability to generalize. This paper introduces QUART-Online, a novel approach that enhances inference efficiency without sacrificing the MLLM's capabilities.
QUART-Online's core lies in Action Chunk Discretization (ACD) and Action Chunk Alignment. ACD compresses the action representation space by mapping continuous action values onto a smaller set of discrete representative vectors. These compressed tokens retain semantic meaning, allowing joint optimization of actions and perceptual data without disrupting the model's learned distribution. Action Chunk Alignment fine-tunes the MLLM to integrate vision, language, and these compressed actions into a unified semantic space. During inference, the MLLM outputs compressed action tokens, which are decoded into a continuous trajectory for the robot. The formula for QUART-Online is:
QUART-Online(Âc|s,w) = D(Âc|Âq)p(Âq|c)τ(c|s,w)
where s and w are input images and language instructions, τ is the tokenizer, p is the vision-language model, Âq are the compressed action tokens, D is the action decoder, and Âc is the reconstructed continuous action trajectory.
Experimental results on the QUARD benchmark demonstrate QUART-Online's effectiveness. Compared to the baseline QUART model, QUART-Online achieves a 65% improvement in average task success rate and increases inference frequency from 2Hz to 50Hz. This allows real-time inference synchronized with the controller's frequency, crucial for dynamic environments. QUART-Online outperforms other baselines in success rates across various tasks, including those with unseen visual elements and language instructions, highlighting its superior generalization.
The improved performance stems from efficient temporal information encoding through longer action chunk durations, enhancing reasoning abilities. The latency-free operation enables swift reactions to dynamic changes, as demonstrated by its ability to avoid obstacles where the baseline QUART model failed due to latency. While QUART-Online currently relies on low-level controllers for joint angle actions, future research will explore direct generation of joint-level commands and performance in more complex locomotion scenarios.
Efficient MedSAMs: Bringing Promptable Segmentation to the Clinic
Efficient MedSAMs: Segment Anything in Medical Images on Laptop by Jun Ma et al. https://arxiv.org/abs/2412.16085
Caption: The figure illustrates the workflow of an international competition focused on developing efficient promptable medical image segmentation models. Part (a) shows example input images with box prompts and the corresponding output segmentation masks, while part (b) details the three phases of the competition: development (model training and tuning), testing (evaluation on a hidden dataset), and post-challenge (performance boosting and reproducibility analysis). The competition aimed to create lightweight, fast models capable of running on standard hardware, as depicted by the neural network architecture and the emphasis on runtime optimization in the workflow.
Promptable segmentation foundation models hold great promise for medical image analysis, but their high computational requirements have hindered clinical adoption. A new international competition aimed to address this by challenging researchers to develop lightweight, efficient models capable of running on standard laptops. The competition focused on promptable medical image segmentation, using a newly curated dataset of over 4,000 cases encompassing nine common imaging modalities from 24 institutions, ensuring a robust evaluation platform.
The competition had three phases: development, testing, and post-challenge. Participants trained their models on a large-scale dataset during development, using an online leaderboard for refinement. The top 20 teams submitted Dockerized algorithms for evaluation on a hidden testing set during the testing phase, ranked by Dice Similarity Coefficient (DSC), Normalized Surface Distance (NSD), and runtime. The post-challenge phase focused on performance boosting and reproducibility, incorporating new datasets and strategies from top-performing algorithms.
Top algorithms employed SAM-like architectures modified for efficiency. Key innovations included replacing the computationally intensive Vision Transformer (ViT) image encoder with lighter alternatives like EfficientViT and RepViT, often combined with knowledge distillation. Further optimizations included embedding caching for 3D images, C++ inference pipelines, and OpenVINO integration. Leading models achieved segmentation over ten times faster than existing SAM-based models, with some achieving NSD near 0.9 and runtime under 2 seconds for 2D images. The top two algorithms were integrated into the open-source 3D Slicer platform.
The post-challenge phase yielded further improvements, with one algorithm surpassing the previous best by using a shared EfficientViT model for all 3D modalities and incorporating C++ implementations. This refined algorithm achieved a 2x runtime reduction while maintaining or improving accuracy. Reproducibility analysis confirmed the winning solution's robustness. While the competition demonstrated significant progress, limitations remain, including limited geographical representation in the dataset and the focus on 2D models for 3D data with bounding box prompts. Future iterations aim to address these by incorporating more diverse datasets and introducing tasks focused on interactive and text-based segmentation.
Conclusion: A Multimodal Future
This newsletter showcases the vibrant and rapidly evolving field of multimodal image and text foundation models. We've seen how researchers are tackling key challenges, from generating high-quality training data to improving inference efficiency and bridging the gap between AI-generated and natural images. The development of new benchmarks and datasets, like those presented in this newsletter, are crucial for driving progress and ensuring that these powerful models can be effectively deployed in real-world applications, ranging from medical image analysis to robot control. The trend towards more efficient and accessible models, exemplified by the Efficient MedSAMs competition, is particularly encouraging, paving the way for wider adoption and impact in various domains. The ongoing research highlighted in this newsletter points towards a future where AI systems seamlessly integrate vision and language, enabling richer human-machine interactions and unlocking new possibilities across diverse fields.