Cohere Introduces Aya Vision: Multilingual AI Unlocks Visual Understanding in 23 Languages

• Architecture & Technical Innovations
  • Core Components:
    • Uses SigLIP2-patch14-384 vision encoder and Pixel Shuffle downsampling (4x image token compression).
    • Aya Vision 8B employs Cohere Command R7B; 32B version leverages Aya Expanse 32B for multilingual proficiency.
  • Multimodal Capabilities:
    • Performs image captioning, visual Q&A, text translation, and multilingual summarization across 23 languages.
• Training Pipeline
  1. Vision-Language Alignment: Freezes encoder/decoder weights while training the connector.
  2. Supervised Fine-Tuning (SFT):
     • Trains connector and LLM on 23 languages, boosting multilingual task accuracy by 17.2% (40.9% → 58.1% win rates).
     • Uses synthetic annotations generated by translating English datasets and AI-generated labels, reducing reliance on manual data.
     • Efficiency Focus: Achieves competitive performance with fewer resources, addressing compute constraints for researchers.
       • Synthetic data now constitutes ~60% of training data (per Gartner), reflecting industry shifts as real-world data scarcity grows.
• Benchmark Leadership
  • Aya Vision 8B:
    • 79% win rate vs. Qwen2.5-VL 7B and Gemini Flash 1.5 8B on AyaVisionBench; 81% on mWildVision.
  • Aya Vision 32B:
    • Outperforms Llama-3.2 90B Vision (2x larger) with 63% (AyaVisionBench) and 72% (mWildVision) win rates.
  • Evaluation Innovation:
    • AyaVisionBench tackles the AI industry’s “evaluation crisis” by providing a multilingual, real-world task framework (e.g., screenshot-to-code conversion, image difference detection).
• Multimodal Model Merging
  • Merging vision-language and base language models improves conversational task performance by 11.9% (70% win rates).
  • Enhances text-only results on mArenaHard via cross-modal knowledge transfer.
• Open-Source Contributions
  • Released under Creative Commons 4.0 license (non-commercial use) with Cohere’s acceptable use addendum.
  • AyaVisionBench:
    • Multilingual evaluation suite with 9 task categories and 135 image-question pairs per language, validated by human annotators.
  • Accessible via Hugging Face, WhatsApp, Hugging Face Space demo, and Colab tutorial.

Why This Matters

Aya Vision bridges the multilingual performance gap in AI, particularly for low-resource languages, while challenging flawed benchmarking practices. By open-sourcing models and evaluation tools, Cohere democratizes access to state-of-the-art multimodal AI, empowering researchers to innovate in education, cross-cultural communication, and real-time visual analysis. The emphasis on synthetic data efficiency and non-commercial licensing reflects a strategic balance between scalability and ethical AI development, setting a precedent for sustainable, community-driven advancements in a data-constrained world.