W1

Module 1: Foundations of Multimodal AI & Text-Vision Fusion

By the end of this module you will be able to understand the core concepts of multimodal AI and integrate text and vision models to create applications that interpret and generate content based on both modalities.

4 videos

3 readings

4 topics

1 homework

Learn

Topics

1.1

Introduction to Multimodal AI

AI That Sees Video & Generates Match Sound | AudioX

1.2

Text and Vision Embeddings

MultiModal Search (Text+Image) using TF MobileNet , HF SBERT in Python on Kaggle Shopee Dataset

1.3

Integrating Text and Vision Models

Large Multimodal Models Are The Future - Text/Vision/Audio in LLMs

1.4

Practical API Usage for Vision and Text Models

Implementing a Practical Vision-Based Android AI Agent

Required Reading & References

W2

Module 2: Advanced Multimodal Integration: Audio & Beyond

By the end of this module you will be able to integrate audio processing with text and vision models, and design more complex multimodal applications that leverage multiple input types for richer context and interaction.

4 videos

3 readings

4 topics

1 homework

Learn

W3

Module 3: Video Understanding and Generation

Understand how to process video data temporally and spatially, and apply multimodal concepts to video generation and search.

3 videos

2 readings

3 topics

1 homework

Learn

W4

Module 4: Autonomous Multimodal Agents

Build autonomous agents capable of interacting with visual interfaces and reasoning across modalities.

2 videos

2 readings

2 topics

1 homework

Learn

Multimodal AI Integration

Module 1: Foundations of Multimodal AI & Text-Vision Fusion

Module 1: Foundations of Multimodal AI & Text-Vision Fusion

Topics

Required Reading & References

Module 2: Advanced Multimodal Integration: Audio & Beyond

Module 2: Advanced Multimodal Integration: Audio & Beyond

Module 3: Video Understanding and Generation

Module 3: Video Understanding and Generation

Module 4: Autonomous Multimodal Agents

Module 4: Autonomous Multimodal Agents

References

Week 1: Module 1: Foundations of Multimodal AI & Text-Vision Fusion

Vision Language Pre-training on Hugging Face

Learning Transferable Visual Models From Natural Language Supervision (CLIP)

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

Week 2: Module 2: Advanced Multimodal Integration: Audio & Beyond

Robust Speech Recognition via Large-Scale Weak Supervision (Whisper)

A complete guide to Audio Datasets

Recent Developments in Speech Recognition

Week 3: Module 3: Video Understanding and Generation

Stable Video Diffusion Release

Sora: Creating video from text

Week 4: Module 4: Autonomous Multimodal Agents

WebVoyager: Building an End-to-End Web Agent

Gemini: A Family of Highly Capable Multimodal Models

Community Insights

Join the discussion