Fine-tuning Vision-Language Models
Adapt multimodal models like LLaVA and PaliGemma for domain-specific visual understanding, document parsing, and image captioning.
About this Course
Adapt multimodal models like LLaVA and PaliGemma for domain-specific visual understanding, document parsing, and image captioning. This AI/ML & Computer Vision curriculum is designed to give you hands-on experience and deep conceptual understanding.
Across 4 intensive modules, you'll tackle real-world challenges and build practical projects that reinforce your learning. By the end of this journey, you'll have the skills and proof of work to demonstrate your expertise.
What you'll learn
Master the core concepts of vision-language model architectures.
Gain hands-on experience with dataset preparation & augmentation for vlm.
Understand the architecture behind parameter-efficient fine-tuning (peft/lora).
Implement production-grade evaluation & deployment of custom vlms.
W1
Vision-Language Model Architectures
Vision-Language Model Architectures
Master the core concepts of vision-language model architectures.
4 videos•51m
3 readings
4 topics
1 homework
Topics
1.1
How AI 'Understands' Images (CLIP) - Computerphile
Image Encoders (ViT, SigLIP) & Projection Layers
18 minutes
1.2
Vision-Language Models Explained | CLIP, DALL·E, Florence & Multimodal AI
Multimodal Fusion Strategies
8 minutes
1.3
Vision-Language Models (VLMs) Explained | GPT-4V, LLaVA & CLIP
LLaVA vs PaliGemma Architectures
8 minutes
1.4
Stop Using GPUs for LLMs : Fine-Tune Gemma 3 with JAX on a TPU (Fast & Cheap)
Tokenizing Images and Text
17 minutes
References
Week 1: Vision-Language Model Architectures
Week 2: Dataset Preparation & Augmentation for VLM
Week 3: Parameter-Efficient Fine-Tuning (PEFT/LoRA)
Rate this course
Community Insights
0Join the discussion
Sign in to share your thoughts and technical insights.
Loading insights...