UV-JEPA
UV-JEPA is an ongoing final-year research project investigating whether latent-predictive vision-language architectures can learn effective multimodal representations in low-resource language regimes—using Urdu as the representative benchmark—with high data efficiency and without conventional autoregressive decoding.
Overview & Research Motivation
Vision-language learning has increasingly evolved from token-level autoregressive generation or pixel reconstruction toward learning shared, continuous representations in latent space. Modern Joint-Embedding Predictive Architectures (such as I-JEPA, VL-JEPA, and TC-JEPA) demonstrate that predicting representations rather than decoding discrete tokens reduces error compounding and improves representation quality.
However, most multimodal foundation models are benchmarked almost exclusively on high-resource English datasets with tens of millions of curated image-text pairs. When applied to low-resource languages, standard models struggle due to severe data scarcity, morphological richness, and caption variance.
UV-JEPA investigates whether latent-predictive vision-language learning remains effective in low-resource settings, using Urdu as the primary testbed. The central hypothesis is that predicting a compact sentence representation from an image representation in latent space provides competitive vision-language alignment while requiring substantially less linguistic supervision than conventional autoregressive caption generation.
Methodology & Architecture
The UV-JEPA framework formulates vision-language learning as a non-generative, latent-predictive task:
-
Vision & Language Encoders: Uses a Vision Transformer (ViT-Small) for visual feature extraction alongside a pretrained language encoder (based on
urduhack/roberta-urdu-small) to process target text descriptions. - EMA Target Text Encoder: The target text representation is generated by an Exponential Moving Average (EMA) encoder with stop-gradient, while the online text encoder updates jointly to provide the source for the target representation.
- Lightweight Cross-Attention Predictor: Maps visual representations into the shared latent space and is trained to predict the corresponding continuous caption representation using normalized latent regression.
- Rigorous Baseline Benchmarks: UV-JEPA is evaluated directly against rigorously tuned contrastive (CLIP-style) and autoregressive decoders under identical, constrained data regimes to measure data efficiency.
Evaluation & Empirical Focus
The project's principal contribution is an empirical study evaluating whether latent prediction offers measurable advantages when paired training data is scarce:
- Cross-Modal Retrieval: Bi-directional image-to-text and text-to-image retrieval performance across multiple data scales.
- Representation Quality & Probing: Semantic caption similarity, feature stability, and probing of learned latent spaces.
- Data Efficiency & Computational Cost: Benchmarking scaling behavior, parameter efficiency, and training overhead against baselines.
- Robustness: Evaluating sensitivity to caption quality variations and linguistic noise inherent to low-resource web collections.
Status & Academic Supervision
Note on research claims: The study is designed to empirically discover the boundaries and data-efficiency limits of latent prediction in low-resource multimodal settings. Rigorously documenting where and why the approach succeeds—or fails—relative to contrastive and autoregressive baselines constitutes the valid scientific thesis outcome. Code and checkpoints will be documented upon thesis defense.