Qwen VL Fine-tuning for AI City Challenge 2026 Track 2

AI City Challenge Track 2 required video-caption fine-tuning on one V100 with 14 GB of memory. Qwen2.5-VL offered a strong baseline but needed task adaptation. The main engineering problem was fitting useful training within the hardware limit.

Full fine-tuning could not fit the model and training state in 14 GB. Cluster networking also ruled out multi-GPU training. Multiple video frames made each sequence too large even with memory-saving settings. Conflicts between system libraries and project versions delayed training further.

I used LoRA to update a small part of the model while leaving most of its three billion parameters frozen. This cut training memory. I extended the context window for longer visual sequences and enabled gradient checkpointing, which recomputes intermediate values to save memory. Together, these changes made training fit on one GPU. The pipeline also supports multi-GPU scaling when available.

Redesigned the training data format to reduce memory consumption per training step. The original dataset structure packed multiple video frames per example, producing sequences that saturated GPU memory before training could begin. Reformatting to a single-frame-per-example structure preserved the task semantics while making each training step memory-safe, a data engineering decision that unblocked the entire fine-tuning run.

I isolated project dependencies from the cluster's system libraries and configured the GPU runtime. This removed version conflicts and made runs reproducible on the shared cluster.

I automated data preparation, configuration, and training in one workflow. It reduced setup for each experiment and can transfer to other limited-GPU tasks. Training remained stable and converged within the 14 GB budget.