Qwen VL Fine-tuning for AI City Challenge 2026 Track 2

AI City Challenge Track 2 required video-caption fine-tuning on 1 V100 with 14 GB of memory. Qwen2.5-VL offered a strong baseline, yet needed training for this task. Therefore, I needed to fit useful training within the hardware limit.

Full fine-tuning could not fit the model and training state in 14 GB. Moreover, cluster networking ruled out multi-GPU training. Multiple video frames made each sequence too large even with memory-saving settings. Meanwhile, conflicts between system libraries and project versions delayed training.

I used LoRA to train small added weights while freezing most of the model’s 3 billion parameters. This reduced training memory. Moreover, I extended the context window for longer visual sequences. I enabled gradient checkpointing, which recomputes intermediate values to save memory. Together, these changes made training fit on 1 GPU. The pipeline also supports training across multiple GPUs when available.

I changed the training data format to reduce memory use per step. Originally, each example contained multiple video frames. The resulting sequences filled GPU memory before training could start. Therefore, I changed the format to use 1 frame per example. This preserved the task’s meaning and kept each training step within memory limits. As a result, the fine-tuning run could proceed.

I isolated project dependencies from the cluster’s system libraries and configured the GPU runtime. As a result, I removed version conflicts and made runs reproducible on the shared cluster.

I automated data preparation, configuration, and training in 1 workflow. This reduced setup for each experiment. Moreover, the workflow can transfer to other tasks with limited GPU access. Training stayed stable and converged within the 14 GB budget.