Visual question answering & reasoning over vision & language: Beyond limits of statistical learning?

Advances in deep learning keep producing impressive results at the junction of computer vision and natural language processing. The task of visual question answering (VQA), once considered incredibly ambitious, is now commonly used to benchmark multimodal models. Despite apparent progress, however, I will argue that some capabilities required for a general solution to VQA, such as strong out-of-distribution generalization, are beyond the reach of prevailing practices in machine learning. I will discuss how causal reasoning helps in formalizing the limits of classical, correlation-based learning. We will use a new layer of understanding of existing techniques to identify what information is missing from typical datasets, where else to find it, and how to test our models for the behaviors we really care about. Speaker: Damien Teney, Idiap Research Institute in Switzerland MSR Deep Learning team: