Deep Learning 2.0: How Bayesian Optimization May Power the Next Generation of DL by Frank Hutter

A Google TechTalk, presented by Frank Hutter, 2022/6/14 ABSTRACT: BayesOpt TechTalk Series. Deep Learning (DL) has been incredibly successful, due to its ability to automatically acquire useful representations from raw data by a joint optimization process of all layers. However, current DL practice still requires substantial manual efforts to define the right neural architecture and training hyperparameters for the data at hand. The next logical step is to jointly optimize these components as well, based on a meta-level of learning and optimization. I predict that this will allow the next generation of DL systems to simply accept data and user objectives to optimize for (which can, e.g., include fairness, robustness, uncertainty calibration, interpretability, etc) and to thereby provide a clean interface between domain experts (who best know the data and the relevant objectives for the application at hand, but do not need to be machine learning experts) on the one hand and the next-generation DL system on the other hand. In this talk, I will discuss several advances towards this goal based on Bayesian optimization, focussing on (1) the joint optimization of several meta-choices in the DL pipeline and (2) the efficiency of this meta-optimization. Bio: Frank Hutter is a Full Professor for Machine Learning at the University of Freiburg (Germany), as well as Chief Expert AutoML at the Bosch Center for Artificial Intelligence. Frank holds a PhD from the University of British Columbia (UBC, 2009) and a Diplom (eq. MSc) from TU Darmstadt (2004). He received the 2010 CAIAC doctoral dissertation award for the best thesis in AI in Canada, and with his coauthors, several best paper awards and prizes in international competitions on machine learning, SAT solving, and AI planning. He is a Fellow of ELLIS and EurAI and the recipient of 3 ERC grants. Frank is best known for his research on automated machine learning (AutoML), including neural architecture search and efficient hyperparameter optimization. He co-authored the first book on AutoML and the prominent AutoML tools Auto-WEKA, Auto-sklearn and Auto-PyTorch, won the first two AutoML challenges with his team, co-organized the ICML workshop series on AutoML every year 2014-2021, and is the general chair of the inaugural conference on AutoML 2022.