Modeling and Optimization for Machine Learning

Optimization algorithms lie at the heart of machine learning (ML) and artificial intelligence (AI). The distinctive feature of optimization within ML is the strong departure from textbook approaches: the focus is now on a different set of goals driven by big data, non-convex deep learning, and high-dimensions. This departure and the different focus make it challenging for newcomers and even experienced users to obtain a solid grasp of the fundamental ideas without getting lost in myriad tutorials, blogs, and papers. 

This course provides an accessible entry point to Modeling and Optimization for Machine Learning, key skills needed to use state-of-the-art software and algorithms from machine learning. It covers underlying theoretical motivations behind widely-used optimization algorithms (the “science”), while diving deep into aspects of mathematical modeling (the “art”) to provide students with an intuitive, foundational introduction to this modern and fast-moving research area.

Modeling reduces messy engineering or computational problems to mathematical forms that can be solved by using standard software and techniques. By recognizing mathematical patterns “in the wild,” participants will develop an intuition for which problems are solvable using standard numerical modeling techniques and gain the knowledge and skills to then solve them in practice.

After we develop an appropriate model for a machine learning problem, the next step is to choose an optimization technique. Participants in the course will learn to pair mathematical models with efficient optimization algorithms, from stochastic gradient descent to cone programming. Participants will delve into the details of how popular optimization methods work and will receive practical experience interfacing with optimization software through case studies and exercises.

By the end of the course, participants will learn how to boil real-world challenges down to their computational essence to make a reasonable estimate of how difficult it would be to design a numerical method to solve them. We will cover a breadth of tools, from numerical linear algebra to convex programming and stochastic/deterministic gradient descent, in the context of practical problems drawn from emerging applications in learning, computer vision, time series analysis, and imaging. Coding and mathematical exercises will reinforce these ideas and expose participants to standard software packages for optimization.

COVID-19 Updates

We fully expect to resume on-campus Short Programs courses during the Summer of 2022. However, the possibility remains of ongoing disruption and restrictions due to COVID-19 which may require that the course be delivered via live virtual format. Please read more here.

Participants in the course will learn how to:

• Recognize classes of optimization problems in machine learning and related disciplines.
• Learn concepts that demystify the “why” and “how” of ubiquitous topics such as regression, deep learning, and large-scale optimization, with a focus on convex and non-convex models.
• Interface with software for computing optimal solutions to a given machine learning problem.
• Understand the mathematical underpinnings of optimization methods via examples drawn from machine learning, computer vision, engineering, and data analysis.
• Understand foundational optimization ideas including gradient descent, stochastic gradient methods, higher-order methods, and more advanced optimization algorithms.
• Classify optimization problems by their tractability, difficulty, and compatibility with existing software.
• Learn to cut through the hype to make more informed choices for their own applications.

Program Outline

The course begins with the fundamentals of modeling and optimization, including case studies converting regression and classification problems to mathematical models as well as the basics of deterministic and stochastic gradient descent. We then broaden the capabilities of our modeling language by showing how to incorporate constraints and accelerate optimization with second-order information. After establishing the basics, we consider a variety of more advanced models in machine learning, including neural network training, sparsity and low-rank regularization, metric learning, time-series analysis, and adversarial training of robust models. We conclude with practical discussion drawn from research projects at MIT as well as from participants’ domain areas.

Our course will include daily practicum exercises in which students will experiment with optimization tools applied to modeling problems drawn from machine learning applications.  We will complete the exercises using Google Colab in Python.

Links & Resources

• Improving global health equity by helping clinics do more with less, MIT News, June 25, 2020
• Smoothing out sketches’ rough edges: MIT-developed tool improves automated image vectorization, saving digital artists time and effort, MIT News, September 11, 2018

This course is designed for people working in data science, finance, marketing, computer-aided design, operations, strategy, engineering, research, or computer vision. Typical roles include engineer, programmer, developer, data scientist, researcher, consultant, or marketing analyst.

Relevant roles include:

• Computer engineers, programmers, and developers seeking to leverage leading-edge machine learning technologies and algorithms 
• Data scientists who want to use advanced tools, such as numerical linear algebra and convex programming, to improve decision-making and results 
• IT and business leaders looking to identify the machine learning strategies that will be most effective for meeting their unique organizational needs 
• Academics and researchers who need a deeper understanding of the latest developments in machine learning modeling and optimization 
• Financial professionals who use—or want to use—models to more accurately predict risk and shape decision making 

Requirements

Participants are required to have a background in linear algebra and multivariable calculus, as well as at least basic programming in Python.

Laptops (or tablets) with Python are required for this course. Participants should have administrative privileges for their computers in case Python packages need to be installed during the course.