MA333      Half Unit
Optimisation for Machine Learning

This information is for the 2025/26 session.

Course Convenor

Availability

Requisites

Additional requisites:

Pre-requisites: Students should be familiar with the fundamentals of continuous optimisation, to the level in Optimisation Theory (MA208) or equivalent.

Machine learning uses tools from statistics, mathematics, and computer science for a broad range of problems in data analytics. The course introduces a range of optimisation methods and algorithms that play fundamental roles in machine learning. This is primarily a proof-based course that focuses on the underlying mathematical models and concepts. The secondary goal of the course is to demonstrate implementations of the discussed algorithms on problems from machine learning, their limitations on large training sets, and how to overcome such obstacles.

After a review of basic tools from convex analysis, Lagrangian duality, and Karush-Kuhn-Tucker conditions, the course makes a deep dive into first-order optimisation methods, and a light touch on second-order methods, and their convergence guarantees. The first-order methods include projected, conditional and stochastic gradient descent, backpropagation in neural networks, and mirror descent. Newton's method from second-order optimisation is also covered. The course also considers online convex optimisation, and covers online gradient and multiplicative weights update methods.

A key component of the course is the application of optimisation methods to machine learning. As such, we will see applications of the methods taught to linear regression, ridge and lasso regularization, logistic regression, binary classification and support vector machines, multi-layer perceptrons, and online learning algorithms such as Perceptron and Winnow. A key learning outcome is how to solve such problems in the presence of large training sets.

2 hours of classes in the Spring Term.
20 hours of lectures and 10 hours of classes in the Winter Term.

During the lectures, the focus will be on the optimisation methods and their convergence guarantees. During the classes, in addition to discussing the exercise sheets, implementations of the methods will be shown and their effectiveness on large training sets will be discussed.

Students will be expected to submit solutions to 3 exercise sheets in the WT.

• Vishnoi, N. (2018). Algorithms for Convex Optimization (2021). Cambridge University Press.
• Boyd, S., & Vandenberghe, L. (2004). Convex optimization. Cambridge University Press.
• Nesterov, Y. (2018). Lectures on convex optimization (Vol. 137). Springer.
• B. Gärtner and M. Jaggi. Optimization for machine learning (lecture notes), 2021.
• E. Hazan. Introduction to online convex optimization (lecture notes), 2021.
• I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning. MIT Press, 2016.

Exam (90%), duration: 120 Minutes in the Spring exam period

Continuous assessment (10%)

A combination of the weekly exercises (set and marked in Winter Term) count as the continuous assessment.