Machine Learning

In this lecture, we will look at support vector machines. We will first look at the connection between maximizing margins and learning linear classifiers. This will give us an objective function for training, namely the SVM objective. This objective is our first sight of the idea of regularized risk minimization.

There are several algorithms for optimizing the SVM objective. We will look at a simple, yet effective, one: stochastic sub-gradient descent and explore the connection with the perceptron algorithm.

Lectures

• Introduction to SVMs

• SVMs: Training with Stochastic Gradient Descent

• Videos:
  • Introduction to SVMs: lecture 1, lecture 2
  • SGD for SVMs
• Older videos:
  • Spring 2024:
    • Introduction to SVMs: lecture 1, lecture 2,
    • SGD for SVMs: lecture 1, lecture 2
  • Previous semesters
    • Introduction to support vector machines (1/2)
    • Introduction to support vector machines (2/2)
    • SGD for SVMs

Readings

• Chapters 3 and 6 of Hal Daumé III, A Course in Machine Learning

• Chapters 14, 15, 16 of Shai Shalev-Shwartz and Shai Ben-David, Understanding Machine Learning: From Theory to Algorithms (Available online)

Additional reading

• Notes on the stochastic subgradient method

• Convex functions

• Stochastic gradient descent tricks

• A talk on Optimization, Support Vector Machines, and Machine Learning that goes into the details of primal and dual forms of SVMs and optimization.