Plans for week 38, September 14-18

Dear all, we hope the weekend has started the best possible way and that you have found  time to relax and do other things. We also wanted to thank you all for great efforts with the weekly exercises and all the truly enjoyable discussions during the flipped classroom sessions on Tuesdays and obviously the Wednesday labs.  Keep up the good work. This coming week we will continue our discussion on the optimization part and try to include a discussion on how we can update the learning rate (cheaply, almost) through methods like ADAM, ADAgrad, RMSprop and other as well as introducing the standard recipe we will use with neural networks and all other methods, that is stochastic gradient descent.  But first we will try to demystify automatic differentiation, the algorithm which saves our day and is so central in all of ML and advanced LLMs applications and more.

Our plan for this coming week looks as follows:

Monday's lecture:

▶ First 10 min: gradient descent with and without momentum, once more following the steps on

a low-order polynomial (Sections 4.3, 4.5 and 4.6; week38.ipynb), basically a reminder from last week

▶ Automatic differentiation: the closest we get to painless calculations, the gradient without deriving it, forward and reverse mode, JAX on

last week's problems (Section 4.14 moved here from week 37, where we ran out of time)

▶ Why one learning rate cannot fit all directions, and the three adaptive methods: AdaGrad,

RMSProp and Adam (Sections 4.84.12)

▶ Stochastic gradient descent, combining everything with minibatches, epochs, what the noise means, learning-rate schedules

and stopping (Section 4.7)

Tuesday session: your questions (please don't forget to send us your questions), then two cases AdaGrad, RMSProp and Adam

in your own code, on the Runge problem of Project 1; and stochastic gradient descent with

minibatches, epochs and schedules. Bring your laptop.

Reading recommendations:

Chapter 4, Sections 4.74.14 of the book. Goodfellow et al., Sections 8.18.5; Kingma and Ba, Adam:

a method for stochastic optimization (2014).  Week 39: Lasso by gradient methods (last ingredient needed in project 1) and start logistic

regression (Chapter 5). We will use logistic regression (for classification problems) as a stepping stone towards neural networks and deep learning.

This week's topic is really tailored to us finalizing the last parts of project 1. The weekly exercises exercisesweek38.ipynb encode the above methods, and with these, we have most of the technicalities we need for project 1. 

As always, the weekly material for Monday and Tuesday, and the exercises are in the folder https://github.com/EducationalMaterialUiO/MachineLearningUiO/tree/main/doc/WeeklyMaterial/week38

Wishing you all a great weekend,

Amanda, Amund, Brit, Kim, Morten and Viktor