Plans for week 37: September 7-11

Dear all, we hope your weekend has evolved in the best possible way.

This week we will revisit Cross-Validation and after that we will dive into the optimization part, which is the second half of project 1. The exercises this week and next week can be  immediately reused in project 1 and they cover the optimization we will ned in projects 2 and 3 as well.

The plans this coming week are:

Monday lecture (2×45 min):

▶ First 30 min: cross-validation, once more and properly – the k-fold loop, the leak, and the

selection recipe of Section 3.14 (grid, standard error, one-SE rule, refit, test set once). Last week

we ran out of time here, and Project 1 parts d) and i) need it.

▶ Why we must optimize numerically; convexity and Newton’s method (Sections 4.1–4.2)

▶ Plain gradient descent, applied to OLS and Ridge where we know the answer (Sections 4.3–4.4)

▶ The learning rate and the condition number: the one number that governs everything (Section

4.5)

▶ Momentum (Section 4.6)

▶ Automatic differentiation: the gradient without deriving it, with JAX applied to the same

problems (Section 4.14)

Tuesday session (2×45 min, flipped): your questions, then two cases – your own gradient descent

code for OLS and Ridge (analytical gradients, learning rate, stopping), and momentum plus automatic

differentiation on the Runge function of Project 1. Bring your laptop.

Reading

Chapter 3, Section 3.14 and Chapter 4, Sections 4.1–4.6 and 4.14 of the book. Goodfellow et al.,

Chapter 4 and Sections 8.1–8.3; Hastie et al., Section 7.10. Week 38: we will add the final elements to our optimization codes: stochastic gradient descent,

AdaGrad, RMSProp and Adam, with Lasso and logistic regression as applications.  Next week will thus also start with our first discussion of classification approaches, using logistic regression, before we start with neural networks and deep learning in week 39.

If you have questions for the Tuesday active learning session, please send them to us, either in person or via email/sms.

The material for this week is to be found (PDFs and jupyter-notebooks) at https://github.com/EducationalMaterialUiO/MachineLearningUiO/tree/main/doc/WeeklyMaterial/week37

Best wishes to you all,

Amanda, Amund, Brit, Kim, Morten, and Viktor