IN5640 – Developing Large Language Models

Upon completion of the course you:

• understand the science behind developing state-of-the art large language models;
• are able to find and critically read relevant academic papers;
• know how to plan the training of different types of LLMs, depending on their intended use, computational and data constraints;
• have the technical skills allowing you to train a large language model from scratch;
• understand common approaches to LLM evaluation and their limitations;
• can design, conduct, analyze, and describe large-scale experiments with modern LLM training frameworks.

Students admitted at UiO must apply for courses in Studentweb. Students enrolled in other Master's Degree Programmes can, on application, be admitted to the course if this is cleared by their own study programme.

If you are not already enrolled as a student at UiO, please see our information about admission requirements and procedures for international applicants.

The class presupposes a solid understanding of the foundations of deep learning and the transformer architecture. It is recommended that students have completed either IN5550 – Neural Methods in Natural Language Processing or IN4310 – Deep Learning for Image Analysis, or equivalent.
For students with no prior exposure to machine learning for natural language, it may also be an advantage to have completed IN4080– Natural Language Processing, or another introduction to NLP.

Finally, the class is applied and practical. Programming experience in Python will be presupposed.

Four hours of instruction per week, mostly split into two hours of lectures and reading groups and another two hours with hands-on workshops. Strong focus on self-supervised work involving reading and discussing recent literature on the topic.

One practical group project unfolding throughout the course and leading to the exam (developing a specific type of LLM from scratch).