UV9256 – Computerized Adaptive Testing
Schedule, syllabus and examination date
Course content
Why ask a person to answer a problem item, when you a priori know they won’t be able to solve it? It is a waste of time and resources, and you won’t gain any new information; this is both inefficient and ineffective. In contrast, computerized adaptive testing (CAT) is based on the principle that more information can be gained when one tailors the test towards the level of the person being tested. Computational and statistical techniques from item response theory (IRT) and decision theory are combined to implement a test that can behave interactively during the test process and adapts towards the level of the person being tested.
The implementation of such a CAT relies on an iterative sequential algorithm that searches the pool of available items (a so-called item bank) for the optimal item to administer based on the current estimate of the person’s level (and optional external constraints). The subsequent response on this item provides new information to update the person’s proficiency estimate. This selection-responding-updating process continues until specified stop criteria have been reached.
The consequence of such an adaptive test administration is that you get an individualized tailored test that is more efficient and more effective. Because you have less of a mismatch between the level of the test and the level of the test taker, there is a lesser burden for the latter and a higher precision for the former, and this with fewer items than a traditional fixed item-set test format. Furthermore, because it is computerized and sequential, test performance can be continuously monitored and reported directly after test completion. Item response models come into play to ensure comparable scores of these individual tailored tests by putting them on the same measurement scale and to pre-calibrate the psychometric parameters of the items that are part of the item bank on which the sequential iterative algorithm operates.
Learning outcome
The workshop intends to tackle issues encountered during the setup of a computerized adaptive test, starting from the design towards the actual delivery of a CAT.
Admission
Participation is free of charge and is open to all, with a maximum of 30 participants. Depending on participation numbers, we might need to disappoint some applicants as priority will be given to students and participants applying from the Nordic region.
Ph.d.-students from the University of Oslo apply through Studentweb. Other apply though Nettskjema.
Registration deadline: April 4, 2016
Teaching
Dates: 25, 26, 27 April 2016
Location: Seminarrom U31/232, Helga Engs Hus
Time: 09.00-16.00 all days
Lecturer: David Magis, David Stillwell, Johan Braeken
Throughout the workshop we will be using R (https://www.r-project.org/), the free open-source software environment for statistical computing and graphics, in combination with the R-package catR developed by David Magis and the Concerto online testing platform developed by the Cambridge Psychometric Centre.
Program Outline
-
Day 1:Introduction to IRT & Computerized Adaptive Testing:Conceptual Level
-
Day 2: Computational aspect and implementation of IRT & CAT in R
-
Day 3: CONCERTO showcase & development of your own example CAT
The same central topics will be covered at each day of the workshop, but each time from a slightly different angle: starting conceptually on the first day to provide a meaningful basis, proceeding towards the computational aspects on the second day to get more detailed understanding, and finally on the third day putting it all together in the practical implementation of a CAT.
Re-occurring topics for item response theory throughout the workshop:
-
Common assumptions & core models
-
Model estimation: Item parameters & Person proficiency
-
Item and Test Information
-
Item bank
Re-occurring topics for computerized adaptive testing throughout the workshop
-
Core Principles & Requirements
-
Decision theory:sequential testing and decision rules
-
Design & Algorithm choices
-
selection of start items
-
Next-item selection
-
Stopping rules
-
External constraints: Item exposure control & Content balancing
-
Examination
To obtain 1 study point 80% attendance in the lectures is required.
To obtain 3 study points a short paper needs to be submitted after the
course.
Grading scale:
Grades are awarded on a pass/fail scale.