Annual Report 2020

After the startup phase, we had planned that 2020 should be a consolidation year at RITMO. Unfortunately, the pandemic changed all our plans.

Closed labs, cancelled experiments, postponed activities, guests unable to come to Oslo, researchers that could not travel elsewhere. The year 2020 was different for everyone, including all of us working at RITMO.

Fortunately, we had already invested in video conferencing solutions before the pandemic and had a lot of internal know-how in organizing online activities. This meant that we quickly managed to convert most of our activities to online and hybrid formats. Several of RITMO's researchers also turned around to utilise the situation's unique research potential.

Despite our relative success in keeping the centre activities going, we all long for getting back to a more normal situation. Meanwhile, here we present some highlights from 2020, in rhythm, time, and motion. 

Anne Danielsen and Alexander Refsum Jensenius

Anne Danielsen and Alexander Refsum Jensenius
Table of Contents
Black and white photography of dancer

RITMO at a Glance

Rhythm is everywhere, from how we walk, talk, dance and play, to how we tell stories about the past and predict the future.

Our heartbeat, brain waves, and other bodily cycles all work through rhythm. Rhythm is a crucial aspect of human action and perception. Our human rhythm is also in complex interaction with the world's various cultural, biological and mechanical rhythms. 

At RITMO, we research rhythmic phenomena and the complex relationships between the human body's rhythms and the brain. The central idea is to establish a link between features of rhythmic phenomena in the world and within the (embodied) mind. The aim is to understand our ability to perceive rhythm and how this affects our actions and experiences.

RITMO is a highly interdisciplinary centre, combining perspectives and methods from music and media studies, philosophy and aesthetics, cognitive neuroscience, and informatics. We employ several state-of-the-art technologies in our research, including motion capture, neuroimaging, pupillometry and robotics.

Structure and Time

Structure & Time Cluster

We investigate the structure and aesthetics of musical rhythm and their link to perceptual, cognitive and motor mechanisms. We are also interested in the role of rhythm, temporal structure and timing in the experience of long-lasting musical form and more fragmented, audiovisual musical formats. Overarching questions are how genre familiarity, technology and socio-cultural context interact with perceptual and cognitive mechanisms in the experience of rhythm and time.

Highlights

  • Timing one’s playing to a beat is something that most musicians practice. In a series of performance experiments, PhD fellow Guilherme Schmidt Câmara and colleagues showed that musicians also use features other than pure timing. Intensity, timbre and duration are all used to communicate whether a beat is played early, behind or on-the-beat. Studies of instructed timing in guitar performance (JASA) and drumming (Music Perception) support previous work showing that perceived timing at the micro-level of rhythm is closely related to the sound's shape.
  • Olivier Lartillot and collaborators have developed a new method for computational analysis of sound (JASA). Previous attempts, based on the reduction of the audio waveform into an envelope curve, decreases its temporal resolution. The proposed approach detects the attack region directly from the audio waveform and shows a significant increase in precision.
  • The seminar Muddy Rhythms and Broken Beats gathered a group of scholars to focus on experimental grooves within hip-hop music, neo-soul, urban and electronic dance music. They discussed questions such as: How do producers experiment with rhythm and groove via the use of processing effects, sequencer tools and samples? What is the relationship between interface design and groove aesthetics? And how do listeners make sense of muddy rhythms and broken beats? By identifying “seasick-, fragmented-, cut-up-, staggering and irregular rhythms” as a topic and gathering pioneering scholars within this field, RITMO has laid the ground for a new subfield within rhythm studies.  

Musical Microrhythm

The project TIME: Timing and Sound in Musical Microrhythm investigates interactions between temporal and sound-related features at the micro-level of rhythm in musical genres where rhythm is a core dimension. The aim is to gain new insights into the micro-level of auditory perception and the role of cultural background and/or training in music in this regard. This is based on qualitative and quantitative methodologies from musicology, ethnomusicology, music psychology and motion research.

Musical Fragmentation

This project examines the pleasure of musical fragmentation and the related play with time, focusing on sample-based and cut-and-paste music. In the MASHED project, the focus is on the underlying aesthetics and temporality of so-called Mashup music

Musical Time and Form

The project Musical Time and Form investigates salient differences in musical interpretation between recorded performances of orchestral compositions in Western Art Music from the last two centuries. Compositions are performed differently depending on the conductor, orchestra, historical and cultural tradition, and other factors such as time and place of the performance. There is also a focus on the listening experiences of those performances, intending to improve the precision with which we discuss musical performance and the listening experience. 

A Comprehensive AI-based System for Advanced Music Analysis

The aim of the project MIRAGE - A Comprehensive AI-Based System for Advanced Music Analysis is to improve the computers' capability to listen to and understand music. A goal is to develop technologies that facilitate understanding and appreciation of music through generating rich and detailed descriptions. These encompass a broad range of dimensions, such as rhythm, tonality, formal structure, melodic shape, sound colour, and emotions. Significant effort is dedicated to designing applications of value for musicology, music cognition and the general public. One main focus is the study of Norwegian Hardanger fiddle music.

Structure and Cognition

Structure & Cognition Cluster

The goal is to expand our understanding of how rhythm and our sense of time are constructed in the human mind. We study the influence of rhythm and time on perception, attention and effort, imagery and illusions, prediction and cognitive control processes across sensory modalities.

Highlights

  • In an article about rhythmic predictability (Scientific Reports), Maja Foldal and colleagues reported on findings from scalp-EEG recordings in healthy adult participants who listened to rhythmic sound stimuli. Participants were instructed to pay attention to either the left or the right ear and ignore the sounds in the other ear. This is the first study to show that the brain predicts the timing of sounds even when they occur in the background and attention is directed elsewhere.
  • Using pupillometry and functional MR imaging, RITMO researchers investigated the mental effort of a professional pianist while playing, listening to, and imagining music (Frontiers in Neuroscience). They found that the motor activity of playing music increases mental effort, whereas levels of effort during imagery and listening are almost similar. Also, they discovered that the music's technical difficulty is reflected in activations of the brainstem’s locus coeruleus as well as in pupil diameters. This combined approach of psychophysiology and neuroimaging revealed the mental work going on during different musical activities.
  • Researcher Alejandro Blenkmann was awarded funding from the Research Council of Norway to uncover the neurophysiological basis of the brain’s ability to predict upcoming auditory events. To do this, he will be using intracranial EEG and single-neuron recordings. Single-neuron recordings in humans have not previously been conducted in Scandinavia. The findings are expected to expand our understanding of how the brain predicts future events. 

Mental effort and cognition

The Mental Effort and Cognition project investigates the use or consumption of attentional resources while carrying out a cognitive task. Mental effort can be measured behaviorally, subjectively and physiologically. Our approach focuses on a) the psychophysiological index provided by pupillometry, a well-established method for measuring physiological arousal, and b) brain activity with fMRI. We are, among other things, interested in the inherent rhythmicity of attention, effort related to different forms of temporal violation of regularity, and the effort used when processing information in different sensory modalities.

Multimodal perception of rhythm, time and illusory time

In the project Multimodal perception of rhythm, time and illusory time we try to understand how the brain keeps track of and makes meaning out of temporal information to produce stable percepts of the world. We use illusions (the Risset illusion and the flash-lag effect) to reveal how predictions can influence the sensory system and override the characteristics of actual physical stimuli. We also study how the brain makes sense of regular accelerations in tempo compared to irregular changes.

The neural basis of temporal prediction.

A primary goal of the project The neural basis of temoral prediction is to understand the neurocognitive basis of predictive processing in the human brain with special attention to auditory stimuli and rhythm. Using scalp and intracranial EEG, eye tracking and pupillometry, we aim to decipher the neural mechanisms involved.

Rhythm as an individual ability

The goal of the project Rhythm as an individual ability is to understand how rhythmic abilities vary between individuals. To do this we investigate the basic cognitive, environmental and genetic factors that shape rhythmicity. We are developing a test battery to cover a broad area of rhythm topics, such as, meter, beat, and microtiming.

Interaction and Pleasure

Interaction & Pleasure Cluster

We seek to understand why rhythms make us move and how rhythm facilitates entrainment and interaction. We also explore the affective and social outcomes of such processes. These are studied through the experience of absorption in musical performance and perception, music-related effort and skilled action, and the dynamics of music-evoked pleasure.

Highlights

  • PhD fellow Agata Zelechowska and colleagues published several papers on human, music-related micromotion. It was shown that people move more when listening to music on headphones than on speakers (Frontiers in Psychology). They also found that people's empathic concern can predict spontaneous movement responses to music (Music & Science).
  • The Oslo Standstill Database was released, containing motion capture data of more than 600 people standing still in silence or listening to music. The database marks RITMO's investment in developing Open Science practices and is a resource for other researchers interested in human, music-related micromotion.
  • When reconsidering her research activities in March 2020, PhD fellow Dana Swarbrick turned around and set up the Quarantine Concerts project in collaboration with several partners in the music industry. The aim is to explore the social and emotional effects of virtual concerts. The unique data that has been collected will help to understand more about “liveness” in online concerts and how performers can enhance engagement with online audiences.

Entrainment, Social Bonding and Pleasure

The project Entrainment, Social Bonding and Pleasure aims to advance our understanding of how entrainment, synchronised behaviour and other related processes lead to prosocial outcomes such as social bonding. Work within the project seeks to uncover the factors contributing to synchronous movement and social bonding in concerts and explores whether entrained listening can evoke affiliation towards fellow listeners, the musicians, or their social groups. The sub-project Quarantine Concerts project aims to explore the social and emotional effects of virtual concerts.

Effort and Attention in Musical Performance and Perception

Musical experiences often provide sensations of affect and pleasure and sensations of sound-producing and sound-accompanying body motion. The project Effort and Attention in Musical Performance and Perception aims to enhance our understanding of the attention and effort, both physical and mental, involved in producing and perceiving music-related sound and related multimodal phenomena.

Engagement and Absorption

Does time and timing matter for musical engagement and absorption? Music is well known for its ability to engage, yet the nature of this engagement remains elusive. The project Engagement and Absorption explores aspects of rhythm, temporality, and meaning in musical engagement, such as listening and performing. The project works from various research angles, including musical sense-making and ethics, absorption in musical performance, spatiality and immersion, and relational interaction.

Human Music-related Micromotion

The project MICRO - Human Bodily Micromotion in Music Perception and Interaction investigates how music influences micromotion, the smallest bodily motion that it is possible to produce and perceive. Data is collected using motion capture and physiological measurements. One of the annual highlights is the Norwegian Championship of Standstill, which is both important for data collection and serves as a dissemination activity.

Musical Chills

The Musical Chills project investigates the brain processes and musical triggers underlying the intense, pleasurable ‘chills’ responses sometimes evoked by music. It explores the role of other neuromodulators (particularly the endogenous opioid system), via pharmacological agents, in the pleasurable response to music. Moreover, the project seeks to investigate what specific musical features can trigger chills.

Interaction and Robotics

Interaction & Robotics Cluster

We investigate aspects of rhythm and motion through robotics and technology. This includes creating interactive music systems with a focus on multi-dimensional mapping from sensors to musical parameters. Moreover, we design and prototype adaptive robot systems and explore both human-robot and robot-robot interactions in collective settings.

Highlights

  • Postdoctoral fellow Charles Martin and Professor Jim Tørresen performed a data-driven analysis on more than 1600 musical touchscreen performances (Computer Music Journal). This was based on MicroJam, a social media-inspired smartphone app. Their work on AI-powered musical interaction was also reported in a Frontiers in AI article.

  • The Musical Gestures Toolbox is a software collection for creating video visualisations. It was primarily developed for music research, but is also used for sports, dance, healthcare, architecture, and interaction design. Alexander Refsum Jensenius and colleagues have worked on a Python implementation of the toolbox, which was released in 2020.

  • Professor Jim Tørresen received funding for the new project PIRC - Predictive and Intuitive Robot Companion from the Research Council of Norway. The project targets a psychology-inspired computing breakthrough by combining insight from cognitive psychology with computational intelligence to build models that forecast future events and respond dynamically.

Modelling and Robots

The Modelling and Robots project's core activity is the investigation of rhythm and motion through the design and construction of models and robots. This has applications to understanding natural processes as well as for the creation of technological systems.

Musical Human-Computer Interaction

The core activity of the project Musical Human-Computer Interaction is investigating aspects of rhythm and motion through the design and construction of interfaces for musical human-computer interaction. This includes the study and design of both acoustic instruments and completely digital systems. We are particularly interested in various types of electroacoustic instruments, in which we explore the complexity of human motion in musical experience and practice.

PIRC - Predictive and Intuitive Robot Companion

Humans anticipate future events more effectively than computers. The Predictive and Intuitive Robot Companion project combines sensing across multiple modalities with learned knowledge to predict outcomes and choose the best actions. The project will apply machine learning and robotics expertise and collaborate with researchers in cognitive psychology to apply recent human prediction models and action decision-making to perception-action loops of intelligent robot assistants. The work will allow such robots to adapt and act more seamlessly with their environment than the current technology.  

Self-playing Guitars

The self-playing guitars are mechatronic/robotic devices capable of producing sounds utilising a classic guitar body's acoustics. Each guitar can sense audio and proximity activity and has a battery-powered onboard processing platform. The devices can thus be deployed in a decentralized, multi-guitar setup.

Dr. Squiggles

Dr. Squiggles is an interactive musical robot that plays rhythms by tapping. The RITMO-designed robot listens for tapping produced by humans or other musical robots and attempts to play along and improvise its rhythms based on what it hears.

RITMO in Numbers

Image may contain: Blue, Text, Diagram, Circle, Organism.

Reported Research Results 2020

RITMO's researchers have published many scientific articles and book chapters in 2020, as well as two anthologies. They have also given talks and presentations at several conferences, although the numbers are lower due to the pandemic. The Centre's research results also include several artistic outputs and media contributions. You can see all of RITMO's results reported in CRIStin.

Gender balance

People

A total of 62 people were employed at RITMO in 2020, of which there were 26 women and 36 men. We work to achieve gender balance in our recruitments.

The World at RITMO - overview

The World at RITMO

RITMO is genuinely an international working environment. RITMO scholars come from all over the world: Argentina, Australia, Canada, China, Colombia, Denmark, Finland, France, Germany, Greece, Hungary, India, Iran, Italy, Japan, Jordan, Mexico, Montenegro, Norway, Poland, Romania, Serbia, South Korea, Spain, Sweden, Turkey, United Arab Emirates, United Kingdom and the United States.

RITMO in the world - overview

RITMO in the World

RITMO has research cooperation with partners from across the world: Argentina, Australia, Austria, Belgium, Brazil, Canada, Chile, Denmark, Finland, France, Germany, Iceland, India, Italy, Japan, Jordan, Netherlands, Poland, Sweden, Switzerland, United Kingdom and the United States.

RITMO Highlights

At a centre with around 60 employees, there are always numerous things happening. In 2020 we were particularly proud of three things.

Muddy Rhythms and Broken Beats

In February 2020, RITMO organized the seminar Muddy Rhythms and Broken Beats focusing on experimental grooves. Such experimental grooves may include seasick rhythms, fragmented or cut-up rhythms, staggering rhythms, pushed and dragged rhythms, syncopated rhythms, or various forms of rhythmic irregularities.

Highlights

Dr. Squiggles

Postdoctoral researcher Michael Krzyzaniak has developed Dr. Squiggles, an interactive musical robot that listens to rhythms and plays along by tapping on the table.

Dr-squiggles

Quarantine research

RITMO's researchers had to be innovative when faced with closed labs and constantly changing Corona regulations.

Corona

RITMO Stories

A project is always more than what you can read from a project page. Here some of RITMO's researchers present their own research.

Georgios Sioros is a postdoctoral researcher at RITMO studying musical rhythm.

Agata Zelechowska is a doctoral fellow at RITMO, studying human music-related micromotion.

Alejandro Blenkmann is a researcher at RITMO, studying attention and prediction.

Dana Swarbrick is a doctoral fellow at RITMO, studying entrainment and pleasure.

RITMO Seminar Series

The RITMO Seminar Series is a venue for presenting world-leading scholars from around the world. These events were also affected by the current situation, and there were fewer seminars organised this year.

Rebecca Fiebrink
"Machine learning as (Meta-) instrument" Rebecca Fiebrink, University of Arts London, United Kingdom
Tecumseh Fithc
"Hierarchy in Rhythmic Cognition" Tecumseh Fitch, University of Vienna, Austria
Shaun Gallagher
"Complicating the mesh: Integrating multiple factors in skilled performance" Shaun Gallagher, University of Memphis, United States
Mark Katz
Hip Hop Turntablism and the Limits of Rhythmic Complexity" Mark Katz, University of North Carolina, United States

Disputation

Agata Zelechowska

PhD fellow Agata Zelechowska successfully defended her dissertation Irresistible Movement: The Role of Musical Sound, Individual Differences and Listening Context in Movement Responses to Music in an online disputation.

RITMO Behind the Scenes

The year turned out to be very different than everyone had expected. Here are some snapshots of physical and online activities during 2020.

Behind-the-scenes

Data management

Data management was an important centre topic in 2020. Many of RITMO's researchers collect or generate data, and it is pivotal that these data are handled well.

RITMO's labs are run by two lab engineers, who also act as data managers. The data manager role is new at the University of Oslo, and RITMO has been involved in several university-wide initiatives to develop data handling policies and procedures. This includes:

  • Developing storage solutions that are both secure and accessible. In collaboration with UiO's Center for Information Technology, we have set up a virtual desktop infrastructure (VDI) that allows all researchers to access and process data securely. This turned out to be crucial once everyone had to work from home. Even without access to labs, RITMO researchers could access their data and utilize processing resources. Since many of the researchers work with heavy processes, and many work with video files, it was beneficial to have hardware-accelerated processing (GPUs) on the virtual machines.
  • Leading the cross-university Human Time Data network project, aiming at connecting research groups working on similar types of human time-series data. There are many differences and commonalities between data coming from motion capture systems, physiological sensors, eye trackers, and various types of brain imaging. The network has identified best practices and suggested future data management procedures.
  • Developing a data management handbook, with guidance for researchers on the steps necessary to collect, store, and archive data. There are many policy documents and generic tools available. However, in our experience, researchers need more concrete advice targeted at the applications at hand.
  • As part of the Open Science innovation project MusicLab, we have worked with the University Library on handling ethical and legal issues related to data collection in public spaces. This includes developing solutions for handling consent forms concerning privacy issues according to the GDPR. We have also started discussions with relevant actors when it comes to recording and sharing copyrighted musical material.

While we did a lot in 2020, there are still many unresolved matters. So data management will surely continue to be an important development topic in the years ahead.

Awards

Two RITMO papers received awards at international conferences:

RITMO People

All RITMO people

Scientific Advisory Board

The members of RITMO's Scientific Advisory Board are highly acclaimed researchers within their respective fields. The Board is vital to providing feedback on on-going and future research, and its members have been carefully chosen to support the different parts of RITMO.

Portrait of Georgina Born
Georgina Born Professor of Music and Anthropology at the Faculty of Music, University of Oxford, United Kingdom
Portrait of Eric Clarke
Eric F. Clarke Heather Professor of Music at the University of Oxford, United Kingdom
Portrait of Nicola Dibben
Nicola Dibben Professor in the Department of Music, University of Sheffield, United Kingdom
Portrait of Rebecca Fiebrink
Rebecca Fiebrink Senior Lecturer in Computing at Goldsmiths, University of London, United Kingdom
Portrait of Robert Knight
Robert T. Knight Professor of Psychology and Neuroscience at University of California Berkeley, United States
Portrait of Serge Lacasse
Serge Lacasse Professor of Musicology at Université Laval, Canada
Portrait of Marc Leman
Marc Leman Professor of Systematic Musicology and Director of IPEM at Ghent University, Belgium
Portrait of Kia Nobre
Kia Nobre Professor and Head of Department of Experimental Psychology, University of Oxford, United Kingdom
Portrait of Xavier Serra
Xavier Serra Professor in the Department of Information and Communication Technologies at Universitat Pompeu Fabra, Barcelona, Spain
Image may contain: Eyewear, Glasses, Face, Head, Forehead.
Jonathan Sterne Professor and James McGill Chair in Culture and Technology at the Department of Art History and Communication Studies, McGill University, Canada
Portrait of Andy Tyrell
Andy Tyrell Professor and Head of Department at the Department of Electronic Engineering, University of York, United Kingdom
Portrait of Peter Vuust
Peter Vuust Professor at the Center for Music in the Brain, Aarhus University, Denmark

Board

Portrait of Mathilde Skoie
Mathilde Skoie Pro-Dean, Faculty of Humanities, University of Oslo
Portrait of Peter Edwards
Peter Edwards Head of Department of Musicology, University of Oslo
Portrait of Bjørn Lau
Bjørn Lau Head of Department of Psychology, University of Oslo
Portrait of Ellen Munthe-Kaas
Ellen Munthe-Kaas Head of Department of Informatics, University of Oslo

Management

Anne Danielsen
Anne Danielsen Director, Cluster Leader
Alexander Refsum Jensenius
Alexander Refsum Jensenius Deputy Director
Anne Cathrine Wesnes Head of Administration

Postdoctoral Fellows and Researchers

Laura Bishop
Laura Bishop
Alejandro Blenkmann
Alejandro Omar Blenkmann
Anders Elovsson
Anders Elovsson
Ingrid Funderud
Ingrid Funderud
Victori Evaristo Gonzalez Sanchez
Victor Evaristo Gonzalez Sanchez
Mari Romarheim Haugen
Mari Romarheim Haugen
Alan Hui
Alan Hui
Simon Høffding
Simon Høffding
Ellis Jones
Ellis Nathaniel Jones
Michael Joseph Krzyzaniak
Michael Joseph Krzyzaniak
Olivier Lartillot
Olivier Lartillot
Sabine Leske
Sabine Leske
Georgios Sioros
Georgios Sioros
Chris Stover
Chris Stover
Finn Upham
Finn Upham

PhD Fellows

Merve Akca
Merve Akca
Fadi Al-Ghawanmeh
Fadi Al-Ghawanmeh
Olga Asko
Olga Asko
Guilherme Schmitdt Camara
Guilherme Schmidt Câmara
Cagri Erdem
Cagri Erdem
Maja Dyhre Foldal
Maja Dyhre Foldal
Image may contain: Nose, Cheek, Smile, Eyelash, Mouth.
Heidi Umbach Hansen
Julian Fuhrer
Julian Fuhrer
Henrik Herrebrøden
Henrik Herrebrøden
Ulf Holbrook
Ulf Holbrook
Seyed Mojtaba Karbasi
Mojtaba Karbasi
Tejaswinee Kelkar
Tejaswinee Kelkar
Dongho Kwak
Dongho Kwak
Qichao Lan
Qichao Lan
Kjell Andreas Oddekalv
Kjell Andreas Oddekalv
Martin Pleiss
Martin Pleiss
Bjørnar Sandvik
Bjørnar Sandvik
Sandra Solli
Sandra Solli
Connor Spiech
Connor Spiech
Image may contain: Face, Facial expression, Head, Green, Cartoon.
Alex Stevenson
Dana Swarbrick
Dana Swarbrick
Benedikte Wallace
Benedikte Wallace
Agata Zelechowska
Agata Zelechowska

Assistants

Synne Baumann
Synne Baumann
Ole Kristian Bekkevold
Elena Callegari
Alena Clim
Alena Clim
Andrei Faitas
Joachim Christian Dalgard
Frida Furmyr
Astrid Garmo
Frida Furmyr
Joachim Christian Dalgard
Jackson Goode
Rahman Habibur
Adrian Simen Holm
Emilie Holte
Tasmia Hussain
Eirik Jacobsen
Eirik Jacobsen
Erlend Krog
Sarjo Kuyateh
Bálint Laczkó
Sandra Ladegast
Martin Langerød
Martin T. Langerød
Mari Lesteberg
Trygve Liahagen
Eigil Lippert
Leigh Murray
Sara Kruge Nossen
Thea Sørli Paulsrud
Øyvind Skjerdal
Mattias Truell Thedens
Aleksander Tidemann
John Melvin Treider
Vegard Volehaugen

Technical/Administrative Staff

Rahul Agrawal
Rahul Agrawal
Lene Angelskår
Lene Angelskår
Kayla Burnim
Kayla Burnim
Marit Johanne Furunes
Marit Johanne Furunes
Nina Krogh
Nina Krogh
Ingrid Stange Ytterstad
Ingrid Stange Ytterstad

Guest researchers

  • Cancino-Chacón, Carlos Eduardo (Austrian Research Institute for Artificial Intelligence, Austria)
  • Milojevic, Andrija (Lut University, Finland)
  • Minami, Tetsuto (Toyohashi University of Technology, Japan)
  • Murayama, Rintaro (Toyohashi University of Technology, Japan)
  • Upham, Finn (McGill University, Canada)

A Centre of Excellence in Norway

A collaboration between

department of musicology

department of informatics

department of psychology