Annual Report 2019

2019 was the first "normal" year at RITMO. However, normal is perhaps not the right way to describe a centre with more than 50 full-time researchers and many students and guests coming and going.

In September 2019 we welcomed the second "wave" of newly recruited doctoral and postdoctoral fellows. We also started saying goodbye to the first colleagues that are leaving us. Keeping track of RITMO alumni around the world will be an essential task in the years to come.

As Centre Directors, we are incredibly proud of seeing how RITMO researchers start to interact across their traditional disciplinary boundaries. Based on feedback from the Scientific Advisory Board, we have restructured our internal organisation to allow for tighter collaboration and more effective daily operation. We now have four clusters as internal focal points, and they will be presented below.

2019 was a year of consolidation, the year we went from start-up chaos to full operation. Here we present some highlights from the year, in time, rhythm, and motion. 

Anne Danielsen and Alexander Refsum Jensenius

Anne Danielsen and Alexander Refsum Jensenius
Table of Contents
Image may contain: Black, White, Photograph, Black-and-white, Monochrome photography.

RITMO at a Glance

Rhythm is everywhere, from how we walk, talk, dance and play, to how we tell stories about the past and predict the future.

Our heartbeat, nervous system, and other bodily cycles all work through rhythm. Rhythm is a crucial aspect of human action and perception. Our human rhythm is also in complex interaction with the various cultural, biological and mechanical rhythms of the world. 

At RITMO, we research rhythmic phenomena and the complex relationships between rhythms of the human body and the brain. The central idea is to establish a link between features of rhythmic phenomena in the world and within the (embodied) mind. The aim is to understand our ability to perceive rhythm and how this affects our actions and experiences.

RITMO is a highly interdisciplinary centre, combining perspectives and methods from music and media studies, philosophy and aesthetics, cognitive neuroscience, and informatics. We employ several state-of-the-art technologies in our research, including motion capture, neuroimaging, pupillometry and robotics.

Structure and Time

Structure & Time Cluster

We investigate the structure and aesthetics of musical rhythm and their link to perceptual, cognitive and motor mechanisms. This includes the role of rhythm, temporal structure and timing in the experience of both long-lasting musical form and more fragmented, audiovisual musical formats. Overarching questions include how genre familiarity, technology and socio-cultural context interact with perceptual and cognitive mechanisms in the experience of rhythm and time. Methodologically, we rely on a broad range of approaches, including music analysis, performance studies, qualitative interviews, interpretation, behavioural experiments, computational analysis, and motion capture.

Highlights

Musical Microrhythm

The project TIME: Timing and Sound in Musical Microrhythm investigates interactions between temporal and sound-related features at the micro-level of rhythm in musical genres where rhythm is a core dimension. The aim is to gain new insights into the micro-level of auditory perception and the role of cultural background and/or training in music in this regard. This is based on using qualitative and quantitative methodologies from musicology, ethnomusicology, music psychology and motion research.

Musical Fragmentation

This project examines the pleasure of musical fragmentation and the related play with time, with a focus on sample-based and cut-and-paste music. In the MASHED project, the focus is on the underlying aesthetics and temporality of so-called Mashup music

Musical Time and Form

The new project Musical Time and Form investigates salient differences in musical interpretation between recorded performances of orchestral compositions in Western Art Music from the last two centuries. Compositions are performed differently depending on the conductor, orchestra, historical and cultural tradition, and other factors such as time and place of the performance. There is also a focus on the listening experiences of those performances, intending to improve the precision with which we discuss musical performance and the listening experience. 

MIRAGE

The aim of the new project MIRAGE - A Comprehensive AI-Based System for Advanced Music Analysis is to improve the computers' capability to listen to and understand music. A goal is to develop technologies that facilitate understanding and appreciation of music through generating rich and detailed descriptions. These will encompass a broad range of dimensions, such as rhythm, tonality, formal structure, melodic shape, sound colour, and emotions. Significant effort will be dedicated to the design of applications of value for musicology, music cognition and the general public.

Structure and Cognition

Structure & Cognition Cluster

The overarching goal is to expand our understanding of how rhythm and sense of time are constructed in the human mind. We study the influence of rhythm and time on perception, attention and effort, imagery and illusions, prediction and cognitive control processes across sensory modalities. Utilising a broad range of methods, including pupillometry, magnetic resonance imaging (MRI), scalp- and intracranial electroencephalography (EEG), we aim to reveal the effects of rhythmic and temporal features, as well as the neurocognitive mechanisms involved. For our experiments, we recruit healthy participants, but also special populations, such as musicians, synaesthetes, and neurological patients with focal brain damage or epilepsy. 

Highlights

  • In the paper Where Are the Months? Mental Images of Circular Time in a Large Online Sample (Frontiers in Psychology), Laeng and Hofseth (2019) report on an online survey with more than 70,000 participants. They were asked to report on how they think about time. The majority of respondents imagined the year as a “circle”, and 75% “saw” the months unfolding in a clockwise direction.
  • The insula is one of the least understood brain areas. In the paper Auditory deviance detection in the human insula: An intracranial EEG study (Cortex), Blenkmann et al. (2019) report on invasive EEG recordings from epilepsy patients that listened to sequences of expected standard and unexpected deviant sounds. This is the first direct neural evidence that a subregion of the insula is engaged during auditory deviance detection, that is, increased and later responses to unexpected sounds.

Cross-modal Rhythms

This research focuses on cross-modal rhythms or the study of multisensory correspondences in both the brain and behaviour. Current experiments investigate rhythm patterns across perceptual modalities processed in parallel. Experiments investigate ‘pulse’, ‘meter’ (grouping) and ‘rhythm’ patterns across perceptual modalities (e.g., auditory and tactile), as well as temporal judgments of simultaneity between predictable versus unpredictable events, in the visual and auditory modalities, separately or combined. This research is also conducted with synesthetic participants, who experience sounds when none have physically occurred. Generally, rhythmic events are hypothesised to be discrete windows of attentional processing of sensory information, or quantal objects of awareness. One hypothesis is that the brain fundamentally uses a dual code (categorical versus continuous) to represent all perceptual information.

Predictive Processing

A primary goal is to understand the neurocognitive basis of predictive processing in the human brain. The brain’s ability to predict future events enables dynamic interplay with our environment. It is mostly unknown how temporal regularities (periodic versus aperiodic) facilitate predictive processes. We challenge assumptions of predictive coding models and use new approaches to study how the brain makes sense of predictive cues. Using both scalp and intracranial EEG, we aim to decipher the neural mechanisms involved.

Interaction and Pleasure

Interaction & Pleasure Cluster

We study the processes and phenomena underlying rhythmic action and interaction, and the social and affective outcomes of these processes. Central aims of the cluster are to understand why rhythms make us move, and how rhythms facilitate entrainment and interaction. The cluster also investigates music-related effort and skilled action, the experience of absorption in music listening and performance, and the dynamics of music-evoked pleasure. The researchers within the cluster use a multitude of different methods: behavioural experiments, motion capture, eye-tracking, phenomenology, and psychophysiology.

Highlights

  • RITMO postdoctoral researcher Simon Høffding published the book A Phenomenology of Musical Absorption (Palgrave Macmillan), in which he has interviewed one of the world’s leading classical chamber ensembles, The Danish String Quartet (DSQ). He presents a detailed analysis of what it means to be absorbed in playing music. Høffding’s work with the DSQ has provided the foundation for a large-scale interdisciplinary collaboration between the DSQ and several RITMO researchers.

Entrainment, Social Bonding and Pleasure

This project aims to advance our understanding of how entrainment, synchronised behaviour and other related processes lead to prosocial outcomes such as social bonding. Work within the project seeks to uncover the factors contributing to synchronous movement and social bonding in concerts and explores whether entrained listening can evoke affiliation towards the musicians or their social groups.

Music-related Micromotion

This project investigates how music influences micromotion; the smallest bodily motion that it is possible to produce and perceive. Data collection is done using motion capture and physiological measurements. One of the annual highlights is the organisation of the Norwegian Championship of Standstill, which is both important for data collection but also serves as a dissemination activity.

Effort and Attention in Musical Experience

Musical experience is often mixed with not only sensations of affect and pleasure, but also of sound-producing and sound-accompanying body motion. This project seeks to enhance our understanding of the physical and mental effort and attentional processes involved in the production and perception of music-related sound and other music-related multimodal phenomena.

Engagement and Absorption

This project explores aspects of rhythm and temporality in musical engagement such as listening and performing. The project aims to unveil how different modes of engagement and absorption are at the core of the musical experience, and can lead to novel insights into human perception and action.

Musical Chills

This project investigates the brain processes and musical triggers underlying the intense, pleasurable ‘chills’ responses sometimes evoked by music. The project aims to investigate the extent to which ‘chilling’ music engages the brain's noradrenergic system, as indexed by pupillary measures.

Interaction and Robotics

Interaction & Robotics Cluster

How can we understand aspects of rhythm and motion through the design and construction of interactive systems and robots? We investigate aspects of rhythm and motion through robotics and technology. This includes the creation of interactive music systems, with a focus on multi-dimensional mapping from sensors to musical parameters. Moreover, we design and prototype adaptive robot systems and explore both human-robot and robot-robot interactions in collective settings. We employ a multitude of methods, including artificial intelligence, machine learning, simulations, modelling, rapid prototyping, as well as iterative and creative design processes. This analysis-by-synthesis approach will lead to a new understanding of rhythmic phenomena in general, and can also lead to applications in rehabilitation and art.

Highlights

  • Two platforms for mechatronic devices have been developed: the self-playing guitars and Dr. Squiggles. They allow for exploring the dynamics of decentralised interaction in a mixed group of machines and humans. This allows for studying algorithms for intelligent synchronisation and task division, processes of entrainment, and human-machine interaction. The machines have so far featured in several interactive installations and public communication events.

Modelling and Robots

This core activity investigates aspects of rhythm and motion through the design and construction of models and robots. This has applications to understanding natural processes as well as for the creation of technological systems.

Musical Human-Computer Interaction

This core activity focuses on the creation of interactive music systems with a focus on multi-dimensional mapping from sensors to musical parameters.

Self-playing Guitars

The self-playing guitars are mechatronic/robotic devices capable of producing sounds utilising the acoustics of a classic guitar body. Each guitar can sense audio and proximity activity, and has a battery-powered onboard processing platform. The devices can thus be deployed in a decentralized, multi-guitar setup.

Dr. Squiggles

Dr. Squiggles is an interactive musical robot that plays rhythms by tapping. The RITMO-designed robot listens for tapping produced by humans or other musical robots and attempts to play along and improvise its rhythms based on what it hears.

RITMO in Numbers

Image may contain: Blue, Text, Diagram, Circle, Organism.

Reported Research Results 2019

RITMO's researchers have published a number of scientific articles and book chapters in 2019, as well as one academic monograph. They have also given talks and presentations at a large number of conferences around the world. The Centre's research results also include several artistic performances and media contributions. You can see all of RITMO's results reported in CRIStin.

Gender balance

The People

A total of 59 people were employed at RITMO in 2019, of which there were 25 women and 34 men. We work to achieve gender balance in our recruitments. Diversity and gender balance was also addressed in the yearly meeting with the Scientific Advisory Board.

The World at RITMO - overview

The World at RITMO

RITMO is genuinely an international working environment. RITMO scholars come from all over the world: Argentina, Australia, Canada, China, Colombia, Denmark, Ecuador, Finland, France, Germany, Greece, India, Iran, Italy, Japan, Mexico, Montenegro, Norway, Poland, Spain, Sweden, Turkey, United Kingdom and the USA.

RITMO in the world - overview

RITMO in the World

RITMO has research cooperation with partners from across the world: Argentina, Australia, Belgium, Brazil, Canada, Denmark, Finland, France, Germany, Iceland, India, Japan, Netherlands, Poland, Slovenia, Spain, Sweden, Switzerland, United Kingdom and the USA.

RITMO Highlights

At a centre with close to 60 employees, there are always numerous things happening. In 2019 we were particularly proud of three things.

Reorganisation

It takes time to build up a centre spanning three departments and faculties. In 2019 we continued the development of the internal organisation.

Re-organisation

NordicSMC Winter School

In February 2019 RITMO hosted the Nordic Sound and Music Computing Winter School. Around 40 early career researchers from around the world gathered in Oslo to learn about recent advances in sound and motion analysis from experts from the NordicSMC partner institutions (Aalborg Copenhagen, KTH Stockholm, Helsinki, and Reykjavik).

Winter-school

RITPART International EEG Workshop at UC Berkeley

RITMO has a strategic partnership with University of California, Berkeley through the RITPART project. In October we organised an EEG workshop in the Knight Lab at UC Berkeley, during which researchers from both institutions presented new findings for each other. During the week-long trip, RITMO researchers also visited a number of other research groups and centres in the Bay Area.

Berkeley

RITMO Stories

A project is always more than what you can read from a project page. Hear some of RITMO's researchers present their own research.

Tejaswinee Kelkar is a researcher at RITMO working on music-related body motion.

Kjell Andreas Oddekalv is a doctoral fellow at RITMO, researching the voice in rap music and its musical properties.

Maja Dyhre Foldal is a doctoral fellow at RITMO, studying how the brain makes predictions about future events.

Michael Krzyzaniak is a postdoctoral fellow at RITMO working on computational creativity and musical robotics.

Disputation

Tejaswinee Kelkar

Tejaswinee Kelkar became the first to defend her thesis at RITMO: Computational Analysis of Melodic Contour and Body Movement.

RITMO Behind the Scenes

To the people working at RITMO, the Centre is so much more than just a working space. Even though the everyday life at the center evolves around topics of research and work, the treshold for socialising and laughing are low, which creates a warm energetic environment for the staff.

social

Code of Conduct

One important part of the work on the internal organisation, has been the development of the Code of Conduct. This was developed jointly by all RITMO employees during one of the retreats and can be summarized in five dimensions:

Friendliness Friendliness: we are happy to share our knowledge and expertise with each other
Diversity Diversity: we acknowledge and respect that we have colleagues with all sorts of backgrounds
Image may contain: Line, Turquoise. Time: we value and respect people's time
Image may contain: Line, Circle. Space: we appreciate the distinction between concentrated workspaces and lively social arenas
Image may contain: Line, Font. Efficiency: we strive for having enough (but not too many) meetings, and with clear agendas

RITMO's Career Development Programme

RITMO has been working together with the other Centres of Excellence at UiO in developing a Career Development Programme for its PhD and postdoctoral fellows.

The objective of the Programme is to help the early career researchers to articulate their career goals, and build experience and skills that are relevant and valuable both within and outside academia.

Mentoring plays a central role in the career development program. Each researcher is assigned a personal career mentor, who will assist in making an individualized career development plan. The career mentors are RITMO faculty members working within relevant disciplines.

The core of the Career Development Programme is formed by six thematic workshops on general topics (see below). These workshops will be organized in 3-year cycles, so that all researchers should be able to participate in all of them.

Time management and Career planning Time management and Career planning
Team and collaboration skills; ethics at the workplace Team and collaboration skills; ethics at the workplace
Project management Project management
Obtaining funding Obtaining funding
Dissemination and presentation skills Dissemination and presentation skills
Alternative career paths Alternative career paths; careers outside academia

Awards

Image may contain: Text, Poster, Design, Display advertising, Advertising.RITMO researcher Ximena Alarcón Diaz was selected as a Marie Sklodowska Curie Individual Fellow of the week, and showcased during the European Research and Innovation Days in Brussels.

RITMO People

All RITMO people

Scientific Advisory Board

The members of RITMO's Scientific Advisory Board are highly acclaimed researchers within their respective fields. The Board is vital to providing feedback on on-going and future research, and its members have been carefully chosen to support the different parts of RITMO.

Georgina Born Professor of Music and Anthropology at the Faculty of Music, University of Oxford.
Nicholas Cook Professor of Music at the Faculty of Music, University of Cambridge.
Nicola Dibben
Nicola Dibben Professor in the Department of Music, University of Sheffield.
Rebecca Fiebrink
Rebecca Fiebrink Senior Lecturer in Computing at Goldsmiths, University of London.
Robert Knight
Robert T. Knight Professor of Psychology and Neuroscience at Berkeley, University of California.
Serge Lacasse Professor in Musicology at Université Laval.
Marc Leman Professor in systematic musicology and director of IPEM at Ghent University.
Xavier Serra Professor of the Department of Information and Communication Technologies at Universitat Pompeu Fabra, Barcelona.
Jonathan Sterne Professor and James McGill Chair in Culture and Technology at the Department of Art History and Communication Studies, McGill University.
Andy Tyrell
Andy Tyrell Professor and Head of Department at the Department of Electronic Engineering, University of York.
Peter Vuust
Peter Vuust Professor at the Center for Music in the Brain, Aarhus University.

Board

Mathilde Skoie
Mathilde Skoie Pro-Dean, Faculty of Humanities, University of Oslo
Peter Edwards Head of Department Musicology, University of Oslo
Ellen Munthe-Kaas Head of Department Informatics, University of Oslo
Pål Kraft Head of Department of Psychology, University of Oslo

Management

Anne Danielsen Director, Cluster Leader
Alexander Refsum Jensenius Deputy Director
Anne Cathrine Wesnes Head of Administration

Postdoctoral Fellows

Laura Bishop
Laura Bishop
Alejandro Blenkmann
Alejandro Omar Blenkmann
Ximena Alarcón Diaz
Ingrid Funderud
Victor Evaristo Gonzalez Sanchez
Mari Romarheim Haugen
Alan Hui
Simon Høffding
Ellis Nathaniel Jones
Michael Joseph Krzyzaniak
Michael Joseph Krzyzaniak
Olivier Lartillot
Sabine Leske
Charles Martin
Georgios Sioros
Chris Stover
Chris Stover

PhD Fellows

Merve Akca
Merve Akca
Olga Asko
Olga Asko
Guilherme Schmidt Câmara
Cagri Erdem
Maja Dyhre Foldal
Julian Fuhrer
Henrik Herrebrøden
Henrik Herrebrøden
Ulf Holbrook
Ulf Holbrook
Seyed Mojtaba Karbasi
Mojtaba Karbasi
Tejaswinee Kelkar
Qichao Lan
Kjell Andreas Oddekalv
Martin Pleiss
Martin Pleiss
Bjørnar Sandvik
Sandra Solli
Sandra Solli
Connor Spiech
Connor Spiech
Image may contain: Face, Facial expression, Head, Green, Cartoon.
Alex Stevenson
Dana Swarbrick
Benedicte Wallace
Agata Zelechowska

Assistants

Aftab Bakhtiari
Ole Kristian Bekkevold
Ole Kristian Bekkevold
Alena Clim
Alena Clim
Andrei Faitas
Andrei Faitas
Frida Furmyr
Frida Furmyr
Jackson Goode
Rahman Habibur
Adrian Simen Holm
Emilie Holte
Tasmia Hussain
Bálint Laczkó
Sandra Ladegast
Martin T. Langerød
Eigil Lippert
Eirik Jacobsen
Eirik Jacobsen
Elise Måsvær
Sam Roman
Austeja Tamalunaite
Aleksander Tidemann
Ruby Topping
John Melvin Treider
Jonas Ulleland
Jonas Ulleland
Image may contain: Hair, Hairstyle, Chin, Forehead, Human.
Marcus Widmer

Technical/Administrative Staff

Rahul Agrawal
Rahul Agrawal
Ole Kristian Bekkevold
Victoria Tømmeraas Berg
Kayla Burnim
Kayla Burnim
Mthilde Espeseth
Mathilde Espeseth
Nina Krogh
Nina Krogh

Guest researchers

  • Bannister, Scott (Durham University)
  • Bruford, Frederick William Caisley (Queen Mary University of London)
  • Côté-Allard , Ulysse (Université Laval)
  • Dahl, Sofia (Aalborg University Copenhagen)
  • Furuya, Koki (Toyohashi University of Technology)
  • Gerry, Linda Joy (Aalborg University Copenhagen)
  • Kinzuka, Yuya (Toyohashi University of Technology)
  • Kontogeorgakopoulos, Alexandros (Cardiff Metropolitan University)
  • Maeda, Hiromu (Toyohashi University of Technology)
  • Minami, Tetsuto (Toyohashi University of Technology)
  • Popescu, Tudor (University of Vienna)
  • Poveda, Jorge (Choreomundus, UCA, UR, SZTE, NTNU)
  • Sanchez, Carlos Vara (Ca’ Foscari University, Venice)
  • Suzuki, Yuta (Toyohashi University of Technology)
  • Taniyama, Yuma (Toyohashi University of Technology)
  • Thompson, Marc (University of Jyväskylä)

A Centre of Excellence in Norway

A collaboration between

department of musicology

department of informatics

department of psychology