Norwegian version of this page

Finn Upham

Researcher - RITMO Centre for Interdisciplinary Studies in Rhythm, Time and Motion (IMV)

Email finn.upham@imv.uio.no

Username

Visiting address Forskningsv. 3A Harald Schjelderups hus 0373 Oslo

Postal address Postboks 1133 Blindern 0318 Oslo

Press photo Download business card

Academic interests

Finn Upham uses measurements of audience behaviour, physiological changes in listeners' bodies, and musical signal characteristics to investigate listeners' interactive experience of music unfolding in time. At RITMO, they are working on the empirical evaluation of temporal relationships between rhythms in musical signals and the oscillatory systems of the human body.

Background

PhD in Music Technology from New York University, dissertation on music listeners' respiratory phase alignment to music. Master in Music Technology as well as a BMus (Theory) and BSc (Mathematics) from McGill University.

Positions held

Postdoctoral fellow with the SSIMSA project at McGill University.

Tags: Music Cognition, MusicLab, Biosensors, Performance Analysis, Movement Analysis, respiration, Data science

Hugh Alexander von Arnim, Finn Upham & Alexander Refsum Jensenius (2026). Exploring Methods for Measuring Audience Body Motion in Concert Halls. In Ivica Ico Bukvic (Eds.), Proceedings of the Sound and Music Computing Conference 2026. SMC Network. Full text in Research Archive Show summary
Measurement of audience members’ body motion in ecological concert settings must navigate environmental constraints, access to audience members, privacy considerations, and behavioural norms of the performance event. This study compares five distinct technologies for capturing motion from “audience member” participants under simulated symphonic concert conditions: visible-range imaging (RGB), near-infrared imaging (IR), thermal imaging, thermal data, and torso-mounted accelerometry. Seven quantity of motion (QoM) time series were generated from these measurements, each cleaned and normalised to reveal similarities and differences in the information they provide regarding four concert-typical behaviours: Large Actions such as posture changes, Applause, Stillness, and Respiration. Results show comparable, though not identical, quantity of motion measurements across devices in this simulated concert setting. We discuss the advantages and disadvantages of the various signals, share recommendations on key measurement parameters, and discuss scaling these motion capture methods for larger audiences in real concert halls.
Finn Upham, Bilge Serdar Göksülük & Alexander Refsum Jensenius (2026). Laughter in the making: social bonding and coordination in dance practices. Frontiers in Psychology. 17. doi: 10.3389/fpsyg.2026.1754425. Full text in Research Archive Show summary
Laughter is a ubiquitous human behavior with distinct social functions and evolutionary value. And yet its role in artistic and cultural practices has rarely been investigated, except in explicitly humorous performances. This study highlights laughter's central role in the participatory and developmental phases of dance. Grounded in social function frameworks, we argue that laughter functions as a vital mechanism of layered coordination, supporting group dynamics beyond verbal interaction. Through a multimodal analysis of three distinct dance contexts—an amateur line dancing lesson, a Norwegian folk dancing session, and professional duo improvisations (co-located and remote)—we investigate how and why laughter emerges in real-time creative and learning processes. Using video, physiological (heart rate, core body motion), and phenomenological data, we show that laughter is not a break in action but an integral, embodied tool for coordination development. Our exploratory study demonstrates that laughter serves to acknowledge mistakes, negotiate decisions, express difficulty, and share joyful success in dance. Critically, the quality and shared nature of laughter instances were shaped by the available communication channels, the modalities involved, and the direction of interaction. We conclude that laughter is a powerful, efficient form of embodied communication that helps dancers navigate uncertainty, build interpersonal synchrony, and co-construct shared understanding. These results underscore the importance of studying the evolutionary and social value of artistic practices as lived, co-created experiences beyond their most professional and elite forms.
Hugh Alexander von Arnim, Anna-Maria Christodoulou, Kayla Burnim, Finn Upham, Tejaswinee Kelkar & Alexander Refsum Jensenius (2026). LightHearted—A Framework for Mapping ECG Signals to Light Parameters in Performing Arts. In Anthony L. Brooks (Eds.), Proceedings of the 14th EAI International Conference on ArtsIT, Interactivity and Game Creation, ArtsIT 2025. Springer. ISSN 9783032269997. p. 719–738. doi: 10.1007/978-3-032-26999-7_18. Full text in Research Archive Show summary
This paper presents LightHearted, an open-source Python framework for mapping the heart’s electrical activity from electrocardiography (ECG) signals acquired from performers to stage lighting in a concert setting. The aim is to provide dynamic lighting coupled to the physiological processes occurring during performance. Drawing upon critical approaches towards using biosignals in interactive systems and live concert research, we outline the framework’s design and implementation and present a case study with the Aarhus Symphony Orchestra. This paper highlights both technical and conceptual challenges of integrating biosensor-based lighting into a large-scale orchestral context. Results from a post-concert audience survey (N = 324) suggest that while responses to the lighting were mixed, it was generally not perceived as distracting and would be welcomed for use in future concerts.
Finn Upham & Fernando E. Rosas (2025). When and How the Audience Moves. Music & Science. 8, p. 1–24. doi: 10.1177/20592043251398280. Full text in Research Archive Show summary
Audience motion is a defining component of the face-to-face concert experience. Motion is also a common component of individual audience members’ reactions and actions to live performances. This article investigates patterns in chest-mounted accelerometer measurements from audience members at a chamber music concert experiment. The aim is to contextualize their behavior during music performance using a sequence of exploratory analyses. Combining insight from subjective reports of audience motion; musical, environmental, and biological frequencies; and similarity of motion timing with neighbors or more distant participants, both in-hall audience participants and live-stream viewers show two distinct music–motion patterns: either unobtrusive dynamic stilling or noticeable tempo-matched bouncing. Audience motion outside of music, during applause, speeches, or survey writing, also shows distinct average quantities of motion and degrees of shared information between seat neighbors, prompting a discussion on the constrained opportunities for audience action and social interaction at live classical concerts. Although exploratory, these extensive analyses provide a useful foundation for future work to further understand audience behaviors in concert music settings and beyond.
Finn Upham & Kayla Burnim (2025). ACCELEROMETER-BASED SYNCHRONISATION PROTOCOL FOR MUSIC-ALIGNED MEASUREMENTS OF DISTRIBUTED DATALOGGING WEARABLES. In Matthias Frank & Sofia Vallejo (Ed.), Proceedings of the Sound and Music Computing Conference 2025. SMC Network. ISSN 9783200106420. p. 415–420. doi: 10.5281/ZENODO.15838480. Full text in Research Archive Show summary
Many commercial wearable biosensors can be set to record (datalogging) cardiac activity, motion, respiration, and other signals in devices easily worn during group musical activities. However, despite offering research-relevant sample rates and exportable data formats, these commercial units are not designed for high accuracy synchronisation to external clocks. Anticipated uses like solo exercise and sleep monitoring do not require inter-device alignment below a few seconds, while musical activities feature discernible events within 200 ms and action alignment between participants under 10 ms. In the absence of a mechanism to digitally communicate timing triggers across datalogging devices, we have developed a synchronisation protocol to embed audio-aligned cues in active accelerometers to allow reassignment of timestamps after recording. This process substantially improves inter-unit signal alignment, correcting initialisation time offsets and clock drift to within accelerometer sample rate in recordings spanning 3 to 10 hours. Implemented in 15 recording sessions over 5 experiments, involving 9 to 95 units of various datalogging wearables, the resultant datasets support analysis of inter-participant coordination across signals and, with audio-based dynamic time warping, relatively high precision comparisons of sensor measurements to repeated musical events, all outside of laboratory conditions.
Olivier Lartillot, Dana Swarbrick, Finn Upham & Carlos Eduardo Cancino-Chacón (2025). Video Visualization of a String Quartet Performance of a Bach Fugue: Design and Subjective Evaluation. Music & Science. 8. doi: 10.1177/20592043251352299. Full text in Research Archive Show summary
Visualizing music—through music notation, analytical representations, or music videos—might potentially boost the appreciation of music in all its richness. The purpose of this study was to design and test a visualization strategy aimed at explicating to a large audience with diverse backgrounds—especially novices—the multifaceted beauty of the final Contrapunctus in J.S. Bach's The Art of Fugue, performed by the Danish String Quartet. At the surface level of the musical structure, the rich fluctuation of pitch shaped by each musician was depicted in the form of undulating pitch curves. At a deeper structural level, the repetition of pitch curves, distinctive of fugues, was highlighted through vertical alignment—inspired by a technique called paradigmatic analysis, originating from anthropology and music semiology. The visualization was initially prototyped in the form of a real-time technology as part of the MusicLab Copenhagen research concert. The concert audience focused on the performance itself, and did not pay much attention to, nor appreciate, the visualization. To evaluate more thoroughly the potential of the visualization, participants with varied musical expertise and taste were invited to listen to a recorded performance of the piece and watch the visualization on their own computer. A large majority reported that they felt they understood the visualization, around half of them felt that it enhanced their musical understanding, and a small group felt that it helped them to better appreciate the music.
Remy Haswell-Martin, Finn Upham, Simon Høffding & Nanette Nielsen (2025). Embodied, Exploratory Listening in the Concert Hall. Behavioral Sciences. 15(5). doi: 10.3390/bs15050710. Full text in Research Archive Show summary
Live music can afford novel, transformative aesthetic interactions for individual audience members. Nevertheless, concert research tends to focus on shared experience. In this paper we offer an account of exploratory listening that foregrounds embodied–enactive engagement and affective resonance through close analysis of the music, physiological measurements, and reflections from interviews. Our analysis centres on data collected from two musician audience members about one specific piece out of a larger interdisciplinary project involving concerts given by the Stavanger Symphony Orchestra and The Norwegian Radio Orchestra in March and June of 2024. Through the combination of in-depth phenomenological interviews with musically skilled audience members and measurements of breathing and body motion, we explore aesthetic enactment beyond common patterns of ‘synchronised’ response, focusing on audience members’ experiences of Harald Sæverud’s ‘Kjempeviseslåtten’ (The Ballad of Revolt) (1943). We find forms of absorbed, both imaginative and embodied involvement, of listeners enacting meaningful contact with, and pathways through, the music that in some ways corroborate crowd patterns but also reveal exploratory expertise and idiosyncratic affective orientations.
Simon Høffding, Rebecca Josefine Five Bergstrøm, Laura Bishop, Pedro Pablo Lucas Bravo, Kayla Burnim & Carlos Eduardo Cancino-Chacón [Show all 28 contributors for this article] (2025). Introducing the MusicLab Copenhagen Dataset. Music & Science. 8. doi: 10.1177/20592043241303288. Full text in Research Archive Show summary
MusicLab Copenhagen was a unique research concert featuring the world-renowned Danish String Quartet in a naturalistic setting. The audience was split between one group physically located in the hall, another group listening to a radio broadcast, and a third group watching a live stream. Qualitative and quantitative data were captured from both musicians and audiences, resulting in a comprehensive dataset that can be used to address many research questions. This document introduces the dataset, explains its structure, and reflects on the related data collection, storing, publishing, and archiving processes.
Finn Upham, Simon Høffding & Fernando E. Rosas (2024). The Stilling Response: From Musical Silence to Audience Stillness. Music & Science. 7, p. 1–14. doi: 10.1177/20592043241233422. Full text in Research Archive
Finn Upham, Jin Ha Lee & So Yeon Park (2023). Audience reconstructed: social media interaction by BTS fans during live stream concerts. Frontiers in Psychology. 15, p. 1–17. doi: 10.3389/fpsyg.2024.1214930. Full text in Research Archive Show summary
COVID-19-motivated social distancing made online concerts common practice in 2020 and 2021, with millions logging into streaming sites to see their favorite artists perform in realtime. For some fans, watching alone at home may have been enough, but concert-concurrent surges of social media activity suggest many virtual performance attendees are doing more. To understand why fans would turn their attention from these precious performance streams to social media, we explored Twitter engagement during four live streamed concerts performed by the Kpop group BTS in 2021. In public Tweets sampled by either concert hashtag or a predefined stream of users and keywords, we evaluated patterns in posting rates in relation to concert program events and investigated the content patterns in 1,200 Tweets sampled from four ranges of popularity (number of Retweets during the concert). Across concerts, short "Shout" Tweets surged at the start of songs, while the rate of retweets often fell during musical performances and shot up when BTS was off stage. Content analysis on the subsample found the materials most widely shared were informational or featured concert visuals, mimicking how fans use their phones at in-person concerts. Most original posts received few Retweets and were more personal and expressive of admiration for the performers. Comparison between the samples (concert hashtag vs. stream) also suggests users were strategic in using or omitting official concert hashtags with the strongest differences in the most widely disseminated content. Postings on Twitter during these performances seemed principally directed to fellow fans and audience members, by individuals choosing to share their own excitement and check in with others. By leveraging their existing social media networks, these concert attendees constructed a collective and interactive concert space, connecting with friends and strangers in the crowd and helping each other capture a richer experience than any broadcasting platform currently supports.
Maham Riaz, Finn Upham, Kayla Burnim, Laura Bishop & Alexander Refsum Jensenius (2023). Comparing inertial motion sensors for capturing human micromotion, Proceedings of the Sound and Music Computing Conference 2023. SMC Network . ISSN 9789152773727. doi: 10.5281/zenodo.8316051. Full text in Research Archive Show summary
The paper presents a study of the noise level of accelerometer data from a mobile phone compared to three commercially available IMU-based devices (AX3, Equivital, and Movesense) and a marker-based infrared motion capture system (Qualisys). The sensors are compared in static positions and for measuring human micromotion, with larger motion sequences as reference. The measurements show that all but one of the IMU-based devices capture motion with an accuracy and precision that is far below human micromotion. However, their data and representations differ, so care should be taken when comparing data between devices.
Dana Swarbrick, Finn Upham, Cagri Erdem, Alexander Refsum Jensenius & Jonna Katariina Vuoskoski (2022). Measuring Virtual Audiences with The MusicLab App: Proof of Concept. In Romain Michon, Laurent Pottier & Yann Orlarey (Ed.), Proceedings of the 19th Sound and Music Computing Conference. SMC Network. ISSN 9782958412609. p. 532–539. doi: 10.5281/zenodo.6798290. Full text in Research Archive Show summary
We present a proof of concept by using the mobile application MusicLab to measure motion during a livestreamed concert and examining its relation to musical features. With the MusicLab App, participants’ own smartphones’ inertial measurement unit (IMU) sensors can be leveraged to record their motion and their subjective experiences collected through survey responses. The MusicLab Lockdown Rave was an Algorave (live-coded dance music) livestreamed concert featuring prolific performers Renick Bell and Khoparzi. They livestreamed for an international audience who wore their smartphones with the MusicLab App while they listened/danced to the performances. From their acceleration, we computed quantity of motion and compared it to musical features that have previously been associated with music-related motion, namely pulse clarity and low and high spectral flux. By encountering challenges and implementing improvements, the MusicLab App has become a useful tool for researching music-related motion.

View all works in NVA

Finn Upham (2026). Familiarity, performance function, and physiological synchrony in concerts. Full text in Research Archive Show summary
When and how do bodies work together in concert? Some music requires unison action from performers while others involve cooperative but independent contributions. Some genres and events invite active audience participation, others expect discreet attention. With a new analysis strategy called subset coherence, this talk compares the prominence of respiratory, cardiac, and body motion synchrony across genres of music and categories of participants. Besides structural differences in genres and pieces, the evaluated ratios of subset coherence distinguish audience familiarity and musical roles across performing ensembles. The opportunities to physically "come together" in music are contingent on what that music is and how we can take part.
Alexander Refsum Jensenius, Joel Clinton Glover, Finn Upham & Annabel Balean Guaita (2026). MusicLab Listening Body Bergen Panel Discussion. doi: https:/www.uio.no/ritmo/english/projects/musiclab/2026/listening-body-bergen/index.html. Full text in Research Archive Show summary
Panel discussion at the MusicLab Listening Body research concert in Bergen, where an audience with and without Parkinson's disease heard classical piano works framed by meditation and attentive listening while wearing sensor vests that recorded micromotion, breathing and heart rate, and the room was filmed with thermal video. The conversation focused on what attentive listening does to the body and brain in a concert, what the measurements can and cannot show, and what the recordings may mean for music used therapeutically by people with Parkinson's disease.
Alexander Refsum Jensenius, Joel Clinton Glover, Finn Upham & Annabel Balean Guaita (2026). MusicLab Listening Body Oslo Panel Discussion. doi: https:/www.uio.no/ritmo/english/projects/musiclab/2026/listening-body-oslo/index.html. Full text in Research Archive Show summary
Panel discussion at the MusicLab Listening Body research concert in Oslo, where an audience with and without Parkinson's disease heard classical piano works framed by meditation and attentive listening while wearing sensor vests that recorded micromotion, breathing and heart rate, and the room was filmed with thermal video. The conversation focused on what attentive listening does to the body and brain in a concert, what the measurements can and cannot show, and what the recordings may mean for music used therapeutically by people with Parkinson's disease.
Alexander Refsum Jensenius, Bálint Laczkó, Poutaraud, Joachim, Widmer, Marcus, Furmyr, Frida & Jinyue Guo [Show all 9 contributors for this article] (2026). musicalgestures: A Python toolbox for visualising and analysing music-related video recordings. doi: 10.5281/zenodo.22287338. Full text in Research Archive Show summary
Software. This is the Python version of the Musical Gestures Toolbox, containing a complete solution for loading different types of video recordings (including 360-degree videos). It was originally built to analyse music-related videos, but it works well for many other applications. It includes a large collection of tools for visualising human body motion, including average images, motion images, motion history images, motiongrams, and videograms. It also includes spatial–temporal tools for creating motion history videos. Finally, it wraps different state-of-the-art pose estimation methods (CV2 and MediaPipe) and derives various quantity-of-motion measures.
Alexander Refsum Jensenius, Finn Upham, Agata Zelechowska, Victor Evaristo Gonzalez Sanchez, Dana Swarbrick & Maham Riaz (2026). micromotion: A Python toolbox for analysing human micromotion in motion time series. doi: 10.5281/zenodo.22080495. Full text in Research Archive Show summary
Software. One implementation of quantity of motion — the average speed of a body part, band-limited to 0.2-5 Hz, in millimetres per second — for optical marker data, body-worn accelerometers and force-plate centre of pressure, together with the readers, resampling rules, scaling measures and alignment methods that surround it.
Alexander Refsum Jensenius, Finn Upham & Kjell Andreas Oddekalv (2026). MusicLab Sinsenfist Panel Discussion. doi: https:/www.uio.no/ritmo/english/projects/musiclab/2026/sinsenfist/index.html. Full text in Research Archive Show summary
Panel discussion during the MusicLab research concert with the hip-hop group Sinsenfist, in which musicians and audience members were recorded with audio, video and motion sensors. Alexander Refsum Jensenius moderated the conversation with researchers Finn Upham and Kjell Andreas Oddekalv about what happens in the bodies of those who play and those who listen during a live performance, and what the concert measurements can tell us.
Annabel Balean Guaita, Finn Upham & Alexander Refsum Jensenius (2026). The Listening Body – for mennesker med og uten Parkinson. doi: https:/www.uio.no/ritmo/english/projects/musiclab/2026/listening-body-oslo/. Full text in Research Archive Show summary
We invite people with and without Parkinson’s disease to attend a unique participatory concert experiment to investigate how meditation and attentive listeni...
Hugh Alexander von Arnim, Anna-Maria Christodoulou, Kayla Burnim, Finn Upham, Tejaswinee Kelkar & Alexander Refsum Jensenius (2025). LightHearted—A Framework for Mapping ECG Signals to Light Parameters in Performing Arts. Full text in Research Archive
Finn Upham & Kayla Burnim (2025). ACCELEROMETER-BASED SYNCHRONISATION PROTOCOL FOR MUSIC-ALIGNED MEASUREMENTS OF DISTRIBUTED DATALOGGING WEARABLES. doi: 10.5281/zenodo.15838480. Full text in Research Archive Show summary
Many commercial wearable biosensors can be set to record (datalogging) cardiac activity, motion, respiration, and other signals in devices easily worn during group musical activities. However, despite offering research-relevant sample rates and exportable data formats, these commercial units are not designed for high accuracy synchronisation to external clocks. Anticipated uses like solo exercise and sleep monitoring do not require inter-device alignment below a few seconds, while musical activities feature discernible events within 200 ms and action alignment between participants under 10 ms. In the absence of a mechanism to digitally communicate timing triggers across datalogging devices, we have developed a synchronisation protocol to embed audio-aligned cues in active accelerometers to allow reassignment of timestamps after recording. This process substantially improves inter-unit signal alignment, correcting initialisation time offsets and clock drift to within accelerometer sample rate in recordings spanning 3 to 10 hours. Implemented in 15 recording sessions over 5 experiments, involving 9 to 95 units of various datalogging wearables, the resultant datasets support analysis of inter-participant coordination across signals and, with audio-based dynamic time warping, relatively high precision comparisons of sensor measurements to repeated musical events, all outside of laboratory conditions.
Finn Upham (2025). Workshop: Coherence and Cause - Exploratory analysis strategies for physiological measurements from live concert participants. Full text in Research Archive Show summary
Concert research involves studying humans engaged in very complex activities: highly predictable but constantly changing stimuli experienced through multiple sensory modalities and socio-cultural constraints. Physiological measurements of audience members and musicians during such activities reflect that richness with substantial differences between participants, with substantial ambiguity around the contributions of the shared musical activity on their respective heart rates, respiratory sequences, and movement timing. Some analysis strategies treat this richness as noise around more obvious correlates identifiable in laboratory conditions (ex: loudness, listener arousal, and heart rate). Others invoke complex data-agnostic assessments of shared information to demonstrate the existence of synchronicity with little opportunity to investigate how intervals of coherence emerge, whether across a crowd or between performers and audience. This workshop presents a data-centric approach designed to facilitate the discovery and evaluation of causal connections between musical information and physiological events during concerts. With examples from audience motion and performer respiration, Finn will demonstrate this process of exploration, identification, and probabilistic evaluation and then share some data from past studies to be explored by attendees and discussed as a group. Key issues discussed will include: Assessing the limits of a dataset, whether it can provide enough information to test for the phenomenon suggested. Controlling the analysis sequence through identification and evaluation. Projecting the consequences of the identified phenomenon and performing orthogonal tests on the same or related datasets. Conditions for dropping the thread when the evidence doesn't hold up, and making the most of analysis mistakes. Hypothesis-lead controlled experimental work may be the gold standard for building scientific knowledge, but there is more to discover than that which we already know to look for. Exploratory approaches that are sensitive to both the mechanisms of music production as well as the characteristics of measured physiological signals can expose patterns with more explanatory power than large-scale evaluations of coherence. Potential exploration datasets include: - respiratory-cardiac interactions in brass players - laughter and concert audience motion - Applause textures, synchronous vs independence
Finn Upham (2025). Respiratory timing in orchestral performance. Full text in Research Archive Show summary
Breathing is a necessary physiological function during music performance, and the timing of respiration by wind players and singers are reliable constrained by their performance actions. The relevance of respiration timing to other instrumentalists and across large ensembles has not be investigated in detail. One study on solo pianists found little reliability, while many studies of string quartets and other chamber groups often have not bother to monitor performers’ breathing. The Bodies in Concert experiments measured orchestral musicians breathing, cardiac activity, and motion during repeated performances in three concert series. Looking closely at the timing of respiratory sequences captured for a set of annotated pieces (6 pieces, 5-8 performances, 30 - 55 musicians per ensemble), we consider specifically the coincidence of inspiration and expiration onsets between performers in each section of the orchestra and within performers between performances with audio-based dynamic time warping. These orchestral musicians’ breathing show consistent timing beyond the most expected cases. Performers varied in their intra-performance reliability, averaging 20-50% consistency while playing (depending on the section) and 10-15% while tacet. Between players, expiration onset coincidence rates were also significantly higher that chance or alternative conditions for all string sections as well as brass and winds. Patterns within string sections over repeated performances highlight passages for which these players reliably recruit their respiratory system. However, these moments do not appear sufficient to fully explain the exceptional degree of respiratory coordination measured in each performance. The respiratory patterns captured show substantial disruptions allowing alignment with musical action rather than coincidental phasing between steady oscillatory systems, even in non winds. For many of these musicians, breathing looks to be an important if not always essential component of precise musical performance.
Finn Upham (2024). Heart Rate consistency and Heart Rate Variability constraints in orchestral musicians across performances. Full text in Research Archive
Finn Upham (2023). Making the best of it: Constraints and opportunities in studying complex human musicality. Full text in Research Archive Show summary
A talk for the Oslo Centre for Biostatistics and Epidemiology on exploratory data analysis with examples of strategies and challenges that arise in studying measurements from real world activities like orchestral concert performances. Presents preliminary findings on audience motion patterns and orchestra respiration, and presents the advantages of studying expert bodies in action.
Finn Upham (2023). Using Metrically-entrained Tapping to Align Mobile phone sensor measurements from In-person and Livestream Concert Attendees. Full text in Research Archive Show summary
Music is often made and enjoyed in large groups, but simultaneously capturing measurements from dozens or hundreds of people is technically difficult. When measurements are not constrained to wired or continuous connected wireless systems, we can record much bigger groups, potentially taking advantage of the wearable sensors in our phones, watches, and more dedicated devices. However, aligning measurements captured by independent devices is not always possible, particularly to a precision relevant for music research. Phone clocks differ and update sporadically, wearable device clocks drift, and for online broadcast performances, exposure times can vary by tens of seconds across the remote audience. Many measurement devices that are not open to digital synchronisation triggers still include accelerometers; with a suitable protocol, participant movement can be used to imbed synchronisation cues in accelerometry measurements for alignment regardless of clock times. In this paper, we present a tapping synchronisation protocol that has been used to align measurements from phones worn by audience members and a variety sensors worn by a symphony orchestra. Alignment with the embedded cues demonstrate the necessity of such a protocol, correcting offsets of more than 700 ms for devices supposedly initialised with the same computer clock, and over 10 s for online audience participants. Audience tapping performance improved cell phone measurement alignment to a median of 100 ms offset, and professional musicians tappings improved alignment precision to around 40 ms. While the temporal precision achieved with entrained tapping is not quite good enough for some types of analyses, this improvement over uncorrected measurements opens a new range of group coordination measurement and analysis options.
Finn Upham (2023). Breathing Together in Music, a RESPY Workshop. Full text in Research Archive Show summary
Respiration is a subtle but inescapable element of real time musical experiences, sometimes casually accompanying whatever we are hearing, other times directly involved in the actions of sound generation. This workshop explores respiratory coordination in music listeners and ensemble musicians with respy, a new python library for evaluating respiration information from single belt chest stretch recordings. Following an introduction to the human respiratory system and breathing in music, the workshop demonstrates how the respy algorithms evaluate phase and breath type, and presents statistical tools for assessing shared information in these features of people listening to or making music together. Rather than only use aggregate statistics such as respiration rate, respy aims to elevate the details of the respiratory sequence to facilitate our exploration of how breathing is involved in musical experiences, second-by-second. Measurable coordination of the respiratory system to musical activities challenges our expectations for interacting oscillatory systems. This session will conclude with a discussion on the different categories of relationships possible between people breathing together in music.
Finn Upham & Kjell Andreas Oddekalv (2023). Fingers and Tongues: Appreciating Rap Flows through Proprioceptive Interaction in Rhythm Hive. Full text in Research Archive Show summary
Rhythm games have been studied for their potential to develop interest in music making (Cassidy and Paisley, 2013) and transferable musicianship skills (Richardson and Kim, 2011), but how might they influence players appreciation for specific musical works? Proprioceptive interaction, a concept by game designer Matt Bloch (Miller, 2017), refers to changes in a game player's perception of music as they practice specific movements to it. By drawing attention to coincidental sounds, players can develop their hearing and appreciation for nuances of production and performance. Many fans of rap enjoy performances in languages they do not speak themselves. Without specific language skills, expertise in rap performance, and/or time to learn lyrics phonetically, their experience of a rap flow is hampered by an inability to imitate and imagine the generative action of performance. Rhythm Hive is a mobile rhythm game based on the music of BTS, Enhyphen, and TXT, Kpop groups with substantial followings outside of Korea. Game play presents players with finger choreographies to these groups’ hit songs, tapping sequences to the vocal performances across four to seven positions in a line. For these groups’ many non-rapping and non- Korean-speaking fans, playing Rhythm Hive may offer deeper understanding of performances by rappers like RM, Suga, and J-Hope. Through expert analysis of rap performance, transcriptions of game play, and reflections on the experience of playing Rhythm Hive, we consider shared structure between the prescribed finger choreographies and the rap flows they accompany. We studied rap verses from four BTS songs along side their Easy and Hard level tapping sequences (vocal versions only) to identify parallels in rhythm, segmentation, repetition, and accents. Easy mode choreographies tend to mark their relationship to rap vocals by hitting the start of lines and then articulating structure with repeated contours tapped on quarter and eighth notes. Hard mode choreographies tend to hit every rapped syllable and incorporate more gestural flourishes to mark pitch changes, ending and internal rhymes, and interesting breaks from a steady 16th note flow. Both Easy and Hard tappings sequences consistently follow the rap track when it deviates from a quantized beat. The finger choreographies of Rhythm Hive illuminate rap performances by directing and rewarding players’ attention to details of flows that may otherwise be missed. Game feedback pushes players to replicate delivery microtiming, while spatial patterns underline linguistic and rhythmic structure. Hard mode tapping sequences articulate distinguishing characteristics of specific rap styles, given players tangible sensitivity to degrees of technicality and nuances of genre. While fans may be motivated to play rhythm games like Rhythm Hive out of a preexisting love of the music and bands, tapping along offers them a chance to attend to, appreciate, and even rehearse key aspects of these rappers’ expert performance choices, regardless of how well they might follow by ear.
Finn Upham & Bjørn Morten Christophersen (2023). Bodies in Concert: RITMO project with the Stavanger symfoniorkester. Full text in Research Archive Show summary
Discussion between Composer and Music Researcher on the physiological trajectories of orchestra members performing a new and challenging work.
Finn Upham (2023). Insight into human respiration through the study of orchestras and audiences. Full text in Research Archive Show summary
Short presentation to a general science audience about what is uncovered in the coordinated actions of an orchestra by wearable psychophysiology sensors.
Laura Bishop & Finn Upham (2023). Bodies in Concert. doi: https:/www.uio.no/ritmo/english/projects/Bodies-in-Concert/. Full text in Research Archive Show summary
Increasingly, research on music performance is moving out of controlled laboratory settings and into concert halls, where there are opportunities to explore how performance unfolds in high-arousal conditions and how performers and audiences interact. In this session, we will present findings from a series of live research concerts that we carried out with the Stavanger Symphony Orchestra. The orchestra performed the same program of classical repertoire for four audiences of schoolchildren and an audience of families. Orchestra members wore sensors that collected cardiac activity, respiration, and body motion data, and the conductor additionally wore a full-body motion capture suit and eye-tracking glasses. Audience members in some of the concerts were invited to wear reflective wristbands, and wristband motion was captured using infrared video recording. We will begin the session with a discussion of the scientific and methodological challenges that arose during the project, in particular relating to the large scale of data capture (>50 musicians and hundreds of audience members), the visible nature of research that is carried out on a concert stage, and the development of procedures for aligning data from different recording modalities. Next, we will present findings from two lines of analysis that investigate different aspects of behavioural and physiological coordination within the orchestra. One analysis investigates the effects of audience noise and musical roles on coherence in (i) cardiac rate and variability and (ii) respiratory phase and rate. The second analysis investigates the effects of musical demands on synchronization of body sway, bowing, and respiration in string sections. We will conclude the session with an open discussion of how live concert research might be optimized.
Remy Richard Martin, Ian Cross, Finn Upham, Laura Bishop, Solveig Sørbø & Frederik Øland (2023). What can one learn from more naturalistic concert research? Full text in Research Archive
Maham Riaz, Finn Upham, Kayla Burnim, Laura Bishop & Alexander Refsum Jensenius (2023). Comparing inertial motion sensors for capturing human micromotion. Full text in Research Archive Show summary
The paper presents a study of the noise level of accelerometer data from a mobile phone compared to three commercially available IMU-based devices (AX3, Equivital, and Movesense) and a marker-based infrared motion capture system (Qualisys). The sensors are compared in static positions and for measuring human micromotion, with larger motion sequences as reference. The measurements show that all but one of the IMU-based devices capture motion with an accuracy and precision that is far below human micromotion. However, their data and representations differ, so care should be taken when comparing data between devices.
Olivier Lartillot, Dana Swarbrick, Finn Upham & Carlos Eduardo Cancino-Chacón (2023). Video visualization of a string quartet performance of a Bach Fugue: Design and subjective evaluation. Full text in Research Archive
Finn Upham, Ahmet Emin Memis, Niels Chr. Hansen, Fernando E. Rosas, Maria-Alena Clim & Alexander Refsum Jensenius (2022). Participatory applause: Interactions of audience members clapping at the end of a classical music concert. Full text in Research Archive Show summary
Participatory applause: Interactions of audience members clapping at the end of a classical music concert According to musicological studies of audience culture, applause is the most overt form of participation allowed to the collections of individuals attending classical music concerts (Brandl-Risi, 2011; Small, 1998; Tröndle, 2020). The final round of applause can exhibit many interesting dynamics related to their collective enthusiasm for the performance (Lupyan & Rifkin, 2003), the local applause culture, and what is on stage during the clapping. Quantitative empirical study of group clapping behaviours has principally depended on participants clapping on request in laboratory settings or A/V recordings from concerts (Neda, 2000) and presentations (Mann et al., 2013). To study the coordination involved in this collective behaviour, we need accurate measurements of individuals clapping voluntarily in a real concert setting. To describe how the appreciative audience members adjust their clapping to each other and the action on stage during the final round of applause, demonstrating their participation at a concert’s end. * After the Danish String Quartet (DSQ) performed their last piece at the Music Lab Copenhagen Concert, the audience clapped continuously for nearly two minutes. During that time, the musicians stood and bowed, had scientific instruments removed from their bodies, left the stage, returned to bow again, and finally left the stage for good. The clapping action of individual participants in this concert experiment was captured by a mobile phone on their chests, and these recordings show how individuals’ clapping contributed to the collective effect shared with the musicians. Through the final applause interval, 70 devices captured clear clap sequences, representing over half of the audience at this chamber performance. In some ways, their applause followed expected patterns for a concert audience. They began to applaud over a very short time interval (Mann et al., 2013), more than half starting within less than a second of each other. After 20 s of independent clapping at rates from under 120 BMP to over 200 BPM, the participants shifted to clapping together on a shared beat, a practice that is common for Danish audiences. This group maintained synchrony for over a minute while steadily accelerating from around 158 BMP to 176 BMP, an expected consequence of mutual adaptation during group clapping (Thomson et al., 2018). The coordinated action was strongest while the musicians were on the stage but a subset of independent clapping broke out while the audience waited for the performers to return for their final round of bows. Participants’ claps were evaluated from two perspectives: the alignment of claps, reflecting the dominant shift from independence to coordination, and the distribution of participants’ clapping rates over time. Despite some measurement challenges, the shift from independent to coordinate clapping emerges strongly from participants’ movements, with the median rate of clapping slowing until a dominant beat takes hold. Individuals’ clap sequences confirm that the independent clapping at the start of the applause is a result of individual participants clapping isochronously at their own rate, separate in rate and phase from their neighbours in the hall. When the audience claps together, they are voluntarily adjusting to the dominant rate and phase of the people in the hall, with little change in the quality of their isochronous clapping action. Drift in the synchronised clapping rate reflects mutual attentiveness while variation in the number of participants contributing to the coordinated claps suggests differences in applause strategy. Many participants opted to coordinate with their peers while some seemed to prioritise reacting to the musicians.
Finn Upham (2022). Audience reconstructed: Fan interactions on twitter during livestreamed BTS concerts. Full text in Research Archive
Finn Upham (2022). Respiratory phase alignment in music audiences. Full text in Research Archive
Finn Upham, So Yeon Park & Jin Ha Lee (2022). Concert tweeting: Textures of fan activity on twitter during Sowoozoo. Full text in Research Archive Show summary
Live-streamed BTS concerts have become a familiar fan activity for ARMY. While the broadcasting platform gives fans reliable access to the show, these remote audiences are also using alternative platforms to connect with each other. We look at Twitter activity using the #SOWOOZOO tag around the muster concerts in 2021 to understand how and when ARMY twitter interacted during the performance. Liking and retweeting activity followed the attentional demands of the concert live stream. We observed greater fan engagement during the less active times of Sowoozoo (e.g., when BTS were off stage or talking for long periods). This suggested fans did not want to look away from the exciting musical performances but did read and engage with circulating tweets when the program allowed, like in-person audience members checking in with their friends between sets. However, most original posts to the hashtag had a different pattern, reflecting fans’ needs to shout about what was on stage regardless of whether anyone was looking to read their reactions in real time. Although they contained highly relatable expressions of the tweeter’s own strong feelings of love and excitement, these reaction posts received relatively little engagement, buried quickly in the continued heavy flow on ARMY twitter’s timelines. Instead, the most popular tweets to share and like had high quality information like video clips and translations.Tweets in the middle range of engagement often included both expressions of excitement and valuable media. Disinterested in the limited interaction opportunities within the concert streaming platform, these fans adapted Twitter to serve multiple objectives, from screaming their feelings in a virtual crowd and to collecting and sharing the concert-related content they loved.
Finn Upham & Kjell Andreas Oddekalv (2022). Tapping into the flow: Proprioceptive interaction with rap in Rhythm Hive. Full text in Research Archive Show summary
Rhythm Hive is a mobile rhythm game released that features music by the Hybe bands BTS, Enhyphen, and TXT. With examples of gameplay, we describe how this app gives fans access to new levels of appreciation for rap verses, particularly for non-korean speakers and non-rappers. Interactive proprioception, a concept by game designer Matt Bloch (Miller, 2017), refers to changes in a player's perception of music as they learn movements to it. Choreography draws attention to coincidental elements of a track, allowing players to intimately and viscerally act with a recorded performance. Players’ actions in Rhythm Hive principally parallel the voiced tracks, the members’ contributions, through dense combinations of finger taps, holds, and slides. Taps to sung phrases often follow melodic pitch but the gestures for rap verses must negotiate different criteria for choosing which syllables to tap to and what shape to give mostly-monotone 16th note runs. In a rap flow - here defined as the rhythm of the words and rhymes in a piece of rap music - the rhythmic structure is created by the placement of the rapped syllables, the stress patterns of said syllables, rhymes and phrase boundaries within a music-rhythmic context. Flow, together with delivery (the way the rhythms are performed) and lyrical content constitute the rap’s expression. In Rhythm Hive, players kinesthetically experience structural features of rap flows: how rhythmic patterns are repeated with slight but significant differences, how long phrases cross music-metrical boundaries to build up tension towards an eventual release, and how phrase-endings with rhymes and punchlines are significant gestures. Comparing Easy and Hard mode sequences to select excerpts of the BTS discography, we show how gameplay illuminates the artistry in these verses by tapping along with and therefore into the flow.
Finn Upham (2022). Uncovering the active listener. Full text in Research Archive Show summary
Our experiences of music are both highly idiosyncratic, special to each of us in each moment, and collective, with common influences across a listening crowd. Through a series of empirical studies on how people feel and behave during music listening, Finn Upham traces their trajectory from a basic model of performed stimulus and audience response to an empowered-listener view of musical engagement. If participation is part of all musical experiences, this poses a question to all producers of music: “What are you asking your audience to do?”
Dana Swarbrick, Finn Upham, Cagri Erdem, Alexander Refsum Jensenius & Jonna Katariina Vuoskoski (2022). Measuring Virtual Audiences with The MusicLab App: Proof of Concept. Full text in Research Archive Show summary
We present a proof of concept by using the mobile application MusicLab to measure motion during a livestreamed concert and examining its relation to musical features. With the MusicLab App, participants’ own smartphones’ inertial measurement unit (IMU) sensors can be leveraged to record their motion and their subjective experiences collected through survey responses. The MusicLab Lock-down Rave was an Algorave (live-coded dance music) livestreamed concert featuring prolific performers Renick Bell and Khoparzi. They livestreamed for an international audience who wore their smartphones with the MusicLab App while they listened/danced to the performances. From their acceleration, we computed quantity of motion and compared it to musical features that have previously been associated with music-related motion, namely pulse clarity and low and high spectral flux. By encountering challenges and implementing improvements, the MusicLab App has become a useful tool for researching music-related motion.
Dana Swarbrick, Finn Upham, Stephen McAdams, Laurel Trainor & Julia Merrill (2022). Concert Experiment Research - Reflections on Past and Future. Full text in Research Archive Show summary
In the twenty years since the seminal concert experiments conducted by Stephen McAdams, many music cognition labs have invested resources and expertise into studying music in a concert setting. Such projects often involve collaborations between teams of researchers and artists, new technologies, challenging measurement conditions, and creative analysis strategies. As music science events, they can be very attractive to the popular press and yet difficult to report in academic circles. Complications from their bid for greater ecological validity can look like a weakness by laboratory standards, but such challenges to common practices and theories may also be a strength of this distinct research paradigm. In this moderated panel, we gather researchers with a range of experiences in researching musical concerts to share lessons learned and hopeful directions for this experimental paradigm. What should we expect to capture from musicians and audiences in these conditions? What research questions can and should be investigated with live performance and joint spectatorship? Can today’s technologies improve the methods used to conduct these studies? The symposium will include short presentations on past research with a focus on methods and strategy, some discussion of the main questions between the panelists, and a substantial portion of time devoted to discussion with the audience, as experience with concert studies is wider than the published record suggests. Specifically, the panel will consist of three sections: 1) introductions of the panelists’ expertise on concert experiment research, 2) question and answer period between panelists, and 3) an audience question period. The panel will consist of four leading researchers in the field of concert studies and it will be hosted by a doctoral researcher with expertise in the field.
Dana Swarbrick, Finn Upham, Cagri Erdem, Kayla Burnim & Alexander Refsum Jensenius (2021). The MusicLab App – Exploring the usage of mobile accelerometry to measure audience movement and respiration. Full text in Research Archive
Dana Swarbrick, Finn Upham, Cagri Erdem, Kayla Burnim & Alexander Refsum Jensenius (2021). MusicLab Algorave – An exploratory study examining the usage of mobile accelerometry to measure movements of a virtual concert audience. Full text in Research Archive
Finn Upham, Agata Zelechowska, Victor Gonzalez & Alexander Refsum Jensenius (2021). Quiet Breathing to Heard Music. Full text in Research Archive
Finn Upham (2024). Peer Review Report For: Is Taylor Swift leading a new Pop revolution? A cross-generation analysis of Pop/Rock cover songs [version 3; peer review: 1 approved with reservations, 3 not approved]. Taylor & Francis Group. Full text in Research Archive Show summary
Open review of an study on patterns in cover songs reported in the website SecondHandSongs. Review covers he contemporary cultural context of cover songs and the structure of this database, its linguistic, geographic, legal, and technical biases in this database, besides making specific recommendations for the paper.

View all works in NVA

Published Nov. 10, 2020 1:28 PM - Last modified Jan. 29, 2026 4:53 PM

Projects

Research groups

fourMs Lab