Melody and Motion

This project aims at studying the relationship between melodic contour perception and body motion. The interaction of bodily responses to rhythmic content are fairly well understood in terms of physical motion, entrainment and synchronization. In terms of pitch material, several models have been suggested regarding the embodied nature of conceptual space as well as in the form of physical motion. In this project, we take melody as the central unit and analyse the gestural responses to melodic contour. The project will include a study of theoretical rules for well-formed melodies from various cultures. The project aims at developing methods for computational analysis of motion capture data for sound tracings. An exploration of the relationships of these to contours, and contour typologies, and additional work in the interaction of melodic contour with sound and timbre features form a part of this project.


The aim of this dissertation is to understand the role of embodiment in melodic contour perception. In other words, it studies how we move our bodies in response to music. Melodies play an important role in both speech and music. This thesis consists of two parts, the first part being a background section discussing the theoretical motivations and methods used. The second part is a collection of four articles. Each of the articles explores a dimension of melodic contour: verticality, motion metaphors, body use, and multi-feature correlational analysis. The empirical work is based on sound-tracing as an experimental paradigm. This brings together the multimodal mappings of pitched sound, gestural imagery evoked by these sounds, and defining geometries of these contours. Two sound-tracing experiments were conducted, resulting in three datasets that have been used in the analyses. In the experiments, participants listened to 16 melodies from four different genres: operatic vocalise, jazz scatting, North-Indian singing, and Sámi joik. The participants listened to each melody twice, the first time standing still, and the second time ``drawing'' the sounds in the air. Infrared motion capture was used to record the participants' body movement, and the analysis is focused primarily on the movement of their hands. The sound analysis is based on signal processing algorithms for pitch detection and methods for contour representation. Cross-correlation of the data is performed using a range of methods from statistical hypothesis testing to canonical correlation analysis. The analysis reveals that although there is a natural propensity to describe pitches in terms of the vertical dimension, the experimental data do not clearly show such an association. Average profiles of movement responses to melodies have an arch-like representation, regardless of the contour of the melody. Spatial height is associated more with relative pitch in the melodic context traced, rather than an absolute pitch scale. In addition, movement indicating metaphoric representation of sound was used more often by participants.


This project uses the following datasets and methodologies for understanding the motor representations of melodic contour. 



Sound Stimuli Distribution:

Genre \ Stimulus Presentation Recordings Pitch extracted + Resynthesized
Joik 4 4
North Indian Classical Music 4 4
Jazz Scat 4 4
Vocalise 4 4

Video examples of sound tracings:

0. A vanilla tracing
1. Percussive tracing
2. Inspired by genre (posture change)
3. A tracing representing high pitch does not necessarily have to be large in height. A different feature might be more important.




Published Aug. 9, 2017 11:34 AM - Last modified Nov. 23, 2019 7:07 PM