Think about a room you like spending time in. What makes the room feel good? How does it look? How does it sound? And how does that affect your emotions and behavior? As a part of the AMBIENT project, this thesis examines how environments affect human physiological responses, with a particular focus on auditory-visual stimuli. The thesis approaches the problem from a novel angle: music technology. Architects, interior designers, and acousticians focus on enhancing the functional and aesthetic characteristics of rooms: they typically treat rooms as objects. In this thesis, the room is chosen to be the 'subject' that produces sound and visuals.
The characteristics and perception of immersive auditory-visual rhythms has been touched upon by multiple different disciplines from different perspectives, and this thesis tries to merge their works to have a more comprehensive view of the question. Therefore, the thesis starts from music technology but includes multiple disciplines. These include music information retrieval, machine learning, immersive media technologies, auditory-visual theories, and embodied perception theories.
The core theoretical development of this thesis draws on Michel Chion's concept of synchresis: the idea that we do not simply hear and see films but hear/see them in a unified manner. This concept is developed to machine synchresis, defined as an extension of music information retrieval and machine listening to encompass the aforementioned disciplines. The goal of the proposed framework is to bridge the gap between audio-video machine learning and auditory-visual perception theory, by introducing concepts in synchresis theory into audio-video signal processing and neural networks. Additionally, the embodied perception theory is adopted to investigate how auditory-visual stimuli affect human bodily reactions through multi-modal neural networks. The spatial-temporal characteristics are studied with Ambisonic audio and 360-degree video processing algorithms. Contributions of the thesis include theoretical developments, curated multimodal datasets (SoundActions, StillStanding365), coding toolboxes for immersive audio/video (MGT-Python, Ambiviz), and multi-modal signal processing and machine learning algorithms (EoPMA, Ambient-rPPG).
Jinyue Guo successfully defended his thesis on 22 September 2026.
Trial lecture
Designated topic: Applying Michel Chion’s concept of synchresis in multimedia research.Time and place: Tuesday 22 September, 11.15.
Thesis introduction
First opponent
Second opponent
Evaluation committee
- Professor Antonio Camurri, University of Genova (first opponent)
- Assistant Professor Annamaria Mesaros, Tampere University (second opponent)
- Professor Emeritus Rolf Inge Godøy, University of Oslo (committee administrator)
Chair of the defence
- Associate Professor Judith Irmela Haug, University of Oslo
Supervisors
- Professor Alexander Refsum Jensenius, University of Oslo
- Professor Jim Tørresen, University of Oslo