Presentation at the International Congress on Acoustics (ICA)

The 22nd International Congress on Acousitcs (ICA 2016) will be held September 5-9th, Buenos Aires, Argentina. Minho Song’s paper “Hidden melody in music playing motion: Music recording using optical motion tracking system” was selected for the oral presentation in the Musical Acoustics session.


This paper shows a feasibility study of recording a sound using optical marker-based motion tracking cameras. Optical marker-based motion tracking system can record the motion of moving object using multiple high-speed infrared (IR) cameras. Recent development of the device enables capturing the detailed motion; with high spatial precision of 0.01m and high sampling rate up to 10kHz. Therefore, not only the global movements of human body or handheld instruments but also the local acoustic vibrations can be recorded within the motion data, which can be transformed to actual sound radiating from the acoustic instrument.

To evaluate the feasibility, several lightweight reflective markers were attached to various positions on the string instruments. Several musical excerpts were selected considering the cameras’ Nyquist sampling rate. The loudness of the excerpts was controlled while playing the instruments. The playing motions were recorded with a high-quality optical motion tracking system. Since the global motion trajectory is a relatively slow motion having the frequency component lower than 10Hz, an audible signal could be retrieved from the motion tracking data with low-pass filter. Although the current professional motion tracking system requires significantly high signal-to-noise ratio and can only retrieve the sound up to far less than 5kHz, but the result of the experiment shows that the optical marker-based motion tracking system can be useful in recording sound information from visual domain.

Figure 1: 6-string bass guitar playing for the experiment of sound recording using motion capture cameras. The motion of ‘A’ string (yellow point in right-top figure) contains both musician’s motion and also the string vibration (right-bottom figure).


Figure 3: Chromatic scale is played and retrieved using motion capture cameras


Published Sep. 1, 2016 12:00 AM