Qichao Lan

In acoustic instruments, sound production relies on the interaction between physical objects. Digital musical instruments, on the other hand, are based on arbitrarily designed action--sound mappings. This paper describes the ongoing exploration of an empirically-based approach for simulating guitar playing technique when designing the mappings of 'air instrument' designs. We present results from an experiment in which 33 electric guitarists performed a set of basic sound-producing actions: impulsive, sustained, and iterative. The dataset consists of bioelectric muscle signals, motion capture, video, and audio recordings. This multimodal dataset was used to train a long short-term memory network (LSTM) with a few hidden layers and relatively short training duration. We show that the network is able to predict audio energy features of free improvisations on the guitar, relying on a dataset of three distinct motion types.

We investigated how the action–sound relationships found in electric guitar performance can be used in the design of new instruments. Thirty-one trained guitarists performed a set of basic sound-producing actions (impulsive, sustained, and iterative) and free improvisations on an electric guitar. We performed a statistical analysis of the muscle activation data (EMG) and audio recordings from the experiment. Then we trained a long short-term memory network with nine different configurations to map EMG signal to sound. We found that the preliminary models were able to predict audio energy features of free improvisations on the guitar, based on the dataset of raw EMG from the basic sound-producing actions. The results provide evidence of similarities between body motion and sound in music performance, compatible with embodied music cognition theories. They also show the potential of using machine learning on recorded performance data in the design of new musical instruments.

From the "Music for stem cells: Aiming to cure diabetes with sound waves" Oslo Life Science Conference event.

An interactive art installation presented during the International Conference on New Interfaces for Musical Expression (NIME) 2020.

QuaverSeries is a live coding language that explores the balance between minimalism and readability in live coding language syntax design. It adopts a parenthesis-free syntax and forms a left-to-right typing flow. It was first implemented with Ohm.js and Tone.js, but in this new branch of QuaverSeries, it is reimplemented with Sema Live Coding Language Ecosystem and focuses on signal modulation. In this performance, the performer will be improvising on sound synthesis and seek to arrive at a self-playing state.

In this project, QuaverSeries live coding language is reimplemented in Sema Ecosystem. The syntax remains identical while the previous audio engine Tone.js is replaced by the one in Sema.