Music and Haptics: Musical Haptic Wearables for Electronic Music Performers’ Communication

Guest post, by Luca Turchet

The Internet of Musical Things (IoMusT) is an emerging research field that lies at the intersection of the Internet of Things and sound and music computing, with a particular focus on multisensory aspects. The IoMusT relates to the network of Musical Things, which are objects and interfaces dedicated to the production, interaction, and reception of musical content. Musical Things embed electronics, sensors, data forwarding and processing software, and network connectivity enabling the collection and exchange of data for musical purpose.

Another emerging research field is “musical haptics”, which investigates the application of haptics research to the musical domain. Some studies within this field have focused on the analysis of how haptic cues affect musicians’ experience and performance. Recently, at the Centre for Digital Music of Queen Mary University of London we have applied IoMusT technologies to the musical haptic domain to investigate the role of haptic cues on communication between electronic music performers.

Communication among performers is a fundamental aspect in music performance. A large number of electronic music instruments based on tangible and screen-based interfaces require a focused visual attention from performers while they are controlled. In certain stage and artistic configurations, this may be an obstacle to face-to-face creative interactions between co-performers and their collaborators.

To address these issues, we adopted a user-centered design methodology to develop a novel class of IoMusT devices that we term “musical haptic wearables for performers”. Such a novel class of wearable devices targeting music performers encompasses haptic stimulation, gesture tracking, and wireless connectivity features. Musical haptic wearables were conceived to enhance creative communication among performers as well as between performers and audience members by leveraging the sense of touch in both co-located and remote settings.

We conducted a co-design workshop with 10 electronic musicians from the “Orchestra Elettroacustica Officina Arti Soniche San Pietro a Majella” of Naples (Italy). This workshop identified numerous creative communication issues among performers in electronic music practice and resulted in mock-up prototypes.

We then developed three chest-, foot-, and arm-worn haptic wearables respectively for co-performer, performer–conductor, and performer–sound-engineer interactions.

Fig. 1: Armband used by a musician while playing. Photo: Luca Turchet

Fig 2: The foot-worn system used by a live sound engineer sitting in front of a mixing desk. Photo: Luca Turchet

The wearables were evaluated throughout performance-experiments conducted in the Performance Lab of Queen Mary University of London, a venue where concerts are regularly held. This provided a simulated real-world setting of a live electronic music rehearsal without audience.

The video above shows one of such performance-experiments where the armband-based prototype was used by groups of four electronic music performers to exchange musical directions during improvisations. By clicking the buttons placed onto their wearable, the performers could wirelessly send messages to the wearables of the other fellow musicians. When received, such messages were immediately rendered by the wearable as tactile vibrations. Each vibration was associated to a specific musical direction previously agreed between performers such as “start playing”, “stop playing”, “change section slowly”, “change section rapidly”, and “make eye contact”.

In the video it is possible to see one of the performers acting as a conductor directing the other three performers. When receiving the vibrations the three performers reacted to them according to the predefined vibration-direction vocabulary.

Overall, the three types of wearables were assessed with 25 participants. High accuracies were obtained for musical actions expected after instructions wirelessly communicated via tactile signals. The results of our experiments provide evidence that musical haptic wearables can be an effective medium of communication in the context of electronic music performances.

Further details about the research briefly described in this post can be found on the following scientific publication:

1. Turchet and M. Barthet. Co-design of Musical Haptic Wearables for electronic music performer’s communication. IEEE Transactions on Human-Machine Systems, 2018 (In press). https://ieeexplore.ieee.org/document/8585031

A video documenting the use of the musical haptic wearables co-designed in this study can be accessed at: https://youtu.be/73b0JgXkmJo

Further information about the topics described in this post can be found on www.iomut.eu, the website of the project “Towards the Internet of Musical Things” funded by a Marie-Curie Action of the European Commission.