Music Tactilization

Wearable, Tactile Design, UX Research

PROJECT SUMMARY

This is my master's project that aims to investigate tactile user experience of listening to music. Based on the subjects' emotional responses in Emery Schubert's study, my project is one step further to bridge 2 tactile variables, vibration intensity and location, with 2 emotional variables, arousal and valence.

ROLE

Programmer, Maker, Researcher

TOOLS/METHODS

Pilot Study: Literature Review

Hardware: Arduino

UI/Data Collection: Processing

Data Analysis/Visualization: Matlab, Pearson Correlation, T-test

DELIVERABLE

MOTIVATION

WHERE IS TACTILE

Now we interact with digital products mainly through visual and auditory channels. Thus we have decent guidelines, frameworks, principles for good visual and acoustic design.

We are missing tactile interface in communication and information consumption.

WHY TACTILE

Intuitive when visual and audio perception channels are under overload condition.

 

Immersive in media consumption.

WHAT MAKES DESIRABLE TACTILE EXPERIENCE

You might not know. I don't know either. Then I have this project:)

INSPIRATION

Emotion is a promising dimension when defining good user experience. So I researched on 2 fields :

AFFECTIVE HAPTICS

 

Haptics has been proved to influence users' emotion. But relevant projects are too varied in terms of :

  • user scenarios (navigation, entertainment, notification, etc)

  • platforms (gloves, jacket, mobile phone, etc)

MUSIC​OLOGY

Measurement and time series analysis of emotion in music, Schubert (1999)

Music also influences emotion. In MEVD (Music Emotion Variation Detection) projects, subjects' emotional responses to music were measured and proved to be valid to be applied to other people. But none of them tried to map music features to tactile patterns. 

Hence, I wanted to consolidate tactile design and go one step further, bridging music with tactile patterns, if any, through aroused emotions.

PROPOSED DESIGN

Based on my research findings above and many others, I identified three key components for this project :

HARDWARE PLATFORM

A vest to convey tactile effects to subjects. 

EXPERIMENT DESIGN

An experiment needs to be conducted to quantify the relationship between emotional responses and tactile variables.

DIGITAL INTERFACE

A desktop application to listen to selected music, to manipulate tactile effects and to collect data with.

Two research questions need to be answered :

CORRELATION

What is the correlation, if any, between vibration intensity and emotional responses ?

SIGNIFICANCE

What is the significance, if any, of the relationship between vibration spots and emotional responses ?

PROJECT ARCHITECTURE

Emery Schubert's work

My work

ITERATION AND IMPLEMENTATION

UI DESIGN

I started sketching on paper to come up different designs, aiming to help subjects to complete the experiment.

THE VIBRATION VEST

Not everyone can fit in a down jacket. Some slim participants in my first test were unable to feel the vibration. In my later prototypes, I changed the garment to a high visibility vest.

THE EXPERIMENT DESIGN

Throughout the repetitive discussions with my team, Aoxing Zhang (data analyst) and Michael Nitsche (advisor), we kept on refining the experiment flow that could be both subject-friendly and well structured to yield valuable data.

FINAL DESIGN & IMPLEMENTATION

The UI walkthrough (developed in Processing 3)

The finalized vibration vest

12 subjects (6 males/femels) participated in the 30-minute experiment

A sample data sheet

RESULTS & DISCUSSIONS

The data shows no correlation between vibration intensity and emotional responses. Below are the scatter plots of 12 subjects' mean of vibration intensity. (when they were asked to listen to 4 selected music pieces, Adagio, Morning, Slavonic Dance and Pizzicato Polka)

The result happens maybe because :

 

  • The music pieces didn’t affect participants’ emotions during the experiment. In order to accomplish the experiment, however, subjects chose to "tactilize" music features in the music, such as the pitch and tempo.

  • Another explanation could be, for the same vibration pattern, different people might hold different meanings.

The data also shows the significance of the relationship between different vibration spots and subjects' emotional responses varies.

The significance of the relationship between different vibration locations and emotional arousal varies

The significance of the relationship between different vibration locations and emotional valence varies

FUTURE WORK

Based on some problems in this project, I list some directions worth considering for iteration :

  • Give subjects  the freedom to choose their own music pieces so that they might find themselves more emotionally related to the music

  • Use qualitative methods for the mapping process instead of quantitative ones (e.g. Pre-define some tactile "effects" and ask subjects to choose the most relevant ones based on some given descriptions)

  • Design a more intuitive interface to eliminate the latency of subject's responses (e.g. A mobile app or a gesture control app)

Designed & built by Donghai Liu 2018. 

42-20 24th St, Long Island City, NY 11101

US: +1 470-755-6674