Sola – A system for preventing music-related sufferings and analysing the play

Image showing a girl playing the cello with a sensor wristband.

"About half of the music students experience a phase with play-related complaints during their university studies."

[Source: Spahn, Claudia (2015): Musikergesundheit in der Praxis. Grundlagen Prävention Übungen. Leipzig: Henschel Verlag in der Seemann Henschel GmbH & Co. KG., S.129]

Context

Play-related complaints from prospective professional musicians are not unusual. Strong physical strain, for example through long practice or the posture on the instrument, can lead to instrument-specific physical complaints. Psychological problems can also arise and, in correlation, intensify physical complaints.

Challenge

The aim of this work was to provide musicians a way to analyse their play while and after they practise. Through that they have better control over their playing and can prevent music-related complaints.

Outcome

Sola is a system for musicians that collects sensor data while playing and interprets and visualizes them. In addition it gives direct feedback in case of potentially health-endangering execution. It promotes self-control during practice and supports the prevention of play-related complaints

→ How might we help young musicians to stay healthy and prevent music-related complaints.

"How might we" question as a kick-off for the project

Solution

Image showing a person holding an Ipad with the app of the sola system.

Before practising

The student can look through the data of his/her last practice sessions and see how he/she has progressed. Before he/she starts the session, he/she puts on the sensor wristband and connects it to the tablet.

A library of sheet music is included in the app and can be enriched by uploading new sheet music. In a separate folder, the student can store and quickly retrieve his current repertoire.

As soon as the user selects a piece of music, he/she gets an enlarged view of the sheet music with all the comments and highlights from the past practice sessions. A loading animation shows the status of the pairing process.

Starting to practise

During the practice the sensor will analyse the movement and muscle tension and provide real time feedback.

The feedback is given in the form of a red frame around the notes and a red mark below the score line. In addition, a hint is given on how he/she should change his/her behavior. These hints are inspired by the instrumental teacher's interjections during the lesson.

While practicing, the student has the possibility to view all the hints and notes that have been created while playing. A popup shows more detailed information but also suggestions for other exercises.

After practising

Prior to and after practicing, a summary of all indications can be viewed via the "Daily Overview". In addition to the description of the problem, the system provides suggestions for changing and adjusting the posture or playing technique, as well as recommendations for suitable exercises.

By selecting the individual points, the student receives further information and can also listen to the corresponding passage again through the recorded audio track.

All sensor data are also stored as raw data and can be viewed under the Data Visualization section.

Other functions

In addition to the analysis of the play, the student is given a history of his progress and a collection of different exercises that can be done before, during or after practice session.

Screen showing the monthöy overview.

The student is shown an overview of his last practice sessions and a short summary of general advices.

Screen showing the comparison option in the sola app.

Under the section "comparison", the data of two time periods can be compared with each other. For this purpose, they are selected by a selection field in the lower half of the screen. Only the dates on which the selected part of the piece of music was practiced are displayed.

Screen showing an overview of different exercise options.

Under the section "exercises" there are a number of different exercises, which are guided with instructions. The system suggests exercises based on the data analysis, which can be found under the heading "suggestions".

Research

Clinical pictures

There is a large number of clinical pictures that occur in connection with making music. Among them are not only physical, but also a number of psychological clinical pictures.Physical and psychological complaints cannot be considered separately, however, since they influence each other. Depending on the instrument group and instrument, different clinical pictures are more typical than others. String players and pianists often have problems with their backs and hands. Especially in the wrists.

In the following I have focussed on this area. This includes clinical pictures from the area of pain syndromes, hand problems and shoulder and neck tensions, which can be a consequence of problems with the hands.

Illustration of possible clinical pictures in form of a mind map.

An overview of different clinical pictures

Triggers

Many problems that arise from playing an instrument have no clear cause and have been provoked by various triggers. It is important to be able to intervene early and take preventive and health-promoting action. Therefore physical and psychological factors should be considered equally.

Possible triggers can be:

→ New movement patterns

→ Non-optimal ergonomics

→ Playing to the point of pain

→ Abruptly intensified practise sessions

Interviews

One of the aims of the interviews was to find out whether a specific clinical picture should be investigated in the further course of the project or a broader topic. For this purpose, two prospective musicians, who were pursuing their instrumental studies at the time, and an instrumental teacher were interviewed.

In the conversation with the students, who are themselves affected by play-related pain and complaints, information about the history of their illness and the treatment, or how they deal with it now, were of great importance.

By talking with an instrumental teacher, questions about the teacher's responsibility to the students should be clarified. General questions about technique/attitude and different disease patterns were also answered.

"During lessons, it's important to find the ideal position for the student. That means observing and adapting and not letting the student play until something happens and the problem occurs".

Instrumental teacher

"Of course, all the treatments [physiotherapy and other] helped, but I think it was mainly the psychological aspect that got you so involved. In the sense of, you have to be good now, you have to practice now, if you don't practice now, then you won't be able to do the solo concert...".

Violin student

Top Findings

The research and interviews resulted in the following findings, on which I was able to formulate first action points

1. The issue of music related complaints is often a taboo among musicians

→ Breaking the taboo

2. The instrumental teacher can only detect problems through symptoms outside the student's body

→ Accurate analysis through sensors

3. Preventive and health-promoting measures are usually only taken when problems occur

→ Little additional effort and integration into everyday lifes

4. Especially for students it is hard to pay attention to many different things when practicing (posture, technique, musicality, scores)

→ External feedback

Conception

Image showing girl standing in front of a wall full of postits and scribbles.

Mapping first wireframes for choosing a direction

First Ideas

The first ideas emerged during the research and discussions with musicians and music students. To get a better overview of possible directions, the ideas were assigned to groups. Roughly, the ideas can be divided into three groups. Firstly, projects in an educational direction, among which falls the group "Awareness/Communication". The second larger group are different ideas for behavior analysis, to take preventive measures and to analyze behavior. This includes the groups "movement/video analysis", "posture analysis", "measurement of muscle tension" and "pressure measurement". The third group contains ideas for optimizing the technique.

First scribbles of ideas on postits.

Storyboards

In order to get a more precise understanding of different directions, two concepts were concretized and visualized through a storyboard, each illustrating a use case. The two concepts selected were, a concept from video and motion analysis and, secondly, one from the group of muscle tension measurement.

Hand drawn storyboards for two of the concepts.

Validating Concept 1 and 2

In order to test the two concepts as quickly as possible, I built two first prototypes.

For concept 1 I firstly tested an EMG sensor, which was connected to the Arduino to measure the muscle tension. I decided to use the MYO wristband instead, because it is an already working product that has EMG sensors built in and is very suitable for testing. Additionally there is a already existing platform that shows the live data which was helpful in the fist testing phase.

The second concept was tested using an open source machine learning software. The software automatically detects the body parts and draws a skeleton directly over the live image. In this prototype, the transparency of the video image could be changed via a slider, so that only the skeleton can be viewed in order to better analyze movements

Image showing the prototyp of the first concept. Girl playing the cello with the MYO sensor wristband on the left and laptop with the live data on the right.

Prototyp of concept 1

Image showing the prototyp of the second prototyp. Augmented reality mapping on the video of a girl playing the cello.

Prototyp of concept 2

Core Concept

Structure

The system consists of three components, the sensor, the database and a digital device. Data is measured by one or more sensors on the instrument, or directly on the body, and stored in a database in the cloud. The graphical interface serves as a visualization option of the real-time data and the collected data sets. This setup is the same for each sensor.

Illustration of different parts of the sola system like sensor armband, cloud and tablet.

1. The live data is sent to the tablet via Bluetooth and displayed there in real time with the scores.

2. If a problem is noticed by the sensors (for example to much pressure while playing or not taking any breaks), the system is alerted and gives feedback in form of a visual and tactile feedback in form of vibration.

3. The data is stored in a database while playing the instrument with a timestamp so that it can be accessed and viewed at a later time.

4. Stored data is visualized by the digital application after playing. In addition, different data sets can be compared with each other.

5. If entries are made through the digital application, they are stored in the database.

Information Architecture

The information architecture served as the basis for the structure and design of the screens.

Image showing the information architecture.

Prototyping

Use Case

This use case is an example of everyday music practice with the system. The person making the music, in this case a student preparing to study cello (17 years old), is familiar with the system and has been using it for a few weeks.

01 Putting on the sensor wristband

The student wants to start practicing and is preparing to do so by putting the sensor bracelet on his arm. She also has her tablet in front of her.

02 Start of the practice session

Before starting, however, the student looks at the detailed view of the notes to see where he needs to pay particular attention and how he can improve in general.

03 Data collection during practice

The student starts the practice session and practices his/her current repertoire. Starting with etudes and finger exercises. Meanwhile, data is collected by the wristband and sent to the system's database.

04 Feedback on problematic parts

The student starts to tense up during a challenging part. The system detects this and classifies the behavior as problematic and gives the student visual feedback. The feedback allows the student to adjust while playing.

05 Overview of the completed practice session

After practicing, the student looks at the summary of the completed practice unit and can listen to the audio file of the matching passage by using the playback function at marked points.

Wireframes

First wireframes were made with pen and paper to quickly map variants. In the next step, they were digitized so that they could be checked directly on the end device.

Image showing various wireframe scribbles.

Sitemap

The sitemap shows all different screens of the application. For the purpose of the thesis I designed all screens except the login and settings.

Image showing the sitemap of the app.

Grid and Spacing

Screen showing the monthöy overview.
Screen showing the comparison option in the sola app.
Screen showing an overview of different exercise options.

Visual Design

The visual design of the application is built on Apple's design system. The elements are kept very simple and colors were used mainly for data visualization. All icons and illustrations were created by me to fit well with the rest of the design.

Icon library created for the application

Illustrations created for the application

Project type

Bachelor thesis, Studentproject at University of Applied Science Schwäbisch Gmünd (HFG)

Year and Duration

2019, 4 months

Supervision

Prof. David Oswald and Prof. Benedikt Groß

My Role

Research, Conception, Prototyping, Visual Design

Tools

Miro, Sketch, MYO Armband, Ml5.js, PHP and MySQL, Javascript, Htlm and CSS, Principle