Sola – A system for preventing music-related sufferings and analysing the play
"About half of the music students experience a phase with play-related complaints during their university studies."
[Source: Spahn, Claudia (2015): Musikergesundheit in der Praxis. Grundlagen Prävention Übungen. Leipzig: Henschel Verlag in der Seemann Henschel GmbH & Co. KG., S.129]
Play-related complaints from prospective professional musicians are not unusual. Strong physical strain, for example through long practice or the posture on the instrument, can lead to instrument-specific physical complaints. Psychological problems can also arise and, in correlation, intensify physical complaints.
The aim of this work was to provide musicians a way to analyse their play while and after they practise. Through that they have better control over their playing and can prevent music-related complaints.
Sola is a system for musicians that collects sensor data while playing and interprets and visualizes them. In addition it gives direct feedback in case of potentially health-endangering execution. It promotes self-control during practice and supports the prevention of play-related complaints
→ How might we help young musicians to stay healthy and prevent music-related complaints.
"How might we" question as a kick-off for the project
The student can look through the data of his/her last practice sessions and see how he/she has progressed. Before he/she starts the session, he/she puts on the sensor wristband and connects it to the tablet.
A library of sheet music is included in the app and can be enriched by uploading new sheet music. In a separate folder, the student can store and quickly retrieve his current repertoire.
As soon as the user selects a piece of music, he/she gets an enlarged view of the sheet music with all the comments and highlights from the past practice sessions. A loading animation shows the status of the pairing process.
Starting to practise
During the practice the sensor will analyse the movement and muscle tension and provide real time feedback.
The feedback is given in the form of a red frame around the notes and a red mark below the score line. In addition, a hint is given on how he/she should change his/her behavior. These hints are inspired by the instrumental teacher's interjections during the lesson.
While practicing, the student has the possibility to view all the hints and notes that have been created while playing. A popup shows more detailed information but also suggestions for other exercises.
Prior to and after practicing, a summary of all indications can be viewed via the "Daily Overview". In addition to the description of the problem, the system provides suggestions for changing and adjusting the posture or playing technique, as well as recommendations for suitable exercises.
By selecting the individual points, the student receives further information and can also listen to the corresponding passage again through the recorded audio track.
All sensor data are also stored as raw data and can be viewed under the Data Visualization section.
In addition to the analysis of the play, the student is given a history of his progress and a collection of different exercises that can be done before, during or after practice session.
The student is shown an overview of his last practice sessions and a short summary of general advices.
Under the section "comparison", the data of two time periods can be compared with each other. For this purpose, they are selected by a selection field in the lower half of the screen. Only the dates on which the selected part of the piece of music was practiced are displayed.
Under the section "exercises" there are a number of different exercises, which are guided with instructions. The system suggests exercises based on the data analysis, which can be found under the heading "suggestions".
There is a large number of clinical pictures that occur in connection with making music.
Among them are not only physical, but also a number of psychological clinical pictures.Physical and psychological
complaints cannot be considered separately, however, since they influence each other. Depending on the instrument
group and instrument, different clinical pictures are more typical than others.
String players and pianists often have problems with their backs and hands. Especially in the wrists.
In the following I have focussed on this area. This includes clinical pictures from the area of pain syndromes, hand problems and shoulder and neck tensions, which can be a consequence of problems with the hands.
An overview of different clinical pictures
Many problems that arise from playing an instrument have no clear cause and have been
provoked by various triggers. It is important to be able to intervene early and take preventive and
health-promoting action. Therefore physical and psychological factors should be considered equally.
Possible triggers can be:
→ New movement patterns
→ Non-optimal ergonomics
→ Playing to the point of pain
→ Abruptly intensified practise sessions
One of the aims of the interviews was to find out whether a specific clinical picture should be investigated in
the further course of the project or a broader topic. For this purpose, two prospective musicians, who were
pursuing their instrumental studies at the time, and an instrumental teacher were interviewed.
In the conversation with the students, who are themselves affected by play-related pain and complaints, information about the history of their illness and the treatment, or how they deal with it now, were of great importance.
By talking with an instrumental teacher, questions about the teacher's responsibility to the students should be clarified. General questions about technique/attitude and different disease patterns were also answered.
"During lessons, it's important to find the ideal position for the student. That means observing and adapting and not letting the student play until something happens and the problem occurs".
"Of course, all the treatments [physiotherapy and other] helped, but I think it was mainly the psychological aspect that got you so involved. In the sense of, you have to be good now, you have to practice now, if you don't practice now, then you won't be able to do the solo concert...".
The research and interviews resulted in the following findings, on which I was able to formulate first action points
1. The issue of music related complaints is often a taboo among musicians
→ Breaking the taboo
2. The instrumental teacher can only detect problems through symptoms outside the student's body
→ Accurate analysis through sensors
3. Preventive and health-promoting measures are usually only taken when problems occur
→ Little additional effort and integration into everyday lifes
4. Especially for students it is hard to pay attention to many different things when practicing (posture, technique, musicality, scores)
→ External feedback
Mapping first wireframes for choosing a direction
The first ideas emerged during the research and discussions with musicians and music students. To get a better overview of possible directions, the ideas were assigned to groups. Roughly, the ideas can be divided into three groups. Firstly, projects in an educational direction, among which falls the group "Awareness/Communication". The second larger group are different ideas for behavior analysis, to take preventive measures and to analyze behavior. This includes the groups "movement/video analysis", "posture analysis", "measurement of muscle tension" and "pressure measurement". The third group contains ideas for optimizing the technique.
In order to get a more precise understanding of different directions, two concepts were concretized and visualized through a storyboard, each illustrating a use case. The two concepts selected were, a concept from video and motion analysis and, secondly, one from the group of muscle tension measurement.
Validating Concept 1 and 2
In order to test the two concepts as quickly as possible, I built two first prototypes.
For concept 1 I firstly tested an EMG sensor, which was connected to the Arduino to measure the muscle tension. I decided to use the MYO wristband instead, because it is an already working product that has EMG sensors built in and is very suitable for testing. Additionally there is a already existing platform that shows the live data which was helpful in the fist testing phase.
The second concept was tested using an open source machine learning software. The software automatically detects the body parts and draws a skeleton directly over the live image. In this prototype, the transparency of the video image could be changed via a slider, so that only the skeleton can be viewed in order to better analyze movements
Prototyp of concept 1
Prototyp of concept 2
The system consists of three components, the sensor, the database and a digital device. Data is measured by one or more sensors on the instrument, or directly on the body, and stored in a database in the cloud. The graphical interface serves as a visualization option of the real-time data and the collected data sets. This setup is the same for each sensor.
1. The live data is sent to the tablet via Bluetooth and displayed there in real time with the scores.
2. If a problem is noticed by the sensors (for example to much pressure while playing or not taking any breaks), the system is alerted and gives feedback in form of a visual and tactile feedback in form of vibration.
3. The data is stored in a database while playing the instrument with a timestamp so that it can be accessed and viewed at a later time.
4. Stored data is visualized by the digital application after playing. In addition, different data sets can be compared with each other.
5. If entries are made through the digital application, they are stored in the database.
The information architecture served as the basis for the structure and design of the screens.
This use case is an example of everyday music practice with the system. The person making the music, in this case a student preparing to study cello (17 years old), is familiar with the system and has been using it for a few weeks.
01 Putting on the sensor wristband
The student wants to start practicing and is preparing to do so by putting the sensor bracelet on his arm. She also has her tablet in front of her.
02 Start of the practice session
Before starting, however, the student looks at the detailed view of the notes to see where he needs to pay particular attention and how he can improve in general.
03 Data collection during practice
The student starts the practice session and practices his/her current repertoire. Starting with etudes and finger exercises. Meanwhile, data is collected by the wristband and sent to the system's database.
04 Feedback on problematic parts
The student starts to tense up during a challenging part. The system detects this and classifies the behavior as problematic and gives the student visual feedback. The feedback allows the student to adjust while playing.
05 Overview of the completed practice session
After practicing, the student looks at the summary of the completed practice unit and can listen to the audio file of the matching passage by using the playback function at marked points.
First wireframes were made with pen and paper to quickly map variants. In the next step, they were digitized so that they could be checked directly on the end device.
The sitemap shows all different screens of the application. For the purpose of the thesis I designed all screens except the login and settings.
Grid and Spacing
The visual design of the application is built on Apple's design system. The elements are kept very simple and colors were used mainly for data visualization. All icons and illustrations were created by me to fit well with the rest of the design.
Icon library created for the application
Illustrations created for the application
Bachelor thesis, Studentproject at University of Applied Science Schwäbisch Gmünd (HFG)
Year and Duration
2019, 4 months
Prof. David Oswald and Prof. Benedikt Groß
Research, Conception, Prototyping, Visual Design