Listen2MySound – Contextual Information Based on Pervasive Speech Analysis


Over the last years, the increasing ubiquity of smartphone devices allowed to completely redraw their traditional concept and applicability. The increasing advances on technology allowed integrating a multitude of different sensors, capable to ’sense’ the world that surrounds the users and their actions, allowing to perform a continuous Human Activity Recognition (HAR). This topic has become of high interest for medical applications. On the one hand, patients with chronic diseases are required to follow pre-defined exercises as part of their long-term treatments. On the other hand, the evolution of social interactions promoted the interest and need for ’self-measurement’.

Traditionally, Human Activity Recognition (HAR) uses a first layer of information based on inertial data which proved to be a valuable asset. However, common available architectures can be improved, if additional sensing mechanisms are introduced to sense the context where the user is inserted.

This project introduced a new layer of information using pervasive sound analysis from the built-in smartphone’s microphones. Sound is everywhere on daily life and certain actions and environments create well-defined fingerprints which can help to increase today’s HAR architectures performance11. Examples of such actions include the sound promoted by closing/opening doors, walking outdoors/indoors and walking on stairs, which help to locate user and help to identify his activity; the sounds of Human speech, which may eventually address social behaviours; the absence of sound, which may help to assure correct circadian rhythms are being followed. Ultimately, an earlier identification of continuous exposure to high intensity noises, which constitute a potential health hazard, may also prevent potential ear diseases.

This project focused on the development of a framework for HAR and Indoor Location using the sound perceived through the device’s microphone, which can enhance the current system’s performance, based on their actions and context.

Author: Ricardo Leonardo

Type: MSc thesis

Partner: Faculdade de Ciências e Tecnologia da Universidade Nova de Lisboa

Year: 2017



Project Flyer