Writing

ImmerseGuard: Bridging Immersion and Environmental Awareness

In short: ImmerseGuard is a phone app that listens through your noise-cancelling headphones and tells you when something important is happening around you. It runs on the device, with no internet…

published
read time
2 min
words
389
lang
en

In short: ImmerseGuard is a phone app that listens through your noise-cancelling headphones and tells you when something important is happening around you. It runs on the device, with no internet connection needed.

The problem with good noise cancelling

Noise cancelling has got very good. That is great for music and for immersive content, and it is a problem for everything else. Once the room is gone, so are the sounds you actually needed to hear.

Conceptual framework: sound events such as door knocks, speech, sirens, pets, alarms and cars reach people using noise cancelling; the earbud microphones feed a real-time on-device sound event detector that sends text or voice push notifications
Wherever users switch on ANC, whether walking down the street, doing something at home or inside a VR headset for full immersion, ImmerseGuard analyses the headphone audio in real time and alerts them with a text or voice notification when it detects a sound event.

What we built

ImmerseGuard puts an on-device sound event detection system alongside active noise cancelling (ANC). It is compressed enough to run on a phone, and it gives you real-time alerts for the sounds that matter without pulling you out of what you are listening to.

App running with Speech and Emergency Vehicles ticked, showing a live classification of Speech with a score of 0.67
App start screen with checkboxes for Speech, Vehicles, Emergency Vehicles, Electronic Alarms, Room and Household, and Pets, and a Start button
Android notification shade showing a Listen Safely alert that says Speech detected
The app listening for the chosen sound classes, and the notification it sends when speech is detected.

How it works

The core is YAMNet, which is known for doing sound event detection efficiently on edge devices. We pair it with a shallower model trained by transfer learning. Together they pick out emergency vehicle sirens, general vehicle noise, room and household sounds, human speech, pets and device alarms.

To make it fit a resource-limited platform we quantized the model. That gets inference times down to something usable in real time and lets the whole thing run locally, with no internet connection.

User experience benchmark chart: mean scores for pragmatic quality, hedonic quality and overall plotted against bands from bad to excellent, with pragmatic quality highest at about 1.8

What happened when people used it

The user experience studies came back positive. People found it practical for staying aware of their surroundings while immersed in digital content, and they liked that it just sat on their phone. No heavy files or models to download.

What is next

The studies had limits and we know it. The next round is about making ImmerseGuard work across a wider range of devices and real environments. From user feedback, we also want more specific sound classes and notifications people can tune themselves.

Balancing immersive content against staying connected to the room is the whole point. On-device machine learning and model compression are what make it possible.

You can read the full paper on ResearchGate.