Dolby Dynamic captioning
An expressive open captioning solution to help deaf and hard of hearing movie-goers better enjoy their theater experience.
6 week project | role: Project management, concepting, user experience, co-animator
The Problem
Going to the movies can be a stressful and unenjoyable experience for the deaf or hard of hearing community. Captioning devices are unreliable and they are not inclusive, ostracizing hard of hearing people when they go to the movies. The captions themselves lack the depth of emotion that hearing viewers experience.
“Going to the movies is not a relaxing experience. I have to work to understand each sentence, or else I’ll miss everything.”
@yellowtonkatruck, auditory processing disorder
Opportunity
How can we design a product that is inclusive for all hard of hearing populations and imbue more emotion than is present in the current captions?
Problem set-up
Solution
Dolby Dynamic Captions creates a visual and emotional representation of all of the audio that creates the atmosphere in a movie, not just the words.
Infrared glasses
The captions are visible on screen using only a pair of infrared filtered glasses. Using infrared technology, viewers wearing the glasses will be able to see the captions and animations on screen without distracting other moviegoers.
Extended Screen
With an expanded projection screen, viewers wearing the glasses will be able to see captions and animations both on and off of the traditional screen. This way, music visualizations can also live underneath the screen without taking away from what’s happening in the movie.
The Captions
Dynamic Captioning creates much-needed context for scenes by using animations, typefaces, and placement of written dialogue to convey the speaker, volume, tone, and other variables.
Prototyped animations
In a scene with really only one word, the music and tone really set the emotion for the scene.
Why Dolby
Since its inception, Dolby's mission has been to “make experiences come alive through technology.” The brand also strives to “give every guest a spectacular viewing experience.” Right now, Dolby is not living up to these promises.
Dolby currently makes the Captiview technology, which does not do enough to make movies accessible and inclusive for deaf and hard of hearing moviegoers. We believe that this solution is in line with Dolby's values and their mission.
Dolby would provide the infrastructure necessary for Dynamic Captioning, including the glasses and expanded screens. Dolby would also provide a set of design guidelines that inform film makers how to create emotion in captions for three main categories: dialogue, sound effects, and music. Production companies will then go on to create the actual captions to accompany their films.
User feedback
“This is the first time anyone has ever wanted to listen to me about my [movie] experiences. And I’ve had years to think about all this stuff in silence.”
@yellowtonkatruck, auditory processing disorder
“Like ASL, there are multiple semantic indicators, which give them a kind of fluidity and a kind of poetry that I think Deaf folks would immediately understand and respond to”
“Conveying sound through images is something I love in comic books, so I’d be excited to see it in a movie!”
— @mightyoctopus, deaf in one ear
“If you and your team are successful with this concept and able to implement it nationwide, I’d definitely enjoy going to the movies more.”
— @divinejourneydesigns, hard of hearing
Process
Design Guidelines
We identified three main components of the audio in a movie: dialogue, sound effects, and music. A set of guidelines serve as inspiration for how to visualize some of the sounds featured in movies.
Dialogue
Dynamic Captioning creates much-needed context for scenes by using animations, typefaces, and placement of written dialogue to convey the speaker, volume, tone, and other variables.
Exclamation
Defeat
The reactive text trembles as the exhausted swimmer finally yields to the ocean. His efforts have culminated in an overwhelming defeat, physically and emotionally.
Yelling
Expanded character distance and capitalization reflect increased volume and intensity, building the hysteria of the moment in a visual manner.
Singing
The musicality of this delivery is distinct from the spoken exchanges in the rest of the scene. Text is reactive to the inflection and cadence of the eerie tune and is drawn upward to resemble the pitch frequency.
Sound Effects
Sound effects can include paralanguage, on- or off-screen sounds, ambient noise, and silence, among other elements. Dynamic Captioning uses on-screen animations and captions to more accurately translate the emotional context of the scene.
Gasping
This sound effect is made distinct with brackets and an alternative typeface. Additional descriptors add dimension to the emotions these sounds represent in the context of the scene.
Chimes
Layered sounds, imagery, and quick cuts make it difficult to discern the elements that contribute to the chaos of the scene. This brief but distinct new sound is represented by a cluster of unique symbols that are organized around the modulation of the sound.
Chatter
The overlapping chatter is difficult to isolate as useful information in a scene where the characters’ anxiety is mounting. Fluctuating scribbles oscillate to illustrate the racket and discord of the scene’s passengers.
Music
Music is often crucial to the emotional resonance of a scene. Dynamic Captioning includes waves that synchronize to the score. The animated waves involve color and move in correspondence to intensity and rhythm of the sound levels, translating the dominant feeling from audio to visual. Intensity can be portrayed through placement, moving higher up on the screen as the intensity builds.
Score
The fluidity of the music is articulated by particles that swell in response to the rhythm of the scene.
Crescendo
A surge in volume is shown by the expansion and elevation of the waveform. The intensity of color correlates with the blare of trumpets, and the music visualizer erupts into an irregular shape that demonstrate the sharp cutting sound of this effect.
teammates
Where to now?
Check out some other digital concepts.
