I got an email from a Notary Sojac who saw my tutorial on playing a sound on hover. In that tutorial we had to resort to some fairly hackish behavior to get the sound to react nicely to our events. Notary Sojac had an update to make the existing code work better, but more importantly, was looking into the Web Audio API for doing stuff like this. I didn’t even know that was a thing. Turns out it’s not controlling HTML5 audio elements, it’s deeper level access than that.
The following is a guest post by Notary Sojac explaining all that.
We’re all impressed with the capabilities of HTML5 canvas. The idea of creating a standardized and logical way of performing 2d drawing operations opened the door to an amazing new world where one programmer could learn one language and develop amazing products that could be deployed to all relevant platforms! Think about how critical this is for a moment. Now think about who would want an amazing graphics application that has a buggy, glitchy sound system that only makes sounds moments after you clicked and sometimes forgets to make sounds all together? That’s the kind of product you get (especially in mobile platforms) if you go the HTML5 <audio>
element route. It’s better in desktop browsers, but unfortunately will never be perfected due to reasons beyond the scope of this article. But as of now, for the desktop, there is still hope. And there is hope in the future for mobile applications as well. That hope is named…
Web Audio API
The web audio API is designed for high precision and low level access. You can literally write bits of data to the samples. I’m not 100% sure what a sample is, but I think it has to do with air pressure against a microphone (aka, the microphone’s driver position) at a single moment in time. That’s pretty low level if you ask me. In Google Chrome, all audio functions are handled by a separate thread to ensure non-wonkyness.
On August 21st, Mozilla began making public the fact that they’re dropping their original, pioneering “Audio Data API” and are beginning to implement Google’s “Web Audio API” in their browser. But that means, at the time of this writing, you’ll need to use a fallback for Firefox.
You can hear a great deal of good information from this Google I/O talk by Chris Wilson:
How To Use It
Unfortunately you need to use a lot more JavaScript with the Web Audio API than you do with plain old HTML5 audio elements, but often times the precision of the Web Audio API is imperative.
First we need to load the sound, which can be done with a function like this:
// you'll put the PCM audio in here
var audioBuffer = null;
var context = new webkitAudioContext();
function loadDogSound(url, variableToBufferSound) {
var request = new XMLHttpRequest();
request.open('GET', url, true);
request.responseType = 'arraybuffer';
// Decode asynchronously
request.onload = function() {
context.decodeAudioData(request.response, function(buffer) {
variableToBufferSoundIn = buffer;
}, onError);
}
request.send();
}
Then we play the buffer, with a function like this:
function playSound(buffer) {
// creates a sound source
var source = context.createBufferSource();
// tell the source which sound to play
source.buffer = buffer;
// connect the source to the context's destination (the speakers)
source.connect(context.destination);
// play the source now
source.noteOn(0);
}
That’s a heck of a lot more code than we need as designers, just trying to play simple mouseover sound effects. Luckily, I snapped together a little JavaScript framework for using the Web Audio API.
With this new API, all one needs to do is:
- Create an AudioContext
- Load a sound into that AudioContext
- Play the sound
All of these steps are one liners! Fallbacks to good old HTML5 are included!
<script src="javascripts/webAudioApiForDesigners.js"></script>
<script>
// 1
var context = initializeNewWebAudioContext();
// 2
context.loadSound('audio/beep.ogg', 'beep');
$("#nav-one a")
.mouseenter(function() {
// 3
context.playSound('beep');
});
</script>
List of reasons why HTML5 audio elements may be a bad choice
- Wonky latency problems where sounds are heard at wrong times
- Unexplainable popping sounds when playing multiple sound files at once
- Limit to number of HTML5 audio elements that can play simultaneously
Reasons why HTML5 might be more suitable to your project
- Very simple and straightforward to use
- Can play background music just fine
- Eventually (wild guess: no sooner than within the next 3 years) the bugs will be ironed out of these sorts of html DOM elements
More Information
HTML5 Rocks has a solid introduction.
great stuff! nice framework, exactly what’s needed. with any luck, the jQuery guys might start integrating some of this stuff into a future release!
I’d like to see a version of this that works by assigning specific sounds to specific keyboard keys, so that I can make a simple drum machine/keyboard/synth.
Help would be greatly appreciated.
OMG you are right! We will be able to play any synthesized sounds, like organs, bass guitar, whatever, right from the keyboard. The possibilities are endless with MIDI. and the keyboard has so many keys it will be pretty cool. Someone will run with this and make a name for themselves.
@Louis
http://stuartmemo.com/qwerty-hancock/
@Norman
I`m afraid that`s really not the case. Controlling audio in real time is not possible without a significant delay. The flash audio api is around for a couple of years now with full MIDI support with the same issue. That`s why we still need low-latency(ASIO) sound cards.
@28inch understood, thanks for the added info, I did not know that.
@louis
http://www.schillmania.com/projects/soundmanager2/demo/mpc/
Happened to be browsing with Safari, clicked the demo button and got the fall back message (“Web Audio API is not supported on this browser. HTML 5 Audio Elements will be used instead.”), only there is no audio. Not sure why. In Firefox, which also uses the HTML 5 fall back, the audio is there.
I have firefox and I too got the message, and also heard the sounds and wondered how, especially after viewing source and saw no HTML 5 audio tags. Anyone know how this is possible?
Only the very latest Safari browser supports the Web Audio API. It seems as though Apple is withholding that software from users who haven’t bought the latest version of Apple. This makes it a lot harder for linux and windows developers to work on solutions for that platform, and more difficult for Apple users who don’t have time to keep track of where to buy the latest update from and whether or not their system is eligible. The good news is that the Web Audio API will be allowed on recent mobile platforms, so kudos to Apple there.
It seems mysterious (but true) that apple is only supporting the .mp3 format with their web browser. The decision only makes sense to me if they’re receiving payments FROM the mp3 patent holders rather than paying them out. FF will not behave in this manor when it’s ready (it’s still in the progress of adopting the technology). And obviously Chrome’s browser is designed to work with open standards like .ogg and webm.
I managed to get out to an apple store and look things over (my libraries only have PCs) and as I suspected, changing the code to load an mp3 instead of an ogg resulted in a success. It seems like a petty task for Apple to have me do so I’ll gripe about it with them for a while and then update the framework to support fallback formats in a clever and logical manor. Right now I’m switching my OS to Linux Mint so I have a lot of development environment stuff to set up and work needs me to do bacula maintenance.
This will solve the..
“Something suboptimal happened”
error if you do something like…
context.loadSound([‘audio/beep.mp3’], [‘audio/beep.ogg’], ‘beep’);
OR perhaps something like this too…
context[‘beep’].loadFallbackSound(‘audio/beep.mp3’)
But give me a day or two to code that.
And *I* got:
“Something suboptimal happened while attempting to decode some audioData.”
(in Safari 6.0)
A general reason why no audio elements should be used on websites at all: They are just and simply annoying! The very same is true for smart phones that sound on touch or cameras that do sound as preset. Ok, that’s a personal opinion, but if you use public transportation you know what is behind my statement…
Agreed.
When I hear most websites play a sound, I’ll instinctively ⌘W my way out of there.
Of course, sounds are still appropriate for games or videos/animations.
“XMLHttpRequest cannot load . – Cross origin requests are only supported for HTTP.”
is there a way to make it work on a local environement ?
are you using local with a WAMP server so it looks like :
http://localhost/etc/
cause technically thats http
Well I just tried to run it in MAMP, but I get this error :
“something suboptimal happened while attempting to decode some audioData.”
sdfsdfsdfsdfsdfsdf s sa sf à á f sà sa ssf
sdfdfdfsdf
sf
sdf
sdf
sf
sdf
I too got some strange messages in Safari when browsing to the demo. It works perfectly in Chrome though.
It looks like support for Web Audio API is shaky in Safari as it was just introduced with the recent 6.0 release. Maybe after a while support will be more reliable. http://www.html5audio.org/2012/07/safari-6-arrives-bringing-web-audio-api-to-apples-browser.html
If you really needed “something” with sound on hover that worked on a variety of platforms you’d probably have to set up some browser detection and have a Flash version available for oldies. Yikes.
So smoother that the old version. Nice work :)
It sounds like there are a lot of technical issues with getting sounds to play just when you want them to. Considering that, my suggestion may not be technically possible, but why doesn’t CSS cover sounds that should play on hover? CSS handles the presentation of the markup, but pretty much only visually. Why doesn’t CSS handle that kind of audio presentation? If you want a button to change colors on hover, you use CSS. Why is it that when you want a button to beep on hover, that’s not CSS? For that matter, why doesn’t CSS handle a sound you want to play on :target? What about at the very end of a CSS animation like a z-index animation?
Great. I’m going to have to give this a go.
I’ve seen sounds played on websites for many years now, (such as when hovering an element). I’m surprised by what few website owners actually do this since its so cool and unique. (goes beyond the “look” of the website and easy to do).
I should stop talking and get around to doing this myself!
Interesting ideas; and as you point out, the number of innovations coming into HTML5 / Canvas is amazing – but I really hope that this doesn’t lead to a proliferation of noisy websites, as one of my number one pet hates when browsing sites is ‘unsolicited noise’.
Any suggestions on how to make the sounds to stop playing when the mouse stops hovering over the element? The soundbites I’m using are 5 seconds long and it gets pretty crazy when the user accidentally hovers over multiple elements, lol.
I’ve tried using this code to no avail:
`webkitAudioContext.prototype.stopSound = function() {
if (source) {
source.noteOff(0);
}
}`
`$(“#button1”).mouseleave(function() {
context.stopSound(‘voice1’);
});`
`$(“#button2”).mouseleave(function() {
context.stopSound(‘voice2’);
});`
Any ideas?
Hey Mel, glad to see you’re thinking outside of the box on this new technology. When I initially laid out the framework, I had one thing in mind, play many sounds at once and let them overlap as needed (echoing peacefully off into the background before finally reaching /dev/null, their origin). This entailed rapidly spinning off new sounds without saving a reference to them in an array or anything, so in the main branch’s current form it wouldn’t have been so easy to program a stop() method.
Luckily the code is still fresh in my head, so I started a new branch on github where I’m testing the code out that allows for playing sounds single file, one at a time without any overlapping of the same sound.
It’s getting really late at night, but checkout the “stop-sound-effects” branch. The code and demo there should allow you to do .stopSound() just fine, and it’s rigged not allow sounds to overlap at all thanks to a new syntax being used under the hood.
initializeNewWebAudioContext({ “enableIE”:false, “oneSoundAtATime”:true})
Let me know if it works for what you had in mind well. It might be buggy still, it’s a bit passed bed time for me.
Cheers guys I’ll have a look at those links.
Very usefully! Thanks.