SoundManager 2: Technical Notes

Requirements + Specifications

What SM2 needs, and how it works.

Serving to HTML5 + Flash Clients

A few notes on HTTP response headers, configuration and so forth.

Tips for serving audio to HTML5 + Flash Clients

HTTP response headers from your server are important. Below is a brief list of recommended practices for serving audio content to clients using HTML5, and/or Flash.

HTML5

• Likes Content-Length HTTP response headers (can affect duration and playback events if missing.)
• Arbitrary seeking, dynamic loading/buffering works when the server supports Byte Serving via byte ranges, also known as HTTP partials / range requests. Without this support, behaviour falls back to a Flash-like single-connection, sequential/progressive-style download.
• Likes a proper MIME type response from the server, e.g., audio/mpeg for MP3 content.

Flash

• Likes Content-Length HTTP response headers. When missing, duration may be unknown and certain events like whileloading() / whileplaying() may not fire.
• Does not care about MIME type or partials in server responses.

In summary...

Always serve a proper Content-Length HTTP response header. For HTML5, review that HTTP partials / range requests are enabled and that you are serving the correct MIME type in your response as well.

Never apply gzip or mod_deflate compression to binary assets. It causes playback problems, costs CPU and in some cases, may even increase the transfer size.

How Clients Download Audio

Progressive, sequential HTTP vs. HTTP 206 partials / range requests.

How HTML5 + Flash Clients Download Audio

In terms of HTTP, assets are typically requested and downloaded in sequential byte order over a single connection. Flash typically works this way, but HTML5 clients can use behaviours that are closer to streaming - instead, using multiple requests for pieces of data.

Browsers can request audio from servers either sequentially or using Byte Serving (AKA "partials"), depending on client and server capabilities. The key difference is that with byte serving, clients request data in a "streaming" fashion and can buffer at will, therefore allowing arbitrary seeking, pausing and requesting of audio data in pieces - similar to how video playback typically works on YouTube, for example. Thus, preloading and onload() is less-meaningful when byte serving is involved.

Traditional, "progressive" download (single request)

The typical HTTP file download can be described as follows:

• Single HTTP request, one TCP connection
• All bytes are sent sequentially "over the wire"
• Network connection is closed when the download completes (or fails)
• SoundManager 2 fires relevant sound onload() callback when the connection closes and the sound is deemed to be valid

In terms of HTTP traffic, the sequence is something like this (excluding some headers for brevity):

• Client request
  • GET some.mp3 HTTP/1.1
• Server response
  • HTTP/1.1 200 OK
  • Content-Length: 158958
  • Connection: close
  • Content-type: audio/mpeg

Progressive downloading behaviour in SoundManager 2

This method is typically used by Flash when requesting audio via standard HTTP, and HTML5 clients in the event that Byte Serving (HTTP 206/Partial Content) is not implemented or cannot be negotiated.

• Playback begins after a small amount of buffering, once the download has started.
• Once started, the download progress continues until all bytes are received. Stopping or pausing playback does not cancel the download.
• During load, a SM2 sound object's whileloading() event will fire at a regular interval with bytesLoaded, bytesTotal and duration properties updating as the file progresses. Because Flash only reflects the "amount of duration loaded", durationEstimate is provided as a means of reflecting the total duration before load has completed (at which point duration is 100% and accurate.)
• During load, the user can only seek within the amount of data (duration) downloaded.
• onload() fires when the connection is closed, and all bytes have been received.

Byte Serving (partials / byte range requests, "streaming"-style delivery)

HTML5 clients will send a Accept-ranges: bytes header in the HTTP request for an audio asset to indicate their capability for Byte Serving, along with a Range header indicating what piece of the file to download.

If the server supports partials, it will reply with a HTTP/1.1 206 Partial Content header and a Content-Range header indicating the bytes it is going to send.

It appears that servers will return the whole range in the first response unless interrupted (and when a client requests a range of "0-" as in this case), but the initial connection may be dropped by the client if it wishes to stop "buffering" at any point, or if the user tries to seek to a new position in the audio that has not yet buffered.

• Client request
  • GET some.mp3 HTTP/1.1
  • Range: bytes=0-
• Server response
  • HTTP/1.1 206 Partial Content
  • Accept-ranges: bytes
  • Content-length: 4237566
  • Content-Range: bytes 0-4237565/7237566
• The first response includes all bytes in this case, but a client may drop this connection and make a new range request, if needed; for example, the user may seek to a new position in the file where data is not yet available, and the client will make a new request to buffer data beginning at the appropriate offset.

HTTP/1.1 206 Partial Content request/response example

• Client request
  • GET some.ogg HTTP/1.1
  • Range: bytes=5210604-5275910
  • If-Range: "508107-4d8a0b4e90d26"
• Server response
  • HTTP/1.1 206 Partial Content
  • Accept-Ranges: bytes
  • Content-length: 65307
  • Content-Range: bytes 5210604-5275910/5275911
  • Content-Type: audio/ogg

Byte Serving behaviour in SoundManager 2

Byte Serving is automatically negotiated between client and server, and offers a number of advantages over traditional downloads. Most notably, Byte Serving is closer to a "streaming" technology and enables clients to seek, buffer and resume playback at arbitrary positions within a file once the duration is known. Larger files benefit most from this technique, since they can be handled in smaller chunks vs. being held entirely in memory a la Flash.

While SoundManager 2 uses the same events for HTML5 and Flash regardless of transport mechanism, there are some notable differences in SoundManager 2's behaviour when using HTML5 and Byte Serving.

Most importantly, the concept of "loaded" (i.e., preloading data and waiting for onload()) with partials is irrelevant due to the way clients are able to pause, cancel and resume requests.

• Playback begins after a small amount of buffering, once the download has started.
• The HTML5-native canplay() event fires when playback is ready to begin. This will cause SoundManager 2 to fire a sound's onload() callback. At this point, the sound's duration should be known, but the SM2-provided bytesLoaded and bytesTotal sound properties may not be known and thus are 0/0 during load time, and 1/1 if still undefined at load time.
• SoundManager 2 attempts to fire whileloading() events as the HTML5-native progress() event fires, but keep in mind that any "bytes loaded" data may refer to blocks (i.e., buffered sequences of data) that are non-sequential and thus bytesLoaded and bytesTotal should not be relied on as an indicator of "total load progress".
• HTML5 clients may use their own heuristic to determine how much to buffer, if and when to pause, cancel or resume requesting data.
• Clients may request metadata from the end of a file in some cases, as with OGG formats, in order to determine information like the total duration of the sound.
• During load (and once the duration is known), the user can seek to any position within the file.
• A client may drop an open request in order to request a new byte range - for example, to buffer and resume playback if the user jumps to half-way through the file where that data has not been downloaded yet.

Mobile Device Limitations

Known restrictions, quirks and annoyances relating to HTML5 + mobile Webkit (iOS / Android) software.

Mobile Device Restrictions, Quirks and Issues

Mobile devices tend to be somewhat limited in terms of battery life, network connection and other resources. Furthermore, they are unlike larger (laptop/desktop) devices in that audio is often not a "shareable" resource, and only one application may be using the sound hardware at any given time. Thus, mobile devices must be treated slightly differently in terms of playing audio via JavaScript.

iOS (mobile Safari/Webkit)

• Sound play() calls are blocked by the OS unless in direct response to a user event like touch or click. ("Auto-play" attempts will be blocked, and the sound may fire a suspend event in this case.) For the curious, setTimeout() calls will also result in playback being blocked.
• Sound load() and load-related methods may also have some limitations, similar to play().
• Only one sound may be actively playing at a time. If an other sound (or application) is playing audio, it will be stopped by the OS in order to play the new sound.
• Chained playback (sequential / playlist-style behaviour) works when using the onfinish event handler. Otherwise, blocking occurs.

Possible Alternatives

The Webkit Audio API (possibly a future standard as the Web Audio API) allows for low-level access, manipulation and control of audio from JavaScript. This API is mostly separate from the HTML5 audio API, but will ultimately allow for better fine-grained control and mixing of sound.

Once playback is allowed, the Webkit Audio API should be able to get around many of the current HTML5 audio limitations present on iOS including multiple sound playback, volume and pan control, and dynamic filtering / processing effects. However, the API is not a standard and is not consistently supported in Webkit, let alone other browsers.

SM2 does not presently use the Webkit Audio API. It may be experimentally or more formally added at some point in the future, when the API is more universally-supported.

Debug + Console Output

Console-style messaging, useful for troubleshooting start-up and runtime issues.

Live Debug Output

With debug mode enabled via soundManager.setup({debugMode: true}), SM2 can write helpful troubleshooting information to JavaScript console.log()-style interfaces. Additionally, output can be written to an optional DIV element with the ID of soundmanager-debug.

soundManager.setup({consoleOnly: true}) can be applied to disable HTML output (using console.log()-only methods) as well.

Additionally, debugging within the Flash portion of SM2 is also available and set using soundManager.setup({debugFlash: true}). Debug messages are written to the flash movie itself.

For info on SoundManager 2 loading/initialization failures and how to fix them, see troubleshooting.

Below is a live example of debug output from SM2:

Flash debug output, as applicable: