Behind the scenes

Explore Touch – A concept site to show how HTML5 and touch is moving the web forward

With the advent of HTML5, the web is becoming a more beautiful place. The HTML5 support in modern browsers like Internet Explorer 10 makes it possible to create rich and interactive experiences that work across tablets, PCs and phones. Now, with advanced touch capabilities in Internet Explorer 10, sites can also be made more intuitive and fun to use.

ExploreTouch.ie is a concept website that lets users use intuitive touch gestures to mix audio files to create an original sound track. The experience centers on an original sound track from Blake Lewis, the singer, songwriter and DJ. The site lets users use their fingers to mix the stem files of Blake’s song to create a killer music experience. The visual aspect of the site is also enhanced by a psychedelic explosion of colors that beats to the user’s every touch.

Check out the video above that goes behind the scenes and takes a look at Blake’s experience helping create the site and the awesome Internet Explorer ad that showcases the site.

The technology behind the site also deserves mention. The creative minds at Fantasy Interactive led the development effort with support from Clarity Consulting and the IE team. Below are some interesting stories behind what it took to build the site.

1. Touch – Expect more from the web

In the past few years, touch has revolutionized the way we interact with our devices. While much of this innovation was primarily for apps, with IE 10 and Windows 8, we can expect more from multi touch and gestures on the web.
For touch events specifically, Microsoft is leading the effort to create a standard for the web. In keeping with the idea of having a single markup for the web, this effort will ensure that developers can use the same markup for building rich web applications that work great with touch. This will also ensure developers will only need to write to one pointer input model no matter if their users are using mouse, pen, touch, or whatever comes next.
For the Explore Touch site, we went beyond the basics and used a combination of the pointer and gesture events to build a touch interaction module that supports gestures for swipe, tap and hold, scratch, pinch and rotate. While the explore touch experience can be enjoyed with just a mouse on IE 10, for the full multi touch experience we recommend trying it only on IE 10 on Windows 8 devices that support multi touch. Below are code samples for implementing the swipe, pinch, rotate gestures.

Touch controller - creating MSGesture object

Javascript

Embed Raw

Identifying global events

Javascript

Embed Raw

Implementing Swipe, pinch and rotate gestures

Javascript

Embed Raw

2. HTML5 – A Fast and Fluid web

Considering the site is pretty graphic intensive with 8 simultaneous audio streams playing in a synchronized manner, performance was an important criterion while developing the site. Below are examples of some of the audio-visual effects we use on the site and how HTML5 and hardware accelerated browsers like Internet Explorer help render these effects in a fast and fluid manner.

Fading transparent canvas

Glow and particles

For the glow and particle effects, the challenge was to fade the canvas content so it loses its opacity over time. This is usually done by drawing a semi-transparent rectangle in a background color over the canvas. However, in this case the canvas had to be transparent in order to allow another underlying canvas to be visible.

One way to solve for this would be by manipulating the canvas alpha values directly, looping through all pixels and diminishing the opacity of each pixel. However, this approach is extremely slow and can be only applied to small canvases.

Luckily, the API offers "global composite operation" mode. If you look at the comparison sheet, you’ll find an interesting and mysterious mode called ‘destination-out’. From W3C specifications it can be interpreted like this: Display the destination image wherever the destination image is opaque and the source image is transparent. Display transparency elsewhere.

So this little trick turns out that’s the perfect and quickest solution to the transparent canvas fading.

Example code:

Javascript

Embed Raw

Offscreen canvases as buffers (caching)

Drawing filled shapes on canvas is a complex operation, especially when radial gradient fills are used. That’s because every time a gradient is drawn, new color values have to be interpolated to create a smooth color spread between the gradient stops. This can be a very CPU intensive operation (even when only 2 gradient color stops are used) and should be avoided.

To optimize the performance, it’s recommended to create a buffer using an offscreen canvas. Instead of drawing and filling the shape with a gradient each time, the buffer can be quickly (by taking the advantage of hardware acceleration) copied onto the target canvas using drawImage() method. The example is presented here:

Javascript

Embed Raw

Depending on the number of colors used, it’s wise to check the memory and regularly detain and delete all buffers that are no longer required.

Additionally, the content of the buffer can also be transformed (scaled, rotated, translated) at the time of drawing, which allows better reuse of the already buffered shape.

SVG vs Canvas

Both Canvas and SVG are highly optimized in IE 10 and they have their strong and weak points so there is no clear winner between them. A more detailed write-up on how to choose between Canvas and SVG explains the trade-offs in depth.

SVG provides a great alternative to Canvas especially when it comes to drawing primitive and scalable shapes. It turned out to be a better choice for touchable rings, significantly boosting their performance.

As the rings are drawn using radial gradients, they could, in theory, use the offscreen canvas buffering technique described above. The difference is that these rings are also highly interactive with changing hover and color states. If Canvas was used, each frame of the hover animation would have to be pre-buffered for each color. Additionally, anytime when the saturation changes (ring goes away from the center), it would have to be reflected in each animation step as well. The buffering approach would therefore use far too much resources, negatively affecting performance.

SVG is highly optimized in IE10, so drawing a ring with a gradient fill is fast, even when changing the colors, opacity, and the gradient stops. Hover animations are quick and snappy.

Creating SVG elements and modifying their attributes on the fly is not difficult, it can be easily done with plain JavaScript.

Dancing lights

Generating real-time, fast, pink fluid-like particle light wouldn’t be possible using JavaScript. There were thoughts on using a pre-recorded video but then applying the alpha mask to it wouldn’t be fast enough. In the end it turned out the transparent PNG sprite-sheet based frame-by-frame animation is fast and reliable. Original animation from the VFX team had about 1500 frames and more than 600 MB weight – way too much for the browser and network to handle.

We hand-picked 100 of the animation frames and then used a custom Python script to create a large 10x10 sheet containing the entire sequence. Every frame in the sequence was also cropped and scaled down to reduce the size of the PNG file.

DOM, CSS and JavaScript were used to play the animation – the DOM element contains a DIV with the light sequence set up as background-image. The element is also scaled up a bit using the CSS transform property to compensate for smaller frame size in the sprite-sheet. Background-position offsets are updated every frame to deliver a smooth animation.

As the sprite-sheet needed to be preloaded while CSS doesn’t allow preloading background images, another trick was used – the sheet is fetched as a normal image and then the same URL is dynamically injected into CSS. Because it’s already cached by the browser, it’s also ready to use in the CSS.

Syncing audio

Another challenge was to play 8 simultaneous audio streams in a synchronized manner. Calling the play() method for all streams at the same time is not enough – browsers usually take some time to initialize an audio stream (allocating the memory, reading the files from the cache etc.) which can understandably impede the synchronized start of other streams.

We have discovered there’s a relatively simple way to sync the streams perfectly. The trick is to still start them at the same time, then immediately reset each track’s playhead (by setting currentTime to zero) and then pause them all. After a second or two, the playback can be resumed for all and they will play perfectly in sync.

Conclusion

The fact that we were able to create “Explore Touch” on the web is another proof point for how the web is moving forward. And as technologies like HTML5 and the advanced touch model available in Internet Explorer 10 on Windows 8 continue to push forward, the future for beautiful and intuitive HTML5 sites continues to get brighter. We hope this concept site inspires the web developer community to push the limits of what is possible on the web and consider HTML5 and multi touch for the sites they build.

DJ Blake

Brought to you by:

Fi
Clarity Consulting
Microsoft