An Idea
A few months ago Kim
Vehon, from our Worship team came, into our office to share a video showing a
bunch of phones hanging from wires flashing on and off. It's pretty cool. She wondered if we could do something like
that during our Christmas services at Central. Building a "fat-app" to control a
bunch of phones wired together did not seem like a challenge nor interesting
enough to be worth pursuing.
A Better Idea
But then we started thinking. What if we tried to
control the congregation's phones and what if we used only their existing 3G
network connection? And, since hardly
anyone would want to install an app from the market/store, what if it only used
their phone's web browser? We also thought it
should be built in such a way as to reduce the dependency on the network -- in
other words the phone should get what it needs from the server and then be able
to loose network connection without impact to the performance. Game on. I agree with Jason (tweet),
this might be one of the funnest things I've worked on in a long time. Jason and I call our software CrowdSync...
Ultimately we also needed to
generate a lights time coded "track" from a Christmas song (midi
file) with each note being assigned to 1-4 colors. A person's phone would receive the track, be
randomly assigned one of the four colors, and start playing the track in sync
with the band. Sounded simple
enough. Thankfully I found the C# MIDI
Toolkit code from Leslie Sanford that reads midi files (thank you Leslie!)
which I was able to modify in order to extract and generate our time coded
light track as JSON data which looks roughly like this:
{
startTime: 123578916,
endTime: 12345667,
ticks: [
{ notes: ["a"], time: 27, duration: 900 },
{ notes: ["b"], time: 982, duration: 900 },
{ notes: ["c"], time: 1940, duration: 900 },
{ notes: ["d"], time: 2908, duration: 900 },
...
]
}
The Time Problem
Pretty quickly
we eliminated web sockets and other client-server signaling technology for a
variety of reasons including lack of consistent device support, chattiness,
lag, and timing control. We wanted to
constrain ourselves in order to force us to think differently about certain
problems -- such as the "how do we get all
of these devices/clients to start at exactly the same time?"
problem.
Initially we were
thinking we could rely on the time from the phone's operating system. You might think two 3G Verizon phones would
have the same time, right? Wrong. Very
wrong. We saw phones that were off anywhere from several seconds to a few
minutes. (Who knows how or where each phone is
getting its time from? It doesn't appear they're using a common Network Time
Protocol (NTP) server.)
After some medium
scale client tests and experimentation, we came up with the following
approach: Each client asks our server
for the current time, calculates the delta (from it's local time), and repeats
this about 20 times over the course of
about 20 seconds. The error introduced
because of network latency is reduced to a minimum because we use only the
'smallest' delta from our samples.
Once we've got the
correct delta we calculate and shift the entire track's note times to represent
the exact localized time each particular note should play (light up) on that
device. Then we basically wait until the
note's time occurs and set some CSS to play that particular note's light.
Other Stuff
Since not all
browsers can handle HTML5, we had to keep things simple and used basic HTML,
CSS and JavaScript (CoffeeScript). Other problems worth mentioning are: cell
phone auto dimming, screen locking, and mysterious time lost as a result of
certain mobile phone browser interruptions - such as screen lock and wake up. (On my Android phone, a screen lock and
re-open would cause a simple JavaScript clock to become out of sync with
correct time. It smells like a timing
bug with the OS - but what do I know.)
Forward to November
when it was decided that our system should also power the worship center's IMAG
side screens with something cool, show an event count down timer, and the drive
the band's click track. That ended up
being a real blessing because we got to work on some really fun stuff. After more experimentation for the
"something cool" part, and more ramping-it-up ™ with statements like
"what if we could ...", we ended up programmatically building a
Christmas Tree as colored circles on the HTML5 canvas to match our church's
Luminous graphic.
We somewhat randomly plot circles in the shape of
a Christmas tree using a little Math.sin() trick to get the curve we
wanted. (Remember that stuff you learned
in high school -- it really does come in handy!) The same JSON encoded light track data would
be used to control the four sets of lights on the tree. We kept the same code base and ended up with
some configuration settings to control whether the simple HTML/CSS (cell
phones) or HTML5 canvas w/tree was being used (side screens).
Ultimately, the midi
file click track turned into a full blown amazing score and arrangement of
Carol of the Bells by Adrian
Darsee in mp3 format. There was more
fun getting the audio track to play which perhaps Jason will cover in his blog.
We also created a small admin panel as an Arena module so that the A/V tech guys at each campus could store the event start time for that service once they knew exactly when they wanted it to start.
Time Shifting Gotchas
Late in the game we
discovered something unfortunate when we put it all together with all
the other equipment (QLab, the worship
center's A/V systems and side screens, a full 30 minute pre-service countdown,
etc.). Not all seconds are created
equal. I should have realized this, but somehow it failed to register. Due to NTP being used on the various systems,
the QLab mac, the mac running Chrome for the IMAG, and our server would slowly
drift by a 1 to .5 seconds during the 40 minute countdown. That's a big problem if you're trying to
achieve millisecond synchronization. We
turned off NTP as a work-around but realize it's something we want to address
in future versions of CrowdSync. We're also going to implement all the server side stuff in Node.js.
Conclusion
The goal was for
each person to "be the light" and "let your light shine" as a literal
and symbolic expression during the worship to our King. Overall it was a
success for the whole opening worship experience but when the congregation
erupted into loud applause at the end I knew they
enjoyed the experience too. I hope we
have some of it captured on video.
We'll post the latest alpha CrowdSync source up on Github shortly.