They use data from the EchoNest platform, which I’d never heard of before. They have data for music tracks that you can download as JSON files. This data includes beats, bars, tempo and notes for each track – it’s a dense file, see for yourself.
Paul was responsible for pulling out that data and making sense of it, while Mr doob sorted all of the 3D rendering.
Paul Lamere demos Music Machine in Cannes. Photo by Eliot Van Buskirk.
Considering the note information is computationally extracted out of audio tracks, it’s astonishingly accurate (although the musicians amongst you may notice certain discrepancies in the notes!).
But the great thing is that, unlike the original Animusic, which must have been a massively labour-intensive project, this will work on any music track that EchoNest have in their database. Here’s the demo running with Herbie Hancock’s classic funk track Chameleon.