What a fantastic idea this is.
They need to have the input audio be a solo piano piece to see what happens. First some Mozart, then some Debussy. Now THAT would be cool.
I’m curious how much the transformation considers the audio spectrum of the output device. What if this was instead driving a sampler? You’d expect a different midi result depending on the samples chosen.
Generating the audio acoustically is a big part of the neat factor though. It reminds me of the coolest thing I ever saw a SIGGRAPH, the Wooden Mirror. It’s literally a low resolution monochromatic display device of wooden pixels. Each square pixel has a servo attached and it would tilt up or down to reflect the right about of light.
Pretty cool stuff.