An orgy of the senses: Merging the eye and the ear

A California polymath has electrified the music world with his images of classical music in visual form, capturing more than 165 million hits on his Internet postings in just a few years.  Only pop singers or weird videos do better. 

And the best is yet to come. Says Stephen Malinowski, the pianist cum computer programmer who raised the animation concept to new heights, “We have barely started learning” how to do this. 

For me, it was Stravinsky’s Rite of Spring that made me sit up and take notice. Watching Malinowski’s animation with a friend, suddenly I understood the intricacies of this eerie masterpiece. The high-pitched bassoon opening theme had floated by my ears for decades, triggering nothing more than a pleasant sensation. Now, watching it snake across the screen in full color, it finally meant something musically. 

One listener wrote after watching it that he “never liked Stravinsky… but this visualization is spell-binding.” 

What is happening here? Another fan says it best: “The structure, the orchestration, the harmony – it is all laid bare before our eyes. A truly revolutionary way of experiencing this art.” 

In his Internet postings, Malinowski has amassed an archive of magical creations that transfer the power of our ears to our eyes – in effect shifting from our sensual to our analytical capacities. This is a revolutionary development enabled by the connection of Malinowski’s right and left brain. He has bridged the arid world of computer programming with the richness of complex musical sounds. 

Says one fan, typical of much of the Internet commentary, “I'm a total beginner to classical music, and these animations are a brilliant way in.” 

And we can now take in Beethoven – or at least more of him -- effortlessly. For me, nothing quite matches Malinowski doing the second movement of Beethoven’s Ninth Symphony.

It is nothing less than an orgy of the senses. Friends who have watched it report tears in the eyes, goosebumps, heart palpitations, an urge to dance. As one commentator was moved to say, watching it  “was gorgeousness and gorgeosity made flesh. It was like a bird of the rarest spun heavenly metal. Or like silvery wine flowing in a spaceship.” 

Moi? I can’t stop whistling it. 

Malinowski is a mild-mannered Californian who devoted his life to music animation in his spare time while making a living as a computer programmer. Now retired at 64, he does it full-time, continually adding to his YouTube channel "www.youtube.com/user/smalin", a treasure trove of visualized masterpieces. 

As his career developed, he struggled to find his way, even sponging off his parents until his late 20s. He recalls in my interview (below) that he worked as a performer, rehearsal assistant, private teacher, composer, music coach, and music copyist, and did office work for Levi Strauss & Co. and Manpower for a few years, even working briefly as a dishwasher and a babysitter. He says he dreaded becoming an adult. 

His roots were in music, however, and he was intrigued first by Bach’s unaccompanied sonatas and partitas for violin. Gradually he progressed to transferring a score to a roll of paper, inscribing the movement of notes horizontally, leading to the present flowering of the idea. The breakthrough was MIDI technology. 

One of his most recent creations, the visualization of the Bach Prelude and Fugue No. 1 in C major from the Well-Tempered Clavier, BWV 846, demonstrated how his visual talents have evolved into ever more vivid and expressive forms.

One of Malinowski’s qualities is his generosity in sharing his techniques and explaining how he surmounted technical problems. Key information for specific animations can be viewed in his fascinating production notes here.

His aim is to spread his lifetime of knowledge and watch the concept grow. As he explains in our interview, he feels the animations community is on the cusp of a growth phase. “I expect that once music visualization tools are widely available, people will start using them. And, based on my own experience – namely, that this is really fun--  people will enjoy this. 

A true enthusiast and a generous communicator, Malinowski responded thoughtfully to my questions about how his passion developed and where it might be headed. 

Q. As you said recently, music animation is in its infancy. Where might all this lead? What is your vision for the future? 

A. I have lots of ideas for how to make animations better. I expect that tools like the ones I'm using ⎯ but much more powerful and expressive ⎯will be available to everyone. Imagine what it will be like when we are able to create animations that express how we feel about the music. Music engages us physically. It is natural to feel like dancing when we hear music. When I make music animations, I am showing how I want the world to dance along with the music. 

Q. Your animations have a mesmerizing visual beauty. Don’t they replace the music, in a way? 

A. Sometimes I watch my videos without sound (and without trying to remember what the music sounds like) and ask: how does this compare to the experience of listening to the music? Of course, it's not comparable. I'm guessing that music is inherently more powerful, more visceral, more compelling, than animation. But that's only a hunch, and even if it’s right, the silent visual display is still interesting ⎯ it gives me a sense of the narrative flow and drama of the music, and it reveals things that I might not otherwise notice. 

Q. What does vision bring that hearing doesn’t? 

A. Hearing is arguably a more sensual sense than vision. Vision is the more analytic sense ⎯ we're very adept at conceptualizing patterns in a visual scene. This means that the music-plus-animation combination has the potential for being more comprehensible than music alone. 

Q. How much better could it get? 

A. I don't know, but I do know that the animations I'm making now are more interesting to watch with the sound off than the ones I made a few years ago. I expect that in the future, more artistically inclined animation artists will work with a broader range of music, and do visualizations that are meaningful in very different ways. 

Q. So “infancy” is really the current state of the art? 

A. Absolutely. We've been learning how to make music for centuries, and we've barely started learning how to make music animations.

Q. Surely there are dozens of unexplored possibilities… 

A. Yes, and if I weren't a musician, the "product" I'd be most interested in would be music visualization exercise equipment, in which you run/dance/skip/climb through a musical score. But the most attractive application might be to let people create their own animations to express their own feelings to a piece of music. 

Q. Are you still working on a video game that teaches sight-reading? 

A. No. But if somebody wanted to work on this, I'd be happy to advise them.

Q. You have said that one prerequisite for animation is love of music. You are an accomplished pianist but in a broader sense how would you describe yourself musically? 

A. My personal tastes in music are actually very narrow. I explore different kinds of music in my visualizations because it's interesting and challenging and I learn a lot, but I think I do my best work on the music I'm most familiar with ⎯ which explains why more than half of my videos are Bach, Beethoven, Mozart and Chopin. And in my private musical life, I've probably spent more time playing Bach than all other composers combined. The instrument I play best is the piano; on that, I would call myself a talented amateur. For listening, though, I'd probably choose string quartets.

Q. What were your initial career objectives and how –and why –did they change as you moved on and on? 

A. My father was physicist who worked as an engineer, helping to design and build weather satellite cameras for a subsidiary of Hughes Aircraft; growing up, I vaguely expected that since I liked math and science, I would follow in his footsteps. However, I had learning

problems (that I didn't fully understand until I was in my fifties) that stopped me from pursuing that path, and I switched from math and science to music. I got a degree in music theory and composition, but I didn't have any idea how to make a living as a musician.

Q. So how did you start? 

A. I taught briefly at the University of California at Santa Barbara, where I had studied, but I discovered that I didn't want to teach in an institution. I worked as a performer, rehearsal assistant, private teacher, composer music coach, and music copyist, but never enough to fully support myself; I sponged off my parents and friends until I was nearly thirty. I also did office work for Levi Strauss & Co. and Manpower for a few years, and worked briefly as a dishwasher and a babysitter. 

Q. In your career, you took the time and energy to train as a computer programmer. Was it your plan to bridge music and technology? 

A. When I was in college, I took an "introduction to computer programming" course out of curiosity (and because it qualified as a "science elective"), and found that I was pretty good at it. I worked on my music animation project in my spare time until 2010, at which point I retired from my paid job and began to work on music visualization full time. But I wasn't imagining anything as ambitious as "bridging music and technology." I was just thinking that I'd make my static graphical scores better by animating them, and learned the tools that would allow me to do that. (See here.) 

Q. Your work seems to be evolving, with more interesting, vivid and expressive graphic ideas as in the Beethoven Grosse Fugue and your new version of the Bach organ Toccata and Fugue. Is continual creative improvement a never-ending process for you? 

A. Yes, I guess, but it's not because I'm focused on improving; I'm

just trying to do good work, and the more time I spend, the better I get, and the higher my standards are. What's never-ending for me is curiosity. Or, at least, it's what drives me, and it hasn't ended yet.

In my first graphical scores, I was only trying to show what was objectively in a piece of music ⎯ the timing and pitches of notes ⎯ and for years that's all I did. But the work of other animators, most notably Oskar Fischinger, showed me that there were ways to express the subjective aspects of music (what it feels like), and in 2010, with my first moving-ball animation... I started experimenting with that.

Q. What is your most recent animation? 

A. The most recent major thing I have completed is the Beethoven string quartet, opus 130 (and the Grosse Fuge, which was originally part of that). I'm currently working on Bach's Art of Fugue, using recordings by Kimiko Ishizaka. Here is what I did with the some of Beethoven’s quartets:

 I'm currently working on Bach's Art of Fugue, using recordings by Kimiko Ishizaka

Q. What is your favorite?

 A. Different animations please me in different ways. I'm not a big fan of Mussorgsky's music, but I'm happy with how my video of Night on Bald Mountain turned out.

Also, the way the motion and dynamics work in the third movement of Vivaldi's Winter (from The Four Seasons) was very gratifying.

But mostly, I'd say that my memory isn't good enough for me to have favorites that last more than a few days. 

Q. You have produced so many animations that you must get a feeling of overload from time to time. 

A. Yes, when I go back and look at videos I've made several months ago, I often have a "wow" reaction ⎯ I've mostly forgotten what I did, and it's almost like seeing it for the first time. 

Q. You are a fine pianist but how did you manage to master all the Chopin études, Op 10 and 25? 

A. During 2015-16, I did videos of all the Chopin etudes. Most of these were my own renditions  (using the conductor program, q.v. http://www.musanim.com/tapper/), but for a few of the harder ones that I couldn't do justice to myself, I used British pianist Paul Barton's recordings. I'm pretty satisfied with these for now.

Q. You have said the back burner is a holding place for your idea of a new animation for the complete Bach Well-Tempered Clavier books. Where does this project stand? What else is in the works? 

A. In 2016 I did animations of all the preludes and fugues in the first book of the WTC based on recordings by Ishizaka, as I said earlier. I will probably do the same when she releases her recordings of the second book. As I mentioned, I’m currently working on the Art of Fugue based on her recordings. 

Q. You have said untrained music-lovers miss a lot by “not noticing stuff”. Is one of your objectives to fill in the gaps and help the layman follow complex music? 

A. That's one of the things my animations are good for, but I wouldn't say that it's an objective -- it's more like a side effect. I'm trying to make scores that are pleasing to watch, and for me, this comes from them doing what the music is doing in ways that bring out what's in the music (as opposed to adding something separate). 

Q. You seem to have created a new subculture of classical music appreciation. Reaction to your YouTube posts is mostly ecstatic. Is that the outcome you foresaw? Did you hope for more than mere adulation? 

A. I was not seeking adulation. When I first made graphical scores and showed them to my friends and fellow music students, I had no hopes other than that people would think they were cool. When I started making animations, I expected people who saw them would steal the idea, and I'd be able to watch their animations. It's taken more than thirty years, but that's finally starting to happen. 

Q. But now you seem to be on fire. 

A. Well, I didn't anticipate YouTube, so I didn't expect that when this sort of thing became known by millions of people, it would be because they were watching my videos. Having my work getting better known means I get to collaborate with more talented composers and performers, and that's great. But I'm a private person, and the most satisfying thing about doing what I'm doing is being able to watch my own animations; second-most is knowing that I'm showing people the potential of this sort of thing. What would really thrill me is if it became popular to the extent that my work was eclipsed. I hope I live to see that.

Q. People who have been hostile to classical music speak of a near-religious conversion after viewing some of your work. Casual public acceptance has been extraordinary. YouTube access has exceeded 165 million. How do you account for this burst of enthusiasm? 

A. Most classical music is more complex than most popular music, and appreciation requires understanding. Popular can't present challenges to understanding that would stymie a majority of

listeners. So my animations help in a couple of ways: they reveal some of what's going on in a piece of music (which helps you understand it), and they are visually engaging (which helps you keep your attention focused). This combination can serve as "training wheels" that help a beginner hear like a more-experienced classical music listener. 

Q. Some of my pianist friends who have viewed your animations are drawn to them but mainly as curiosities. They say they already know what the animations convey. Is your work therefore mainly for newcomers to classical music or listeners who have tended to take in Mozart, say, without real understanding? 

A. Reactions to my animations are all across the spectrum, and seem to be based more on the perceptual apparatus of the viewer than on their musical background. Experienced musicians who are visually oriented tend to appreciate them, and non-musicians who are less visually oriented tend not to. What I'm doing is very narrow, very specialized: my focus has been on showing structural aspects of classical music, and it's not surprising that it's not everyone's cup of tea. 

Q. Some people believe your work could be the savior of our great musical heritage by enlarging the base of listeners. Does that please you? Worry you? Frighten you? 

A. Frighten me?  Don’t be silly. I hope it helps people appreciate music more. I have no idea how great its effect will be.

Q. Are Asians (Chinese, Japanese, Koreans) more receptive to the educational potential of animations as they strive to understand the Western classical harmonies and structures? 

A. These days, people in Asia are exposed to Western classical music about as much as children in Western countries are, so I don't think there are significant differences in how they relate to my animations based on that. 

(MJ: The press is starting to take this on board, as in this recent article from London.) 

Q. You have said that complex orchestral scores were beyond your own comprehension when you were studying composition at university level. Might your work help people hear music by looking at a score? 

A. Sure, they can help. Lots of music students have said that they learned such things by watching my videos. 

Q. The meaning of major compositions often eludes even the most authoritative musicologists. Debates on the big compositions have spawned learned essays and books. How can your animations help us understand better something like, say, the Liszt B Minor Sonata or Beethoven’s Fifth? 

A. Good question, but maybe impossible to answer. What is the "real meaning" of a piece of music? I don't have a good answer to that. What I have noticed, and where I do think my animations can help, is in this: there are features of music that you might not notice, and which might, if noticed, result in you hearing a piece differently. 

Q. Bruce Lamott of the San Francisco Conservatory has written a ringing endorsement of your work. Is advanced music education generally taking an interest in your work? 

A. Some educators are using my videos but I don't know about "generally" ⎯ I don't have a good way of finding out.

Q. Now you have stopped working as a paid programmer and are devoting yourself full-time to animation. Do you see commercial prospects for your animations? Hasn’t this been slow to happen? 

A.  I've done next to nothing to try to commercialize my work, so I guess you could say that it's my own fault that it's been slow to happen. I've done some paid work for various clients but I haven't tried to promote myself. There was a time when I considered the possibility of earning a living from music visualization, but I decided against it ⎯ it didn't seem certain to be lucrative, and I'd had enough exposure to the business side of software development that I wasn't attracted to taking on more of that kind of uncertainty, stress and responsibility. 

Q. Anyway, your specialty doesn’t seem to correspond to traditional categories. 

A. Yes, it's rare for an artist to make a living from their art alone. And a practitioner in a new art form that doesn't fit into any existing category has to just do his work on his own penny (with rare exceptions). Anyway, it's not even clear to me what the “commercial side" of music visualization would be. Videos/movies? Live performance? 

Q. Are others out there looking for opportunities to use or integrate your developments? 

A. For this, the Music:Eyes project might lead the pack. In any case, whether it's been "slow to happen" depends on what you compare it to. How long did it take for other art forms to develop? Some happened overnight, but some took

millennia. I expect that once music visualization tools are widely available, people will start using them. And, based on my own experience (namely: this is really fun), people will enjoy this. 

Q. How do you make a living? 

A. These days, I don't. You might say I'm "retired," since I don't work for anybody else and I don't make enough money to support myself.

I'm 64 years old, and I'm hoping that my savings (and Social Security, when I start drawing on it) will let me do music visualization full-time until I lose my faculties.