top of page
  • Writer's pictureandreahywong

Project: Interactive Visual Audio Novels

Updated: Nov 30, 2020

This project is something I’ve actually wanted to do for a very long time. It came from my desire to learn how to use an audio middleware through some application. It didn’t see like a near-future opportunity at work yet, and so I decided a personal project would be the way to approach it. I previously thought of scoring for films and short videos to practice writing music to a certain mood and encapsulate a specific feeling (although, this heavily abandoned Ikoria project would be representative of my progress in that :/). However, films are watched and experienced in a linear fashion, which is a little different than some more interactive mediums like video games and other similar interactive experiences. I’m no game developer, and I’ve thought of finding some game templates to add audio to as a project to learn and practicing using audio middleware, which is definitely still a #futureproject I’d like to pursue… but wanted to do something in between a fully working interactive video game and a non-interactive film/video – an interactive experience, but simpler.

I wouldn’t say I grew up as the hugest comic book fan, but I certainly read them, and enjoyed escaping into their worlds. It’s been a few years now, but there is one graphic novel that I’ve been keeping up with and still continue to: Monstress written by Marjorie Liu and illustrated by Sana Takeda. I wouldn’t save a read as much fiction as I did as growing up, and that was definitely already less than what I wanted. I enjoyed reading books with magic and adventure that tested the limits of what seemed possible, even if only in the imagination; fantasy and science fiction were great go-to’s. I’d say Monstress is what I can delve into, read, enjoy, and escape with in my current adult life. Off the top of my head:

  • The plot is action-packed and densely layered.

  • The world is vast. It’s magical and so imaginative.

  • The characters are complex, dynamic, and diverse in so many aspects. There are many strong female characters. In fact, majority of the characters are female.

  • There are so many other aspects but I wanted to lastly mention: the art.

Sana Takeda does the most phenomenal job bringing this story to life. I have to say it was flipping through the pages and seeing her meticulous gorgeous work that made me commit to purchasing and reading the first volume. The detail and style in this steam-punk aesthetic is remarkable. The costume design, the architecture, the gadgets… I’m just going to have to put images here for you to drool over for yourself. My chunky thoughts and blog-writing skills simply do not do it justice.

So if it’s not obvious enough, I’m completely in love with this graphic novel.

I’ve decided to finally jump into this project idea of mine: an interactive visual-audio novel (an IVAN?). I’ve always struggled with listening to music while reading, because the overall mood of the music might not mirror the situation I’m supposed to visualize and feel in what I’m reading. For example, if I’m reading a high-intensity action-packed chase scene in a book, listening to some Chopin Nocturnes feels very wrong and disorienting. It’s hard for me to enjoy both things at the same time. I very often get distracted by one, leaving the other not appreciated and thus, a bit unnecessary for that moment. But I want both? But matching! And no Spotify playlist is going to be able to read my mind that is reading my book!

I had thought about making an #audiobook. Those are rather popular today, but the one time I tried listening to that, I think I fell asleep. It’s not that they can’t be interesting – you can definitely add music, foley, and sound effects to narration… even having different voices for different characters. Heck, that’s how stories were told on the radio before movies no? It can definitely be entertaining, and it’s actually another project option I’ve had on the back of mind for trying out! But it’s not interactive, and that’s what I currently want to create – something interactive to learn and apply the use of an audio middleware (likely FMOD, and hopefully Wwise in the future).

That’s where the idea of having an IVAN came from. If sound was linked to what and where I was in my story, I could read at whatever pace I wanted, read forwards, go back and review a section, come back to where I was… And what better of a medium than a novel with graphics to apply this to? I have a bit of an idea how to implement this and what this would look like. I’ve seen some graphic novel-videos where essentially the graphic novel panels are presented in the form of a video so that the reader doesn’t have to flip the pages of the book. I didn’t love this format as it similar isn’t as interactive (besides pausing the video to finish reading I suppose).

I’m wanting to create a simple experience so the reader can be more immersed in the story and experience rather than the logistics of the interactivity. I was thinking of a simple scrollable experience. The reader simply scrolls through the story like one would flip through the pages of a book – at your own pace. I can’t wait to dive in and see how I can trigger changes in the audio. I’ve decided to make a living document to log my work and progress on this project and save the blogging more for my thoughts and reflection throughout the learning process! I’m excited! I guess in a way, it’s a fan project ><” Let’s see where this goes!



bottom of page