Jeremy is a co-founder & lead developer of Symphonia – an iPhone app that enables musicians to create and share music with their friends. In a recent conversation, I had with him, he shares invaluable insights into how he developed this amazing product from the bare bones.

So, without further ado. Let’s get on with it.


Tell Us A Bit About Yourself?

Well, I have been programming since the age of 12. I also, did one year of engineering school, dropped-off as it wasn’t challenging enough for me. Later I went to study mathematics at the university and finished with a PhD in algebra applied to computer science. Four months later, I co-founded Symphonia.


Tell Us About Your Product? What Pain Points Are You Trying To Solve?

The key idea of our product is – You sing a melody in your microphone and the app converts it to music notes, in real-time.

It is very common for a musician to have an idea while he is not in his studio. Having a simple way to record the melody and experiment with it without being in your studio really makes a difference. But we didn’t want to keep this tool accessible only to professionals, so we really made our best to allow anyone to use it and create music with it. Even if you’ve never played music, you can create something that sounds good in a few minutes. I don’t think there should be a clear delimitation between professionals, amateurs and neophytes. We all were born neophytes.

In the app, the user can select numerous music instruments that can replay his melodic idea, and also various tools to allow easy editing of notes, transposition, and creation of chords. But your song is not finished until you share it with your friends. We just released a feature that allows you to share your whole project to a friend. You just send a link, and when he opens it on his phone the app will start with a copy of your project. This really creates a room for collaborative composing through social media and instant messengers.


What Inspired You To Write This Product? How Did You Come Up With The Idea?

My co-founder has grown up in a music environment, and have many musician friends who really loved this concept. And I am a programmer so we joined forces.


Would You Like To Talk About Your Customer Acquisition Strategy & The User Engagement & Retention Rate? 

We released our product two weeks ago. I think it is too early for now. We also, don’t have any revenue yet as we haven’t implemented the monetization system.


Cool, Let’s Talk About The Technology Stack You’ve Leveraged To Build Your Product

Back in early 2018 when we started prototyping I used Kotlin to test the viability of an algorithm running in real-time on an android phone. At this point, it was not clear if a phone could handle this type of computation and if one could build an algorithm that does give a quality result.

This prototype had nothing more than an app turning microphone input into debugging graphs, but it was enough for me to know that this idea could indeed be turned into a real product.

I then rewrote the algorithm completely in C alongside a Python binding (with CPython). Python allowed us to make numerous tests when it came to improving our algorithm. We also made a small Flask web service to test the algorithm on wav files.

Once we got the algorithm working we integrated it with Swift for iOS (you can easily bundle a C library into a Swift framework) and started to build a “shell” around the feature. Integrating all the features you see today took a huge amount of time, and the choice of making our own unique design didn’t make the journey easier. But the result is worth the effort.

Regarding the audio handling in the app, we started with the AudioKit Swift library only to discover that it didn’t fit our usage and we had to rewrite part of our code using directly AVFondation to have an application that is stable. We now use Audiokit only for individual music instruments (samplers) and sequencing which instrument should play at which time (sequencer). For everything else use directly Apple’s AVFondation.

Recently, we added the possibility to share your project through a link. What happens behind the scene is that the app will POST your project file through HTTP to a Sinatra Ruby backend. You will then get a deep link (also called the universal link) that when clicked will directly open the app on your phone and download this project.

All our internet-based services, including our site web and our ruby backend, are running on the Google Cloud Platform.

To sum up, until now our codebase contains Kotlin, Python, Swift, C and Ruby.


Would You Be Willing To Talk A Bit About Your System Design?

All the algorithms run in your phone, it is readily embedded. The amount of computation that this little object has today is just crazy. Do you know an iPhone has more than 100,000 times the processing power of the Apollo 11 guidance computer?

We could have implemented a web service that would do the computation (indeed we wrote a service like that) and send audio from the app to the server, but we won’t be able to have real-time because of latency.

Furthermore, you lose the ability to record your voice when you are on the subway or in some part of the countryside etc. This is why we made the choice to do the computation directly on your phone. We may have more constraints on how we implement it, but we can provide an awesome experience that lasts as long as you have the battery in your device. 🙂

I include here a small diagram of the main blocks inside the iOS mobile app. It is a little oversimplified but it helps to get a grasp of how the code is segmented. I put a single block for all the UI for simplification, but keep in mind that more than half of the application’s code is made of Views and Controllers. Since this part is common to all mobile app, I did not give more details.


Symphonia app architecture - system design

Our app targets iPhones, but we also run on iPads. We think it would be possible to targets even smaller devices like Apple Watch, but this would require some R&D to adapt our algorithms to more constrained environment. It would also require a complete rethinking of the user interface.


What Are Your Scaling Strategies?

For now, on the backend, we only have to handle the user’s sharing of music. We use GCP’s Google Run with Docker to run our servers and Google storage for storing the shared files. Scaling from this won’t be a problem as storage is cheap and Google run provides automated horizontal scaling. This will also probably do its job pretty well as we add more similar features.


Speaking of the product design process & I have a few questions regarding this:

What Are Some Of The Architectural, Code Design & Picking The Fitting Technology For Your Use Case Decisions You & Your Team Took During The Initial Design Phase That You Are Pretty Satisfied With?

I would say we are pretty satisfied with how we isolated the actual musical project data using a `DTO` pattern and the`Codable` serialization protocol provided by Swift

The whole musical project constitutes of numerous classes composed together. Each of them has an associated DTO structure into/from it can convert. Each of this class also has a list of delegates (instead of a single one) that are called when its internal data is modified. This constitutes a nice “signalling” mechanism that tells the UI when a specific part of the project (for example the instrument on the 3rd track) has been modified. You can see it as a simplified analogue to the store-view interactions in Flux.

The second point I would mention is the fact that the musical instruments feature is completely independent of the user interface. This makes developing and improving the app way easier. For example when we introduced the sound of a note which is triggered when you move it on the editor’s grid. This allowed the user to hear how the note sounds like without the need of playing the song. Writing loosely coupled components is a little detail we focussed on that completely changed the experience of our users.


If You Were To Rebuild Your Service From The Bare Bones, Is There Anything You Would Do Differently?

I think any change to the actual design would result in increased cost on other areas, so I would say I am quite satisfied with our design choices up to today. But if I had to redo one thing differently, it would be to be more strict on the first code review with our first trainee. I merged some code I shouldn’t have, and I faced some issues due to that months later.


Are There Any Invaluable Learnings During The Whole Product Development Process That You Would Like To Share With The Readers? 

We had to remove and postpone many features we had imagined. We made the choice to only release things that were working well and it provided a fluent experience to the users. It is always a bit hard to let go of a feature on which you worked and is expected to be in the first release, but the overall experience of the user should be first. We also learned that handling audio on iPhone can be tricky.


Is There Any Advice That You Have For The Devs & Entrepreneurs Who Aspire To Or Are Already Building Their Product From The Ground Up?

I don’t have much to say than what would sound common sense, but it’s always good to have words on the ideas :

  • Focus on the core experience/feature of your product.
  • Always think and feel the product from the point of view of the user. It should be instinctive, easy to use. Most people (and I am guilty too) never read the instruction. Your product needs to be usable right out of the box. Users will skip the onboarding, so make it small and non-mandatory.
  • It has to look nice. We are in 2020, and when it comes to apps, users judge the book by its cover. If your app looks awful, users will think the features will be awful too and won’t even try it. You don’t need an overly complex or expensive design but simply make sure that your app looks clean and professional.
  • Ask your target users to test your app and use their feedback wisely. Having feedback early from a musician helped us fixed many flaws in the UX and made us focus on the key features of the app.
  • Look at numbers (Download and retention) and iterate over it.


Where Can We Learn More About You?

You can know a lot about myself from my personal blog, and you can reach me on twitter @jeremycochoy, Instagram @jeremycochoy and LinkedIn. I will try to answer everybody.

Here is the link to the app on the Apple app store.

Jeremy, thank you for this insightful conversation & we wish you great success as an entrepreneur.

If you guys are working on something cool or have already developed something & started up and would want to share your story. Check out this page.

Also, before you leave you might want to check out this Product Development Roadmap – Insights, By A Developer, Into The Process Of Developing New Products From The Bare Bones article that I recently published.

I am Shivang, the founder of this blog. You can read more about me here.

If you liked the interview, do share it with your folks. Consider following 8bitmen on Twitter,     Facebook,          LinkedIn to stay notified of the new stories published.

Here is my LinkedIn profile in case you want to say Hi!! 🙂

Cheers!!

More On The Blog

Web Application Architecture & Software Architecture 101 Course

How Liyas Developed An Open Source API Testing Tool With Over 16K Stars On GitHub

YouTube Database – How Does It Store So Many Videos Without Running Out Of Storage Space?

Instagram Architecture – How Does It Store & Search Billions of Images

What Database Does Facebook Use? – A Deep Dive