Martin Schwerdtfeger and Tim Flohrer of zplane

zplane is a provider of audio processing and music analysis technology operating out of Berlin. Studio One takes advantage of several powerful zplane technologies—our new Chord Detection feature takes advantage of zplane’s KORT, and our Harmonic Editing features leverage a combination of KORT and reTune. Furthermore, their élastiquePro time stretch has been in Studio One since the very beginning!

We were able to get some time with Tim Flohrer, CTO and founder, and ask him a few questions about zplane.

How and when did zplane come to be?

We started off as a three-man company in 2000 directly from University. Martin, Alexander and me, we all studied electronic engineering but we were always into music. Our start was kind of naive to say the least: our business plan was basically ‘let’s have a company that offers development services for something with computer and music’. So, as you might imagine, it took us a while to find our business model. Back in 2003 we had a customer who needed a high quality and performant time stretching but didn’t want to pay the whole development costs. So, we developed the time stretching for him at a much lower rate but kept the rights for the algorithm. That could be considered the start of our current licensing business. We still had to learn a lot about the business but the foundation was laid back then. From that point on we continued to develop more and more audio algorithms for licensing such as beat tracking, key and chord recognition, auto segmentation and monophonic as well as polyphonic pitch manipulation.

Later we started our own consumer product line so we could make our algorithms available in a way that we thought they should be used.

Tell me a bit about the current zplane team.

In 2013 Alexander retired from the active business at zplane since he was offered a professorship at Georgia Tech, Atlanta and moved to the USA. Now, there is Martin and me left as original founders, 4 employees and currently one intern.

Martin mainly handles the business side of things while I keep track of the technical stuff. Maik and Daniel do the hardcore programming and application development. Holger and Till take care of the research and science. Jonas – our current intern – helps the latter two. And everybody does customer support and internal testing.

All our employees did either do their master thesis at our company or at least had an internship before.

The Partner List on zplane.de is quite impressive. I see a few names of companies that a musician or producer using Studio One may not immediately associate with music, like Konami and Vinyl Dreams. What do you feel are some of the lesser-known applications of zplane technology that some users might not expect? I can see a use case for audio restoration in forensic audio, for example.

Thanks! It certainly took a while to grow this partner list. The first thing that comes to my mind when thinking about the less obvious applications is games – board games to be precise. Last year Harmonix and Hasbro released a board game using ELASTIQUE called Dropmix. It lets you construct music played by your smartphone by dropping playing cards on a game board. So, as you might guess elastique is running on the smartphone – but still the use case is kind of unusual.

Concerning forensic audio we currently have no customer using it for something comparable but we’re certainly open for that. Besides audio restoration even time stretching or pitch shifting may be very useful in that area.

Electronic music in particular is rife with happy accidents, where a technology developed for one purpose eventually finds its home in another. The most notable example being perhaps the Roland TB-303, or Auto-Tune’s obtuse origins in interpreting seismic data. Have the zplane team experienced anything like this in the development and subsequent application of their own technologies? Any surprising uses of the technologies that were not in mind in the development cycle?

Well, to be honest, I was always waiting for something like this to happen: people doing something completely different than what they were supposed to do. It didn’t happen as in the mentioned examples so far. But I was seriously impressed when I saw what people did with the chord tracks in Studio One: they applied the chord progressions to the reverb tail of other tracks. Obviously really didn’t expect that use case. However, it is truly amazing how musicians adapt new technology and convert it creatively into some new kind of expression.

How has your experience been working with the PreSonus Software team?

We’ve been working with the PreSonus Software team already for a long time even before they belonged to PreSonus. So, we have a very good and open relationship. Especially when implementing new technologies you need a lot of communication in order to get the most out of the technology but also to explain the limits of what is possible technology-wise. This communication back and forth does not only help on the implementation side but also gives us a lot of feedback and in the end improves the technology. So, especially with the implementation of the chord track and the harmonic editing this has been a very fruitful co-operation and it doesn’t end here. We’re working on the future already.

Can you talk about the difference between élastique as a plug-in and the different versions of élastique used in DAWs? Anything special in Studio One?

In fact, as long as the DAWs are using the latest version of ELASTIQUE PRO they are all using the same algorithm. Sometimes people still think that it sounds different in different hosts. This is partly voodoo and partly due to implementation differences: so in one host you have a fixed time stretch factor in others this is adapted continuously which will obviously cause audible differences. But still the base algorithm is the same everywhere. Our plugin Elastique Pitch takes the algorithm to only do pitch shifting which is mostly due to the limitations of a real-time plugin interface which is unable to handle time stretching.

Where do you see audio chord detection and harmonic editing go in the future?

Of course, I see that the accuracy of the chord detection will go up as well as the quality of the polyphonic pitch manipulation. It’s all already in the making. We’re currently moving away from classic signal processing and experimenting with modern deep learning and artificial intelligence approaches.  And our first results for the chord detection look very promising. Currently, we see the most potential in the chord detection to improve the overall performance of the harmonic editing – wrong assumptions on the original chord have a major negative impact on the result. That being said, one should always be aware that this will never be perfect – even professional musicians disagree on certain chord transcriptions and so will computers generate debatable results from time to time.

Also, the RETUNE algorithm still has a long way to go but improving the internal pitch detection using the above-mentioned techniques will most probably take us giant steps ahead. But we haven’t started to work on this yet.

One should never forget though that all of this is just a tool to help people to get creative.

There was a time when making edits to pitch and length independently of one another seemed impossible. The same could be said about isolating a single voice from a chord for independent editing. As barriers like these continue to be broken down… what’s the next big sonic quandary for a company like zplane to solve?

The holy grail of music signal processing has always been ‘demixing’ and that hasn’t been solved yet. There are new approaches but all of them don’t get close to studio quality. So that’ll stay on our roadmap.

However, we think that taking it step by step rather than only searching for the holy grail is the way to go. So, we’ll continue to improve the existing algorithms using new techniques and learn from that ourselves.

Also, it is not always about the new algorithm, sometimes it’s ‘just’ combining existing algorithms to something new. Similar to what we did with PreSonus in Studio One: we had the chord detection and the RETUNE algorithm and put it to together to create harmonic editing.

Similarly, we’re currently working on a new product combining a lot of our technologies to support musicians with their daily practice, train their ear and eventually help them become better musicians. So, keep your eyes and ears open for that next year.

Big thanks to Tim for taking the time for this interview! Learn more about zplane at the following links:

• zplane’s website
• zplane on Facebook