How RoEx Automix works under the hood

In this post, we'll peek under the hood of our RoEx Automix technology to give you an idea of how we treat your audio to give you back a professional and balanced sounding mix in minutes instead of days.

Revolutionising Audio with AI Music Mixing and Mastering

The field of audio mixing is rife with complexities. Balancing diverse sound components is a challenge faced by sound engineers worldwide. A key issue is 'masking', a phenomenon where a sound source, say a kick drum, becomes inaudible due to a louder element, like a bass guitar. To tackle this, engineers employ various techniques and audio effects, creating a harmonious and balanced mix where each sound has its distinct presence.

Fine-tuning volume levels, managing stereo positioning or 'panning', and applying equaliSation (EQ) to modulate specific frequencies are critical steps. Dynamic range compression (DRC), a technique used to control the loudness of sound sources, also plays a significant role. But imagine if this intricate process could be automated, eliminating manual intervention and leaving less room for error.

Welcome to the world of AI Music Mixing, where artificial intelligence plays the maestro. AI in audio mixing and mastering systems streamline and enhance the mixing process, resulting in perfectly balanced, high-quality audio. Join us as we delve into the fascinating domain of automated audio mixing and mastering powered by AI, and explore the future of music production.

Our Mixing System

Automix, our innovative AI music mixing system, redefines audio mixing by automating the complex process of determining the perfect audio effect settings. It meticulously maintains spatial balance, minimises masking, and regulates perceived loudness across any multitrack audio submitted for mixing.

Harnessing advanced music information retrieval (MIR) techniques, Automix thoroughly analyses each track or stem within the context of all other tracks/stems influencing it. This analytical stage takes place in the 'Multitrack Analysis Module', as depicted in the figure above. Here, we analyse and extract a multitude of multitrack audio features, some of which can be processed in real-time, offering the potential for real-time audio mixing.

The multitrack features extracted from each track/stem are then fed into an AI model that understands established mix engineering rules. This model discerns the most suitable settings for volume, EQ, DRC, Panning, and Reverb based on the sonic characteristics of each submitted track/stem, their interplay, and the musical style. Consequently, Automix exhibits an adept capability to mix individual stems and full multitracks effectively.

Upon the model's final determination of the best multitrack audio settings, these settings - EQ, DRC, panning, and loudness - are applied to each track/stem. The multitrack audio is then combined and peak-normalised to -3dBFs to reserve the necessary headroom for mastering.

Additionally, we've developed a mastering module based on similar technology, although its use is optional. This module takes the mixed audio and user's loudness preference into account to apply our AI mastering signal chain. The final product is a wave, FLAC, or mp3 file, fully optimised and ready for distribution on platforms like Spotify, Soundcloud, or Bandcamp.

Introducing Our API: Tonn - Powering AI Music Mixing and Mastering

We house our groundbreaking Automix technology on the robust Google Cloud Platform (GCP) as part of our Tonn API. It operates as a scalable, containerized application, capable of adjusting to the number of mixes needed at any given time. This flexibility allows us to efficiently scale up to match growing demand and meet the unique needs of our customers. Furthermore, it enables any external application to initiate multiple mix tasks simultaneously, significantly accelerating the mixing process of large multitrack projects.

Let's take a practical example to illustrate this: imagine a multitrack comprising 40 tracks - 10 guitar tracks, 10 drum tracks, 10 string tracks, and 10 synth tracks. With our Tonn API, users can create a mix task for each instrumental group to run concurrently. After the individual guitar, drum, string, and synth mixes are complete, users can then compile a final mix and mastered track from these mixes, as depicted in the figure above.

Our current benchmark using Tonn API demonstrates that our RoEx Automix technology can mix eight tracks of three minutes each (typical pop song duration) in approximately 4.5 minutes. We're continually striving to enhance this performance. If you're interested in experiencing the power of Tonn API firsthand, please reach out to us for an API key. You can find the Tonn API documentation here.

RoEx Realtime Mix: AI Music Mixing for Real-Time Applications

Building on the transformative technology behind Automix, we're currently developing RoEx Realtime Mix - a system designed to tackle real-time audio challenges. It's equipped to handle multiple audio channels simultaneously, analyse each channel, and apply audio effects on the go, aiming to reduce masking and boost auditory clarity.

Ideal for applications like live broadcasting, video games, or VR, where multiple sound sources dynamically interact over time, RoEx Realtime Mix can adapt and respond to changing stimuli. For instance, if the main character in a video game is speaking, the system would automatically highlight their speech while subtly filtering out other sounds to minimize masking.

Conclusion: The Future of Music Production with AI Music Mixing

Traditional music production or 'mixing' is a labour-intensive process that calls for a different set of skills than music creation. Typically involving numerous sound sources, each generated in a unique environment and with distinct attributes, the aim is to allow each source to be heard clearly, contributing to a harmonious and crisp blend of sounds. Achieving this balance is challenging and typically necessitates the skills of a professional sound engineer.

However, the advent of automated music production tools like Automix is transforming this landscape. By leveraging the power of AI music mixing, these tools tackle the complex aspects of music production, enabling musicians to deliver their content to their intended audience more rapidly, easily, and cost-effectively than if they had undertaken mixing and mastering themselves or outsourced to professional services.

This technology lowers the barriers to entry in the music industry, making a career in music more accessible to those lacking a technical background. As we move forward, AI music mixing and mastering systems like RoEx Automix promise to democratise music production, opening new doors of creative expression for artists around the world.