What is RVC? Retrieval-based Voice Conversion explained

What is RVC?

RVC stands for Retrieval-based Voice Conversion. It is a free, open-source voice conversion framework published on GitHub by the RVC-Project team under the MIT license, built around a web app (the RVC WebUI) for training and using voice models. Its own tagline is "easily train a good VC model with voice data <= 10 mins".

Voice conversion means taking a recording of one voice and making it sound like another. RVC keeps what is being said or sung (the words, the melody, the timing, the phrasing) and swaps the voice that says it. That is why it became the standard way to make AI covers: you give it a vocal, it gives you the same performance in a different voice.

An RVC model is what you get when you train RVC on recordings of one voice. Community voice libraries are mostly RVC models, and Jammable trains RVC models too, so a voice you train on Jammable is the same kind of model people run on their own computers.

What does "retrieval-based" mean?

Most voice conversion has a problem called timbre leakage: a little of the original singer's tone survives the conversion, so the result sounds like a blend of two people. RVC's fix is in its name. When it trains a voice, it also saves the sound features of the training audio in a searchable index. When it converts, it looks up the closest matches in that index for each moment of your input and blends them in, so the output is built from sounds the target voice actually made.

The RVC README describes this as using retrieval "to replace the source feature to training-set feature" to reduce tone leakage. How strongly it does that is a setting, usually called the index rate or feature ratio.

How RVC turns one voice into another

1. 1

   Separate the vocal

   RVC works on a voice on its own. For a full song, the vocal is first split from the music. Jammable does this step for you; locally you use a separation tool first.

2. 2

   Read the content

   A speech model (ContentVec or a HuBERT model) turns the vocal into features that describe what is being said or sung, with as little of the original voice in them as possible.

3. 3

   Track the pitch

   A pitch extractor follows the melody note by note. RVC's recommended extractor is RMVPE; Applio also offers crepe, fcpe and swift. Transpose shifts this pitch curve up or down.

4. 4

   Look up the target voice

   If there is an .index file, RVC finds the closest matching features from the training audio and blends them in, by as much as the index rate says.

5. 5

   Generate the new voice

   The .pth model turns the content and the pitch into audio in the target voice. For a cover, that vocal is then mixed back over the music.

The .pth and .index files

The RVC FAQ says that with long, high-quality training audio the index matters less, and some people skip it. With shorter or noisier datasets it is what stops the result drifting back towards the original singer. If you have both files for a voice, keep them together.

What is RVC v2?

RVC v2 is the second generation of RVC's pretrained base models, the starting point every new voice is trained from. The RVC changelog added v2 base models in May 2023 (a 40k sample-rate version first, then 32k and 48k in June 2023). A v2 model uses the content encoder's full 768-value features for each moment of audio, where v1 squeezed them down to 256, and Applio trains v2 models by default.

You usually don't need to do anything about it: Applio reads the version from the model file and runs v1 and v2 models alike. The RVC README says a v3 base model, larger and trained on more data, is on the way.

RVC vs text-to-speech vs real-time voice changers

Text-to-speech creates speech from text, so it decides the rhythm and delivery itself. RVC needs a performance to convert, which is why it can sing: the singing comes from the input. Real-time voice changers convert your microphone as you speak. RVC can do that too: its WebUI ships a real-time GUI, and the README reports about 170 ms end-to-end latency (90 ms with ASIO audio devices, depending on drivers). Applio has a real-time tab as well.

The lines blur in practice. Jammable's text-to-speech speaks your text in a base voice first and then converts it with the RVC voice you picked, so you can hear typed text in a community voice. Jammable does not change your voice live, so for calls or streams you need a desktop tool. For converting a recording you made, see the AI voice changer.

How to use an RVC model

There are three ways, depending on whether you have model files and how much setup you want.

• Pick a voice in the browser. Jammable's voice library has more than 60,000 community voices, all ready to use. Nothing to install, and it works on a phone. Covers need a plan, from $1.99 for the first month.
• Upload your own .pth and .index. If you trained a model elsewhere or kept files from another site, upload them to Jammable and use the voice like any other.
• Run it locally. The RVC WebUI and Applio are free, open source and run on your own computer. RVC's README covers NVIDIA graphics cards as well as AMD, Intel and CPU-only machines. You control every setting, and you find and manage model files yourself.

How to upload a .pth and .index file to Jammable

1. 1

   Open your Library

   Go to your Library and choose Upload a voice.

2. 2

   Add the files

   Drop in the .pth file (required) and its .index file (optional, but recommended). Jammable checks the files and refuses ones that are invalid or corrupt.

3. 3

   Name it and add a picture

   Give the voice a name of up to 40 characters, a picture and, if you like, a short description and categories.

4. 4

   Choose who can use it

   Pick Private, only me or Public in the library, agree to the Terms of Use and press Publish. The voice gets its own page and is ready for covers and text-to-speech.

Only upload models of your own voice, or of a voice whose owner has agreed to it.

How to get good results with RVC

• Set the pitch first. Pitch (called transpose in RVC) is in semitones, and 12 semitones is one octave. If the input is a lower voice and the model is a higher one, try +12; the other way round, try -12. On Jammable the range is -24 to +24, and on a single-voice cover a shift that isn't a whole octave moves the music by the same amount so the vocal stays in key.
• Feed it a clean vocal. RVC converts whatever it hears, including reverb, harmonies and background noise. On Jammable, turn on Split out backing vocals when a song has harmonies or ad-libs, and My audio is just vocals when your file is already a dry vocal.
• Use the index, but not always at full strength. Locally, the index rate (Applio calls it Search Feature Ratio, default 0.75) trades closeness to the target voice against artifacts. Lower it if the result sounds metallic.
• Protect breaths and consonants. RVC's protect setting keeps "s", "t" and breath sounds from tearing. In the RVC WebUI, 0.5 turns it off and lower values protect more, at some cost to the index's effect.
• Use RMVPE for pitch. It is the default in Applio and the extractor RVC recommends: fast, light and good at avoiding dropped notes.
• Start from a good model. No setting rescues a voice trained on noisy audio. The RVC FAQ recommends 10 to 50 minutes of clean audio (5 to 10 minutes can be enough if it is very clean). Jammable accepts 3 minutes and voices usually sound better with 6 or more.