Piper voices licensing question

[DevTester89]

Hi,
I have a licensing question about using voices along with piper and if they can be used for commercial use:

I was looking at the MODEL_CARD file of many different voices and I noticed that most or some of the voices have "CC BY 4.0" license, but the voice was trained by using another voice such as lessac voice which has a "blizzard" license which is unclean to me and looks intimidating and I am not sure by that if I can use that voice commercially because of that (Such as libritts_r or "Joe" for example).
I also looked at the regular libritts (high) voice which is both CC BY 4.0 and was trained from scratch using a "train-clean-360" model which also has a CC BY 4.0 license.

My questions are:

1. If I see a voice licensed CC BY 4.0 that was trained or finetuned by another voice such as lessac (which is a blizzard licensed) - Can I still use it commercially ?
   Example: https://huggingface.co/rhasspy/piper-voices/tree/v1.0.0/en/en_US/libritts_r/medium
2. Can the regular "libritts (high)" voices that were trained from scratch be used commercially on any project ?
   Example: https://huggingface.co/rhasspy/piper-voices/tree/v1.0.0/en/en_US/libritts/high
3. Do all 904 voices of the "libritts (high)" have the CC BY 4.0 license if the base voice model is CC BY 4.0, such as "libritts (high)" ?
4. Can I use the voices licensed CC BY 4.0 on any project commercially ?

I appreciate you answer to each one of my questions if you know the answer.

Thanks !

[synesthesiam]

Partially quoting myself from a discussion elsewhere:

I'm not a lawyer, so I can't offer any legal advice unfortunately. The Piper project is intended for text to speech research, and does not impose any additional licenses on the checkpoints or voice models. I've done my best to include links to the original licenses and training history, so I personally believe it is the responsibility of the end user to make the ultimate judgement.

---

Sorry for the vague answer, but the truth is I have no idea how licenses are supposed to apply in the context of machine learning 🤷‍♂️

[DevTester89]

Thanks for the answer !

[DevTester89]

Since the piper project itself is licensed under the MIT license it means that it can be used in any other projects for any purpose, even commercially. Am I right ?

[synesthesiam]

That's right. However, most voices use the espeak-ng phonemizer from piper-phonemizer, which has a GPL license. I don't know if that somehow infects WAV files, but just beware.

I'm working with some folks to add more phonemizers, but it will likely require retraining voice models.

[DevTester89]

Thanks a lot for the quick reply !

[StoryHack]

I've been reading and rereading all those licenses. (I'm also not a lawyer, so, you know, this is not legal advice) Here are my thoughts on your questions:

(1) I believe that the blizzard license is restrictive and will not allow any derivative to be used commercially. It's the hardest to read license, but it restricts all usage of thhe dataset (which it calls "the Materials") to be "Research Purposes". It defines "Research Purposes" as (and I'm cutting out irrelevant pieces to hopefully add clarity):

"Research Purposes" means only those purposes associated with research and exploration ... using, incorporating or based upon the Materials (in whole or in part) ... , and ... excludes ... copying, ..., developing, adapting, amending or otherwise using the Materials for any commercial purpose, including the development, ... commercialisation, ... or licencing of voice synthesis ... products or services including, ... voice synthesis products or services... and other speech technology products and services."

(2) The license on the libritts dataset used for voice is "CC BY" which does permit commercial usage so long as you give attribution properly.

(3) Yes, all voices are covered. The dataset as a whole has the same license. That dataset was prepared from librivox recordings, which are all in the public domain.

(4) Again, yes, as long as you follow the attribution requirement.

---

The only big, ready to go dataset I know of is the LJSpeech dataset (https://keithito.com/LJ-Speech-Dataset/). I trained a high quality voice based on that for 2000 steps, which I released to the public domain as well. I'm not extatic with the results of some of the testing sentences, but for regular usage, it is fine (ymmv). I have no idea what could be done to train that voice better (maybe train with a larger batch size? Does that affect anything?). I have a RTX 3060 in a spare machine, and it took almost a month of constant thinking for that little guy to knock it out, so it's not practical to do too many tests. If I can ever figure out runpod.io better, so I can cheaply use better hardware, I may experiment on it again.

That said, I'd love to have a good multispeaker voice that I don't ever have to worry about licensing for. There are some good speakers in the libritts voice, but most sound tinny or noisy. I spent hours and hours going over the 900 samples looking for ones I like.

I have considered just starting a new training of the libritts-r set from scratch. I just need to be prepared to lose that machine for another month.

I have also experimented with various automated tools to prepare my own similar dataset, but only using good sounding (to my ear) recordings from librivox. And also having more samples from a smaller set of speakers. Prepping a dataset by hand from good librivox recordings takes forever. So far, many of my experiments end up with the same occational weird pronounciations as I got with the LJSpeech voice. I don't know if it's a problem with phoneme usage in the dataset, or something else I'm doing wrong with training.

[StoryHack]

@synesthesiam The LJSpeech sentences are from 7 public domain works. There's a list on the LJSpeech site. The man who put it together placed the dataset, including his preparation of sentences into the public domain as well.

@rhohndorf If you are interested, I'll upload a couple of datasets I've been paying with and put links in this thread.

[StoryHack]

@synesthesiam Oops, sorry, I just reread your question, this time correctly. My brain switched out LJSpeech for Lessac. I'll have to go look again.

[StoryHack]

@synesthesiam - looks like Lessac used the books Black Beauty, Mansfield Park, and Pride and Prejudice as the corpus, all of which are in the public domain.

[StoryHack]

@rhohndorf - here are a couple of datasets you can play with.

Libritts-r clean 100. This was from the same project as the libritts-r multispeaker voice that already exists. The libritts-r project has three main sets: clean-100, clean-360, and other-500. As I understand it, the numbers are approxomate hours of recordings that the set was made from. Various steps have whittled the total hours down a bit. The libritts-r voice that Piper already released came from the clean-360 set of recordings. This set I'm posting comes from the clean-100 set, so it has different narrators. I filtered out the files that the creators of the set said were failed restorations. Edit: I swapped out the version I posted before for one that has the same 22050 Hz sample rate as other voices. This set is about 7 GB.

Six Voices This dataset I built from six different Librivox narrators who I felt had high quality recording environments. Each narrator has 2-4hrs of recordings. This one is about 2GB.

My old process used audacity to automatically split the audio, using gaps of silence to mark when to split. This didn't always align with the start and end of sentences from the original text, but was okay. I ran those sentance(ish) cuts through whisper AI to transcribe them. Then I checked the whisper-generated texts by searching in the original text from Gutenberg to make sure the sentences exist (and are therefore more likely to be correct.) I threw out anything that could not easily be found. This way was pretty wasteful, as i usually started with about 12 hrs of recordings per narrator to get the 2-4 that ended up in this second set.

I've been working on a better process for building datasets from Librivox recordings, one that more closely follows how the original set that eventually became libritts-r was put together. I should be able to build much larger and correct datasets this way.

[rhohndorf]

Awesome! I'm travelling right now but I'll be home in a couple days. So i'll try to to give it a shot next weekend. I'll keep you updated.