PieSquared

Probably worth adjusting the title -- air quality is not just a measure of pollution. In this case, both SF and Portland are filled (unfortunately) with wildfire smoke, rather than a man-made pollutant...

Are you folks planning on extending this to speech? I'm always been disappointed by how speech vocoder networks aren't built with any great inductive biases for waveform generation (besides very long receptive fields), and have desperately wanted something like this tuned for speech. It'd be great if a DSP-based architecture could be shown to outperform WaveNet / Parallel WaveNet / WaveRNN / WaveFlow / etc, and I'd love to use that in our own work. (There's been some attempts based on source-filter models like the "neural source filter (NSF) network", but nothing's caught on as best as I can tell.)

I don't think anything about the current set of tools is specific to sample rate; WaveNet, Tacotron, WaveRNN, etc, should work fine to generate 44.1Khz audio. They might just need slightly different hyperparameters or sizes to work well, or may take longer to train due to longer sequence lengths.

First of all, it's important to note that Tacotron and WaveNet are responsible for different parts of the speech synthesis pipeline, so the comparison here isn't quite accurate. Specifically, Tacotron takes a representation of the text (characters, phonemes, etc) and converts it into a frame-level acoustic representation (spectrograms, log mel spectrograms, etc, spaced every 5-25ms). WaveNet takes a frame-level representation of the audio (for example, the output of Tacotron, or phonemes with frame-level timing information) and converts it to a waveform.

Second, I don't see any reason why there shouldn't be an open-source Tacotron or WaveNet implementation that's as good as Google's model implementations. Implementing and training these models is expensive but not prohibitively so (nowadays, you could probably do it with $5,000 - $10,000, including experimentation costs).

That said, quality of text-to-speech systems is determined only partially by the quality of these models -- much if not most of the work of building high quality text to speech systems goes into things like high quality data collection systems, good data annotations, good normalization and NLP tailored towards the domain of the TTS system, multilanguage support, optimized inference implementations for server or mobile platforms, etc.

I'll try to answer these one at a time.

1. Does text-to-speech require AI?

This one is a bit tricky to answer since it requires defining "AI". AI as a moniker has been used to describe deep neural networks, search algorithms, expert systems and logic systems, particle filters, SVMs, etc etc. Almost all text-to-speech (TTS) systems are based on a combination of some of these machine-learning methods and digital signal processing (DPS), so I would say yeah, text-to-speech is exactly what AI describes, even if it doesn't resemble human-like thinking like other AI applications do.

2. Is there any active work in non-AI methods?

This one again is a bit tricky for the same reason as before. However, there's a ton of pieces of the TTS pipeline that aren't AI in the current sense of the word (machine learning with neural networks or HMMs or other classifiers). For example, concatenative systems will traditionally take a large database of audio, divide it into chunks, and then recombine those chunks, using some interpolation method such as (OLA, PSOLA) to overlap those chunks. Choosing the chunks to overlap to create the target speech becomes an AI / search problem, using some sort of acoustic model to predict the acoustic parameters of each frame and then using a Viterbi search algorithm with target / join costs to find the optimal chunks. As another example of non-AI parts of the pipeline, text normalization tends to involve a lot of hand-written rules; for example, should you say "5/10/2019" as "May tenth, twenty nineteen", "the tenth of may twenty nineteen", "the tenth of may two thousand nineteen", or even "october fifth twenty nineteen". This decision and the conversion is often done with a ton of handwritten rules or grammars (see Kestrel, Google's text normalization system, and the open-source version, cleverly named Sparrowhawk). Anyways, the real answer is that TTS is always a combination of AI (machine learning) approaches with specialized text and audio processing algorithms.

3. Which bits are the AI bits?

The AI bits are the bits where you need to make some sort of heuristic decisions, and you'd like to make them by imitating some target speech. For example, things like part of speech detection, predicting acoustic parameters (spectrograms, F0, etc), more recently waveform synthesis as well.

4. Do deep methods significantly improve on the state of the art?

Yes, though they also come at a cost. For example, deep sequence-to-sequence networks make great frame-level models: Tacotron and similar models can do things like emotional and stylized voice synthesis much better than what I've seen HMMs and other non-deep models do. As another example, WaveNet / WaveRNN / etc are some of the only parametric speech models (that is, generating the waveform from scratch instead of copying it from a database of audio) that can match the quality of concatenative models (copying audio from a database), but they can be quite difficult to deploy due to high computational cost. Overall, though, yeah, deep methods and all the improvements to neural networks in the past few years are having a profound impact on the quality and naturalness of TTS.

I'm an author on a few of these papers referenced (the Deep Voice papers from Baidu). I'm happy to answer any questions folks may have about neural speech synthesis, as I've been working on this for several years now.

In general, it's a fascinating space. There are challenges in text processing (not even mentioned in the blog), such as grapheme to phoneme conversion, part of speech detection, word sense disambiguation, text normalization, challenges in utterance-level modeling (spectrograms), and challenges in "spectrogram inversion" / waveform synthesis. The NLP components of the pipeline are often overlooked but are no less important than they were a few years ago -- part of speech / word sense is the difference between "Time is a CONstruct" and "I'm going to conSTRUCT a tower", and is the difference between "Let's drop that bass" being about a DJ or about a fish. The acoustic modeling phase (e.g. Tacotron, Deep Voice 3) works fairly well, and can produce some awesome demos with things like style tokens ("GST-Tacotron"), but still has a ways to go until it can encompass the full range of human inflection and emotion. At the waveform synthesis level, models like WaveRNN (with subscale modeling) and Parallel WaveNet make it possible to deploy modern waveform synthesis models, but it's still a major issue to deploy them onto low-power devices due to compute restrictions. Overall, lots of interesting challenges to work on, and we're making a lot of progress quite quickly; and I haven't even started talking about voice conversion or voice cloning!

The new Gmail interface has snooze, so if that's the only thing you're missing it'll stick around. I mostly miss the "Trips" feature, having recently switched from Inbox to new Gmail...

"Monoid" is an adjective that describes a data type.

Anything you describe as a monoid has to have three properties: you can add them together, there's an "empty" or "zero" value, and (a + b) + c = a + (b + c).

For example, strings with concatenation are a monoid, because you can use an empty string and the string + operator.

For example, integers with addition are a monoid, because you can use 0 and the integer + operator.

For example, sets are a monoid, because you can use the empty set and the union operator.

For example, a "Picture" data type could be a monoid, because you could have a fully-transparent picture and use an "overlay" operator to put one picture on top of another.

Why do you care? Well, for one thing, it just makes it easy to find the function or operator you want to use: as the OP said, if I feel like my data type should have a concat or append operator but I don't know what it's called, I just use mappend / mempty / mconcat.

But, as another practical example: "sum" in Python works for numbers, but I've seen folks assume that it works for anything that you can use + on, and so try and use strings with it. But, because "sum" isn't a general purpose tool, that doesn't work! In Haskell, on the other hand, the equivalent -- mconcat -- will work on anything that is a monoid. (And can be specialized to work faster for specific data structures.)

Other languages also have monoids. If someone tells you Haskell has monoids and other languages don't, what they really mean is, Haskell makes explicit the monoid pattern / interface, where other languages have it implicitly for different data types. Talking about the pattern explicitly isn't revolutionary, but it can be pretty useful for discoverability and writing "general" algorithms.

Instead of calling data structures monoids, you'd get the same effect if, as a programming language community, you decided that as many types as possible should support "+", and that every data type that can should have a makeEmptyThing() method, and that it would be weird and not ok if x + makeEmptyThing() didn't equal x and if (x + y) + z didn't equal x + (y + z), and then subtly shamed any libraries that defined makeEmptyThing() and + in ways that didn't follow that pattern. But if your programming language community did this (for consistency) you'd probably want to come up with a name for it -- "Addable", "Concatable", etc -- and the Haskell community chose "Monoid" (because of relationships to theory etc etc).

To be fair, this article is really pretty "cutting edge" as far as Haskell goes. It describing a new feature they're planning on adding to the Haskell compiler, one that significantly extends the type system in an interesting way. I think it's fair for them to assume comfort and familiarity with Haskell syntax and code when writing this article; their audience is folks who are likely to know about and use somewhat esoteric GHC extensions...