Loren Lugosch

What does Hegel mean by “Reason”?

2021-05-15T00:00:00-07:00

I finally finished reading the big and baffling Phenomenology of Spirit by Georg Wilhelm Friedrich Hegel.

The gist of the book is that something called “Spirit” (“Geist”) develops from simple consciousness of the “Here and Now”, to abstract concepts, to knowledge of self and the social/ethical world, to more sophisticated forms of art, science, religion, and finally Absolute Knowledge. Along the way, Spirit invents big world-historical things like Stoicism, Skepticism, Christianity, and Kant. (See the German audiobook cover below, which incidentally cracks me up because it looks like something from a cult.)

For me, as for other people, the most fruitful way to get through the book turned out to be not to try following the logic (if it exists!) of the text from section to section, but rather to sit back and enjoy the stream-of-consciousness of an extremely erudite man, who occasionally drops an interesting phrase or formulation that people like Marx and Sartre later picked up.¹

Still, there was one aspect of the Phenomenology of Spirit that defied my stream-of-consciousness-style reading and gave me pause: namely, the way in which Hegel defines the word “Reason” (“Vernunft”).

Here are a few selections from the beginning of the “Reason” section of the book:

[Self-consciousness as Reason] is certain that it is itself reality, or that everything actual is none other than itself… [Sec. 232, p. 139]

Reason is the certainty of consciousness that it is all reality; thus does idealism express its Notion. [Sec. 233, p. 140]

Reason is the certainty of being all reality. [Sec. 235, p. 142]

Reason, as it immediately comes before us as the certainty of consciousness that is is all reality, … [Sec. 242, p. 146]

This is an odd definition for “Reason”. It sounds more like a definition of “idealism”. I think most philosophers, or AI people like me, would instead define “Reason” as “producing (or the faculty of producing) valid new assertions given other assertions”, or something like that. But Hegel is definitely very keen on his own unusual definition, as he makes a point of saying it at least four times.

Yet Hegel’s definition doesn’t seem that useful or fitting even in his own book. Take, for example, this description of what Reason does:

Reason sets to work to know the truth, to find in the form of a Notion that which, for ‘meaning’ and ‘perceiving’, is a Thing; i.e. it seeks to possess in thinghood the consciousness only of itself. [Sec. 240, p. 145]

Plugging in “the certainty of consciousness that it is all reality” for “Reason” does not really seem to work here:

[The certainty of consciousness that it is all reality] sets to work to know the truth…

Or here, in which Hegel starts to take one of his many potshots at “sound common sense”:

Since self-consciousness knows itself to be a moment of the being-for-self of this substance, it expresses the existence of the law within itself as follows: sound Reason knows immediately what is right and good. [Sec. 422, p. 253]

$\rightarrow$

sound [certainty of consciousness that it is all reality] knows immediately what is right and good (?)

One possibility is that Hegel is not actually using “the certainty of consciousness that it is all reality” as a definition of “Reason”. Instead, maybe he just wants to firmly establish the notion that “consciousness is all reality”, and this advanced stage of the development of Spirit in his book seemed like a good place to do it. We know what Reason is; so instead of wasting the reader’s time giving a “definition” they already have, why not use the gap where a definition would normally go as a free space to stick an idea he wants to make sure we share with him? Hegel seems to pull many such tricks in this book.

Another possibility is that when Hegel says “Reason is the certainty of being all reality”, the “is” is not indicating identity but rather just marking a predicate: that is, “the certainty of being all reality” is one aspect of the thing Hegel calls “Reason”, but not the complete definition (which he never provides). But the predicate of “the certainty of being all reality” still does not really jive with our commonsensical notion of “Reason”. So maybe Hegel is just casually asserting that “if you’re a thinking person, a reasonable German intellectual in the year 1807, and you’ve read your Kant, your Fichte, and your Schelling, then of course you’re an idealist and you already know that consciousness is all reality”.

A third possibility is that “Vernunft” has overtones in German that “Reason” does not have in English and that could make Hegel’s odd usage more sensible. This is at least true of “Geist” (cognate with English “ghost”), which we could translate in English as “Spirit” (as is done for Hegel), “Mind”, or maybe “intellectual activity” (“humanities” in German is “Geisteswissenschaften”, in contrast to “Naturwissenschaften”, “natural science”). My German isn’t good enough for me to know whether something similar can be said of “Vernunft”.

(A fourth possibility is that Hegel is full of shit and we should just put the book down. In math, this is called “the trivial solution”.)

It’s also interesting to look at how Hegel uses “Reason” (“Vernunft”) in some of his later writings. In the preface² to his 1820 Philosophy of Right Hegel attributes an idea about Reality and Reason to Plato:

Yet [Plato] has proved himself to be a great mind because the very principle and central distinguishing feature of his idea is the pivot upon which the world-wide revolution then in process turned: What is rational is real; And what is real is rational. (Was vernünftig ist, das ist wirklich; und was wirklich ist, das ist vernünftig.)

Another famous instance is his phrase “the Cunning of Reason” (“die List der Vernunft”) in his Lectures on the Philosophy of History. The idea is that individual humans who think they are simply acting in their own self-interest, with no regard for whatever shenanigans Spirit is trying to pull with world history at the moment, are, by being rational, unwittingly carrying out what Reason wants to have happen in the world:

The particular interest linked to passion is thus inseparable from the actualization of the universal principle; for the universal is the outcome of the particular and determinate, and from its negation. … This may be called the Cunning of Reason, that it allows the passions to work for it, while what it brings into existence suffers loss and injury.

Reason as logic; Reason as idealism; Reason as Spirit moving history forward by tricking us rational peasants into doing what it wants. Is there some simple notion underlying these conceptions for Hegel? You’d probably have to get a philosophy degree to prove it. As a programmer, polysemy frustrates me. As a poet, it fascinates me.

For a more systematic reading by someone who is not an amateur like me, I recommend Gregory Sadler’s “Half Hour Hegel”, a years-long lecture series in which Sadler attempts to explain or interpret every section of the book. I’m not that patient! ↩
At the end of that preface, Hegel gives a beautiful metaphor for his conception of philosophy: “When philosophy paints its grey in grey, one form of life has become old, and by means of grey it cannot be rejuvenated, but only known. The owl of Minerva takes its flight only when the shades of night are gathering.” (“Wenn die Philosophie ihr Grau in Grau malt, dann ist eine Gestalt des Lebens alt geworden, und mit Grau in Grau läßt sie sich nicht verjüngen, sondern nur erkennen; die Eule der Minerva beginnt erst mit der einbrechenden Dämmerung ihren Flug.”) ↩

First PC build

2021-03-03T00:00:00-08:00

My everyday work computer is a MacBook Pro that I’ve had since 2013. It’s a great machine and continues to serve me well, but I was moved in a moment of pandemic malaise to treat myself to a little upgrade.

I decided to build (for the first time!) a desktop PC instead of getting another laptop, for a few reasons:

I’m working from home now, and I’ll probably have to for the foreseeable future.
As a holder of electrical and computer engineering degrees, I felt a little embarrassed that I had never built my own computer. (Cf. the engine repair episode of Frasier.)
I wanted to try modern PC games that require a little more horsepower than Undertale and that I can’t play on my Nintendo Switch.

So here’s how my first PC build went.

The GPU

As you might guess from my website’s favicon, and my all-around Stack More Layers demeanor, I started with the GPU.

I wanted a not-too-expensive GPU that I could use both for gaming and for training reasonably large neural nets. (I do have access to lots of powerful GPUs in the Mila cluster, but it’s nice not to have to share with other people sometimes, and to be able to leave a model training for, like, a month.)

The best option looked to be the Nvidia (nVidia? NVIDIA?) RTX 3080. But whether due to pandemic disruptions, or Bitcoin bros, or BERT bros, or something else, it’s off shelves everywhere.

I waited a month to see if the RTX 3080 would come back in stock anywhere, but no luck. Instead, I consulted Tim Dettmers’s magisterial deep learning GPU guide and found that, among not-unavailable GPUs, the GTX 1080 Ti had the best performance per dollar, and good performance for training transformer models. So I picked one up from a guy nearby in Montreal off Kijiji.

The rest

Using PCPartPicker’s Completed Builds search feature, I found a build for a PC based around the RTX 3080 and followed it to the letter. I probably could have optimized the components more for what I want, but having not ever built a PC before, I didn’t want to accidentally select components that were incompatible with each other. (PCPartPicker does have a compatibility checker, but I don’t know how idiot-proof it is.)

Then I bought all the components, put it together, and breathed a sigh of relief when I pushed the power button for the first time and the fans turned on.

The build process felt like assembling a LEGO (Lego?) set, but with the instructions distributed across multiple boxes and the additional stress of unfamiliar sensitive electronic components that would cost a lot of money to replace.

I started by reading the motherboard manual, which told me to put in the CPU. That was a little nerve-wracking because I really had to pull the little whammy bar (or so I’m inclined to call it) tight to get the chip in, which felt wrong.

Then I installed the fan, with the help of a video guide. I wasn’t even sure what the orientation of the fan within the case was supposed to be, so I tried to infer it from a photo taken by the creator of the build. Because I hadn’t put the motherboard into the case yet and had to operate solely on other landmarks, this was mystifying, until I realized that the photo showed a machine with glowy RAM sticks instead of the non-glowy RAM I had. Thus I learned that people use RAM that glows.

The PC case itself was a complex enough beast that I had to follow yet another (thorough, excellent) video guide on how to get it open and wire things up to the motherboard.

It’s alive! And it glows. But the RGB effects are a little distracting, so I’ll probably deactivate them. Besides, I don’t need them to feel like a real Gamer. As it is written: “When you Game, do not be like the hypocrites, for they love to be seen as Gamers with their showy RGB effects. Instead, when you Game, go into your room, close the door, and Game in secret.”

Some extras

The build I followed did not include any peripheral devices, so I picked some up, again using PCPartPicker. I’ve never been picky about keyboards, so I just filtered for low prices, sorted by rating, and picked the top one. Same for the monitor. For the mouse, I bought this nice glowy one with some sort of fangly beast on it. More RGB!

Also, apparently, motherboards cannot connect to the Internet without help! As I discovered when I turned on the PC. (Stop laughing!) I don’t have an easily-reachable Ethernet cable, so I had to wait a couple days more for a Wi-Fi PCIe card to arrive before I could actually use the computer.

Incidentally, the package for the Wi-Fi card featured maybe my favorite acronym ever:

Testing ‘er out

For my first game, I picked up Star Wars: Squadrons, the recently released spiritual successor to the old Rogue Squadron series of dogfighting games. I’m bad at it. But it’s fun! And it runs great on my new machine. It’s very pretty, but I haven’t looked up how to do a screenshot yet, so here’s a picture swiped from the Wikipedia article on the game.

I also downloaded the original Half-Life, a game I bought on Steam a while back that my MacBook rudely made unplayable with the Catalina macOS update.

Yay! Almost done. The last step for me is getting Linux dual-booted so that I can use all my machine learning tools.

End-to-end models falling short on SLURP

2020-12-19T00:00:00-08:00

There’s a hot new dataset for spoken language understanding: SLURP. I’m excited about SLURP for a couple reasons:

It’s way bigger and more challenging than existing open-source SLU datasets:

The authors found that an end-to-end model did not work:

We have tested several SOTA E2E-SLU systems on SLURP, including (Lugosch et al., 2019b) which produces SOTA results on the FSC corpus. However, re-training these models on this more complex domain did not converge or result in meaningful outputs. Note that these models were developed to solve much easier tasks (e.g. a single domain).

Having just written a SpeechBrain recipe for end-to-end SLU with my much simpler Timers and Such dataset, I thought I’d give that recipe a whirl on SLURP. The model used in the recipe looks like this:

The output dictionaries for SLURP look a bit different from the ones in Timers and Such or Fluent Speech Commands:


{
  "scenario": "alarm",
  "action": "query",
  "entities": [
    {"type": "event_name", "filler": "dance class"}
  ]
}

But—much like a honey badger—our autoregressive sequence-to-sequence model doesn’t care. It just generates the dictionary character-by-character, no matter what the format looks like.

The authors of the SLURP paper provide baseline results with a more task-specific model, HerMiT, which uses an ASR model to predict a transcript and a conditional random field for each of (scenario, action, entities). The outputs are generated using the Viterbi algorithm.

The authors kindly released their tool for computing performance metrics, including a new “SLU-F1” metric they propose. I used this tool and got the following results (EDIT: I left the model training a little longer and updated the numbers EDIT EDIT: I got rid of a coverage penalty term in the beam search and updated the numbers again):

Model	`scenario` (accuracy)	`action` (accuracy)	`intent` (accuracy)	Word-F1	Char-F1	SLU-F1
End-to-end	~~71.17~~ 81.73	~~65.43~~ 77.11	~~61.85~~ 75.05	~~45.57~~ 61.24	~~49.23~~ 65.42	~~47.33~~ 63.26
HerMiT	85.69	81.42	78.33	69.34	72.39	70.83

Turns out HerMiT does way better than our end-to-end model. This isn’t totally surprising because HerMiT has a lot of structure built in that our autoregressive model has to learn from scratch. For instance, I don’t think it’s possible for their model to output "brussia" (one of the more charming output mistakes I noticed early in training).

Another difference is that their ASR model is trained on Multi-ASR, a massive 24,000 hour dataset formed by Captain-Planeting the LibriSpeech, Switchboard, Fisher, CommonVoice, AMI, and ICSI datasets—whereas my encoder is only pre-trained using the 1,000 hours of LibriSpeech.

So: how much of the gap is due to more/better audio? The authors also report some results when applying HerMiT to the gold transcripts instead of the ASR output; similarly, we can feed the gold transcripts into our sequence-to-sequence model instead of audio and compare the results (EDIT again, I let the model train a bit longer/got rid of the pesky coverage penalty and updated the numbers):

Model	`scenario` (accuracy)	`action` (accuracy)	`intent` (accuracy)
End-to-end (text input, gold transcripts)	~~89.91~~ 90.81	~~86.54~~ 88.29	~~85.43~~ 87.28
HerMiT (gold transcripts)	90.15	86.99	84.84

Here the simple sequence-to-sequence model actually does about as well as HerMiT (EDIT actually a bit better!). This suggests that the audio side of things is more where our problems lie.

In summary, I have attempted to defend the honor of end-to-end models: we can indeed train one on SLURP and get semi-reasonable outputs. Note, though, that I’ve done no hyperparameter tuning on the model (except to increase the number of training epochs and getting rid of the coverage penalty term), so it’s possible we could do better with a little elbow grease—maybe starting by swapping out the now-unfashionable RNNs I used in the encoder and decoder with ✨Transformers✨.

Siri from scratch! (Not really.)

2020-12-11T00:00:00-08:00

I make fairly heavy use of the voice assistant on my phone for things like setting timers while cooking. As a result, when I spent some time this summer at my in-laws’ place—where there was no cell signal and not-very-good Wi-Fi—I often tried using Siri only to get a sad little “sorry, no Internet :(“ response. (#FirstWorldProblems.)

This reminded me of a tweet I saw a while back:

like 90% of my voice assistant usage is setting timers, math, and unit conversions

can we ship an offline on-device model that does this well and nothing else?
— Arkadiy Kukarkin (@parkan) April 19, 2019

Now, it just so happens that I 1) am a big fan of doing things offline rather than in the cloud, 2) have some experience training speech models, and 3) wanted to procrastinate. (The perfect storm.) So here’s the record of me taking a crack at it.

The goal

The goal is to make a box that can take as input the speech signal and output what the speaker wants. The output takes the form of a dictionary containing the semantics of the utterance¹—i.e. the intent, slots, and slot values, and maybe some other fancy things like Named Entities. I shall casually refer to this output simply as the “intent”, a common synecdoche. This is called “spoken language understanding” (SLU).

The “decoupled” approach

The first thing I tried was a straightforward “decoupled” approach: I trained a general-purpose automatic speech recognition (ASR) part and a separate domain-specific natural language understanding (NLU) part.

To train the ASR part, I used LibriSpeech, a big dataset of American English speakers reading audiobooks. (Or rather, my SpeechBrain colleagues did, and I just loaded their best model checkpoint. Thanks, Ju-Chieh + Mirco + Abdel + Peter!)

To train the NLU part, we need text labeled with intents. For this, I wrote a script to generate a bunch of labeled random phrases for the four types of commands I wanted: setting timers, converting units (length, volume, temperature), setting alarms, and simple math. Here’s a few examples:


("how many inches are there in 256 centimeters",
{
  'intent': 'UnitConversion', 
  'slots': {
    'unit1': 'centimeter', 
    'unit2': 'inch', 
    'amount': 256
  }
})

("set my alarm for 8:03AM",
{
  'intent': 'SetAlarm', 
  'slots': {
    'am_or_pm': 'AM', 
    'alarm_hour': 8, 
    'alarm_minute': 3
  }
})

("what's 37.67 minus 75.7",
{
  'intent': 'SimpleMath', 
  'slots': {
    'number1': 37.67, 
    'number2': 75.7, 
    'op': ' minus '
  }
})

I trained an attention model to ingest the transcript and autoregressively predict these dictionaries as strings, one character at a time.

This model gets perfect accuracy on the test transcripts, which makes sense, since this is a pretty simple domain.

Does that mean our system as a whole will get perfect accuracy on test audio? No, because there is the chance that the ASR part will incorrectly transcribe the input and that the NLU part will fail as a result. To measure how well our system works, we need some actual in-domain audio data.

Some end-to-end test (and training) data

So I recorded a few friends and colleagues speaking the generated prompts. My recordees kindly gave me their consent to release their anonymized recordings, which you can find here. I manually segmented and cleaned (= fixed the label, when someone misspoke) their recordings, yielding me a modest 271 audios. (This exercise builds character. I highly recommend it! So does Andrej Karpathy.)

I split the recordings into train/dev/test sets so that each speaker was in only one of the sets, with:

144 audios (4 speakers) for the train set (train-real),
72 audios (2 speakers) for the dev set (dev-real), and
55 audios (5 speakers) for the test set (test-real).

(For the decoupled approach, we don’t need audio during training, but we will shortly for an alternative approach.)

It’s hard to get meaningful accuracy estimates with only 55 test examples, so I also generated a bunch of synthetic audio (train-synth, dev-synth, test-synth) by synthesizing all my NLU training text data with Facebook’s VoiceLoop text-to-speech model.² Like I did for the real speakers, I split the 22 synthetic speakers into train/dev/test sets with no speaker overlap.

In total, I generated around 200,000 audios. I couldn’t figure out how to parallelize the generation process—with simple multithreading I ran into some concurrency problems with the vocoder tool called by VoiceLoop—so I just did it sequentially, which took more than a week to finish.

Lucky for you, I’ve uploaded both the real speech and synthesized speech here! I call the complete dataset Timers and Such v0.1. It’s v0.1 because the test set of real speakers is probably too small for it to be used to meaningfully compare results from different approaches. It would be nice to scale this up to more real speakers and make a v1.0 that people can actually use for R&D.

First results

I tested the system out on test-real and test-synth and measured the overall accuracy (= if any slot is wrong, the whole utterance is considered wrong). Here’s the results, averaged over 5 seeds:

Model	`test-real`	`test-synth`
Decoupled	23.6% $\pm$ 7.3%	18.7% $\pm$ 5.1%

Wow, it’s only getting the answer right a fifth of the time. That sucks! What could the problem be?

Let’s look at some of the ASR outputs for test-real:


True transcript: "SET A TIMER FOR EIGHT MINUTES"
ASR transcript: "SAID A TIMID FOR EIGHT MINUTES"

True transcript: "HOW MANY TEASPOONS ARE THERE IN SIXTY SEVEN TABLESPOONS"
ASR transcript: "AWMAN IN TEASPOILS ADHERED IN SEVEN ESSAYS SEVERN TABLESPOONS"

Now, this ASR model gets a WER of 3% (not SOTA, but good) on the test-clean subset of LibriSpeech. So why do the outputs look so bad here?

One issue is accent mismatch. LibriSpeech has only American English speakers, whereas only 3 of 11 real speakers in Timers and Such have American accents. There’s not much we could do about this, save re-train the ASR model on a more diverse set of accents.
The other issue is language model (LM) mismatch. The ASR model has an LM trained on LibriSpeech’s LM text data. That text data comes from Project Gutenberg books, which is a very different domain: for instance, “SAID” is more likely to appear at the beginning of a transcript from a book than “SET”.

So I trained an LM on the Timers and Such transcripts, and used that as the ASR model’s LM instead of the LibriSpeech LM. This ends up fixing a lot of ASR mistakes, but not all of them.³

Model	`test-real`	`test-synth`
Decoupled (LibriSpeech LM)	23.6% $\pm$ 7.3%	18.7% $\pm$ 5.1%
Decoupled (Timers and Such LM)	44.4% $\pm$ 6.9%	31.9% $\pm$ 3.9%

Sort-of end-to-end training: the “multistage” approach

Instead of training on the true transcript, we might want to train the NLU model using the ASR transcript, to make the NLU part more resilient to ASR errors. Google calls this a “multistage” end-to-end SLU model: it still uses distinct ASR and NLU parts, but the complete system is trained on audio data.

Running this experiment, multistage training works a lot better than the decoupled approach, with or without an appropriate LM.

Model	`test-real`	`test-synth`
Decoupled (LibriSpeech LM)	23.6% $\pm$ 7.3%	18.7% $\pm$ 5.1%
Decoupled (Timers and Such LM)	44.4% $\pm$ 6.9%	31.9% $\pm$ 3.9%
Multistage (LibriSpeech LM)	69.8% $\pm$ 3.5%	69.9% $\pm$ 2.5%
Multistage (Timers and Such LM)	75.3% $\pm$ 4.2%	73.1% $\pm$ 8.7%

For-real end-to-end training: the “direct” approach

Still, there’s a few disadvantages to the multistage ASR-NLU approach.

We need an intermediate search step to predict a transcript during training. The search is inherently sequential and ends up being the slowest part of training.
It’s difficult⁴ to backpropagate through the discrete search into the encoder, so the model can’t learn to give more priority to recognizing words that are more relevant to the SLU task, as opposed to less informative words like “the” and “please”.
Ultimately, we don’t actually care about the transcript for this application: we just want the intent. By predicting the transcript, we’re wasting FLOPs.⁵

So: why not train a model to just map directly from speech to intent?⁶ To quote Vapnik: “When solving a problem of interest, do not solve a more general problem as an intermediate step.”⁷

The direct approach faces an additional difficulty: it has to learn what speech sounds like from scratch. To make the comparison with the ASR-based models more fair, we can use transfer learning. This can be done simply by popping the encoder out of the pre-trained LibriSpeech ASR model and using it as a feature extractor in the SLU model.⁸

We get pretty good results with this approach: around the same performance as the multistage model with an appropriate language model (slightly worse, within a standard deviation). The direct model also trained a lot faster: 1h 42m for one epoch (running on a beastly Quadro RTX 8000), compared with 2h 53m for the multistage model on the same machine.

Model	`test-real`	`test-synth`
Decoupled (LibriSpeech LM)	23.6% $\pm$ 7.3%	18.7% $\pm$ 5.1%
Decoupled (Timers and Such LM)	44.4% $\pm$ 6.9%	31.9% $\pm$ 3.9%
Multistage (LibriSpeech LM)	69.8% $\pm$ 3.5%	69.9% $\pm$ 2.5%
Multistage (Timers and Such LM)	75.3% $\pm$ 4.2%	73.1% $\pm$ 8.7%
Direct	74.5% $\pm$ 6.9%	96.1% $\pm$ 0.2%

Notice anything interesting? The direct model performs a lot better on test-synth than the other models. This makes sense: the direct model has access to the raw speech features, so it can learn the idiosyncrasies of the speech synthesizer and recognize the synthetic test speech more easily. (Of course, we don’t care about the performance on synthetic speech; we only care about how well this works for human speakers.)

Are we done?

Did we achieve our goal from the outset of “an offline on-device model that [recognizes numeric commands] well and nothing else”?

As for “on-device”: The model has 170 million parameters, most of which live in the pre-trained ASR part. This requires 680MB of storage using single-precision floats; for reference, right now the maximum app download size for Android is 100MB, so we would probably have a hard time putting this on a phone. We would need to fiddle with the ASR hyperparameters a bit to shrink the encoder, but this is definitely doable. In fact, the direct SLU models I trained in my previous papers had a little over 1 million parameters—this was one of the main selling points for end-to-end SLU models in the original paper.
As for “well”: Does 75% accuracy count as good? Probably not, unless you’re OK with your cooking timer being set for 10 hours instead of 10 minutes now and then. For starters, we saw that the training data—LibriSpeech and the synthetic Timers and Such speech—is almost entirely American English, so we would need to collect some more accents for the training data. But I’m American, so it works well for me! (The linguistic equivalent of “it runs on my machine”.)

Code and data

I wrote the code for all these experiments as a set of recipes to be included in the SpeechBrain toolkit. It’s not available to the public yet, but it will be soon. In the meantime, you can train a model on Timers and Such v0.1 using my older end-to-end SLU code here—though I would recommend waiting until SpeechBrain comes out, since my SpeechBrain SLU recipes are a lot cleaner and easier to use.

The disadvantage of formulating the problem this way is that we need someone to design this output format and write a program to map the intent to a sequence of actions. A better way to formulate the problem might be to use reinforcement learning: let the agent act in response to requests and learn to act to maximize some reward signal (“did the agent do what I want?”), without the need for any hard-coded semantics, as was originally suggested in this paper. The question then becomes: what should our action space look like? High-level actions, like pushing buttons in an app (inflexible, but easier to learn)? Or low-level actions, like reading and writing to locations in memory (flexible, but more difficult to learn)? And can we train the model with some sort of imitation learning or simulation, so that we don’t have to wait forever for it to learn from human feedback? Interesting and challenging questions that I won’t linger on here, but which I’d like to think more about in the future. ↩
I wrote a whole paper about this idea! ↩
We could probably do better using an FST-based speech recognizer, which would allow us to perfectly constrain the model to only outputting sentences that fit a certain grammar (that is, the “G” part of “HCLG”). ↩
Though not impossible, using tricks like Gumbel-Softmax and the straight-through estimator. ↩
A fourth (more subtle) disadvantage of ASR-based SLU models is that the speech signal may contain information that is not present in the transcript. For example, sarcasm is not always apparent from just looking at a transcript. This is not really relevant for the simple numeric commands we’re dealing with here, but for more general-purpose robust language understanding in robots of the future, non-transcript information might be crucial, in addition to other multimodal information like visual cues. ↩
This was proposed by my friend Dima Serdyuk in his 2018 ICASSP paper. A few other groups—including Fluent.ai, where I worked before my PhD—had had similar ideas earlier, but as far as I’m aware, Dima was the first to get a truly end-to-end SLU model to work, without any sort of ASR-based inductive bias or transfer learning. ↩
Actually, there are tons of examples of solving a more general problem yielding better results for the problem of interest, like language model pre-training for text classification. But as the amount of data we have for the problem of interest goes to infinity, Vapnik is right. ↩
In this paper, I proposed a somewhat more complicated way of pre-training the encoder using phoneme and word targets from a forced aligner. The idea was that using word targets would be ideal (and more amenable to an idea I had for using pre-trained word embeddings to help the model understand the meaning of synonyms not present in the SLU training set), but using too many word targets would be expensive, which is why we used phoneme targets as well. Using the pre-trained ASR model’s encoder was a lot simpler to implement, though I haven’t done a fair comparison with the forced alignment approach yet. ↩

Sequence-to-sequence learning with Transducers

2020-11-16T00:00:00-08:00

The Transducer (sometimes called the “RNN Transducer” or “RNN-T”, though it need not use RNNs) is a sequence-to-sequence model proposed by Alex Graves in “Sequence Transduction with Recurrent Neural Networks”. The paper was published at the ICML 2012 Workshop on Representation Learning. Graves showed that the Transducer was a sensible model to use for speech recognition, achieving good results on a small dataset (TIMIT).

Since then, the Transducer hasn’t been used as much compared to CTC models (like Deep Speech 2) or attention models (like Listen, Attend, and Spell). Last year, however, the Transducer got some serious attention when Google researchers showed that it could enable entirely on-device low-latency speech recognition for Pixel phones. And more recently, the Transducer was used to achieve a new state-of-the-art word error rate for the LibriSpeech benchmark.¹

So what is the Transducer, and when might you want to use it? In this post, we will see where Transducer models fit in with other sequence-to-sequence models and a detailed explanation of how they work.

This post also includes a Colab notebook with a PyTorch implementation of the Transducer for a toy problem—which you can skip straight to here.

Attention models

The problems we’re interested in here are sequence transduction problems, where the goal is to map an input sequence $\mathbf{x} = \{x_1, x_2, \dots x_T\}$ to an output sequence $\mathbf{y} = \{y_1, y_2, \dots, y_U\}$.

The go-to models for sequence transduction problems are attention-based sequence-to-sequence models, like RNN encoder-decoder models or Transformers.

Here’s a diagram of an attention model. (In the diagrams below, I’ll use red to indicate that a module has access to $\mathbf{x}$, blue to indicate access to $\mathbf{y}$, and purple to indicate access to both $\mathbf{x}$ and $\mathbf{y}$.)

The model encodes the input $\mathbf{x}$ into a sequence of feature vectors, then computes the probability of the next output $y_u$ as a function of the encoded input and previous outputs. The attention mechanism allows the decoder to look at different parts of the input sequence when predicting each output. Here, for example, is a heatmap of where the decoder is looking during a translation task (from Bahdanau et al.):

Attention models can be applied to any problem, but they are not always the best choice for certain problems, like speech recognition, for a few reasons²:

The attention operation is expensive for long input sequences. The complexity of attending to the entire input for every output is $O(TU)$—and for audio, $T$ and $U$ are big.
Attention models cannot be run online (in real time), since the entire input sequence needs to be available before the decoder can attend to it.
Attention models also don’t take advantage of the fact that, for speech recognition, the alignment between inputs and outputs is monotonic: that is, if word A comes after word B in the transcript, word A must come after word B in the audio signal (see image below, from Chan et al., for an example of a monotonic alignment). The fact that attention models lack this inductive bias seems to make them harder to train for speech recognition; it’s common to add auxiliary loss terms to stabilize training.

This leads us to Connectionist Temporal Classification (CTC) models, which are more suitable for some problems than attention models.

CTC models

CTC models assume that there is a monotonic input-output alignment³. This ends up making the model a lot simpler.

So simple! We only need a single neural net to implement a CTC model, and no expensive global attention mechanism.

But CTC models have a couple problems of their own:

Problem 1: The output sequence length $U$ has to be smaller than the input sequence length $T$. This might not seem like a problem for speech recognition, where $T$ is much larger than $U$—but it prevents us from using a model architecture that does a lot of pooling, which can make the model a lot faster.
Problem 2: The outputs are assumed to be independent of each other. The result is that CTC models often produce outputs that are obviously wrong, like “I eight food” instead of “I ate food”. Getting good results with CTC usually requires a search algorithm that incorporates a secondary language model.⁴

Can we do better than CTC? Yes: using Transducer models.

Transducer models

The Transducer elegantly solves both problems associated with CTC, while retaining some of its advantages over attention models.

It solves Problem 1 by allowing multiple outputs for each input.
It solves Problem 2 by adding a predictor network and joiner⁵ network.

The predictor is autoregressive: it takes as input the previous outputs and produces features that can be used for predicting the next output, like a standard language model.

The joiner is a simple feedforward network that combines the encoder vector $f_t$ and predictor vector $g_u$ and outputs a softmax $h_{t,u}$ over all the labels, as well as a “null” output $\varnothing$.

Given an input sequence $\mathbf{x}$, generating an output sequence $\mathbf{y}$ can be done using a simple greedy search algorithm:

Start by setting $t := 1$, $u := 0$, and $\mathbf{y} :=$ an empty list.
Compute $f_t$ using $\mathbf{x}$ and $g_u$ using $\mathbf{y}$.
Compute $h_{t,u}$ using $f_t$ and $g_u$.
If the argmax of $h_{t,u}$ is a label, set $u := u + 1$, and output the label (append it to $\mathbf{y}$ and feed it back into the predictor).

If the argmax of $h_{t,u}$ is $\varnothing$, set $t := t + 1$ (in other words, just move to the next input timestep and output nothing).
If $t=T+1$, we’re done. Else, go back to step 2.

A couple cool things about Transducers to note here:

If the encoder is causal (i.e., we’re not using something like a bidirectional RNN), then the search can run in an online/streaming fashion, where we process each $x_t$ as soon as it arrives.
The predictor only has access to $\mathbf{y}$, and not $\mathbf{x}$—unlike the decoder in an attention model, which sees both $\mathbf{x}$ and $\mathbf{y}$. That means we can easily pre-train the predictor on text-only data, which there’s a lot more of than paired (speech, text) data.

Alignment

Given an $(\mathbf{x}, \mathbf{y})$ pair, the Transducer defines a set of possible monotonic alignments between $\mathbf{x}$ and $\mathbf{y}$. For example, consider an input sequence of length $T = 4$ and an output sequence (“CAT”) of length $U = 3$. We can illustrate the set of alignments using a graph⁶ like this:

Here’s one alignment: $\mathbf{z} = \varnothing, C, A, \varnothing, T, \varnothing, \varnothing$

Here’s another alignment: $\mathbf{z} = C, \varnothing, A, \varnothing, T, \varnothing, \varnothing$

We can calculate the probability of one of these alignments by multiplying together the values of each edge along the path:

$\mathbf{z} = \varnothing, C, A, \varnothing, T, \varnothing, \varnothing$
↓ $p(\mathbf{z} | \mathbf{x}) = h_{1,0}[\varnothing] \cdot h_{2,0}[C] \cdot h_{2,1}[A] \cdot h_{2,2}[\varnothing] \cdot h_{3,2}[T] \cdot h_{3,3}[\varnothing] \cdot h_{4,3}[\varnothing],$

where the value of an edge is the corresponding entry of $h_{t,u}$.

Training

How do we train the model? If we knew the true alignment⁷ $\mathbf{z}$, we could minimize the cross-entropy between $\mathbf{h}$ and $\mathbf{z}$, like a normal classifier. However, we usually don’t know the true alignment (and for some tasks, a “true” alignment might not even exist).

Instead, the Transducer defines $p(\mathbf{y}|\mathbf{x})$ as the sum of the probabilities of all possible alignments between $\mathbf{x}$ and $\mathbf{y}$. We train the model by minimizing the loss function $-\log p(\mathbf{y}|\mathbf{x})$.

There are usually too many possible alignments to compute the loss function by just adding them all up directly. To compute the sum efficiently, we compute the “forward variable” $\alpha_{t,u}$, for $1 \leq t \leq T$ and $0 \leq u \leq U$:

$$\begin{eqnarray*} \alpha_{t,u} = \alpha_{t-1,u} \cdot h_{t-1,u}[\varnothing] \\+ \alpha_{t,u-1} \cdot h_{t,u-1}[y_{u-1}] \end{eqnarray*}$$

We can visualize this computation as passing values along the edges of the alignment graph:

After we’ve computed $\alpha_{t,u}$ for every node in the alignment graph, we get $p(\mathbf{y}|\mathbf{x})$ using the forward variable at the last node of the graph:

$$\begin{eqnarray*} p(\mathbf{y}|\mathbf{x}) = \alpha_{T,U} \cdot h_{T,U}[\varnothing]\end{eqnarray*}$$

We need to do everything in the log domain, for the usual reasons. In the log domain, the computation becomes:

$$\begin{eqnarray*} \log \alpha_{t,u} = \text{logsumexp}([\log \alpha_{t-1,u} + \log h_{t-1,u}[\varnothing], \\ \log \alpha_{t,u-1} + \log h_{t,u-1}[y_{u-1}] ]) \end{eqnarray*}$$ $$\begin{eqnarray*} \log p(\mathbf{y}|\mathbf{x}) = \log \alpha_{T,U} + \log h_{T,U}[\varnothing]\end{eqnarray*}$$

Finally, to compute the gradient of the loss $-\log p(\mathbf{y}|\mathbf{x})$, there is a second algorithm that computes a backward variable $\beta_{t,u}$, using the same computation as $\alpha_{t,u}$, but in reverse, starting from the last node.

In the notebook, I provide a simple PyTorch implementation of the loss function that only writes out the forward computation and uses automatic differentiation to compute the gradient. This is a lot slower than a lower-level implementation, but easier to program and to read.

Memory usage

In general, Transducer models seem like a good idea. But here’s the catch (and possibly the unspoken reason that the Transducer never caught on until recently):

Suppose we have $T=1000$, $U=100$, $L=1000$ labels, and batch size $B=32$. Then to store $h_{t,u}$ for all $(t,u)$ to run the forward-backward algorithm, we need a tensor of size $B \times T \times U \times L = $ 3,200,000,000, or 12.8 GB if we’re using single-precision floats. And that’s just the output tensor: there’s also the hidden unit activations of the joiner network, which are of size $B \times T \times U \times d_{\text{joiner}}$.

So unless you are, ahem, a certain tech company in possession of TPUs with plentiful RAM (guess who’s been publishing the most Transducer papers!), you may need to find some way to reduce memory consumption during training—e.g., by pooling in the encoder to reduce $T$, or by using a small batch size $B$.

Ironically, this is only a problem during training; during inference, we only need a small amount of memory to store the current activations and hypotheses for $\mathbf{y}$.

Search

We saw earlier that you can predict $\mathbf{y}$ using a greedy search, always picking the top output of $h_{t,u}$. Better results can be obtained using a beam search instead, maintaining a list of multiple hypotheses for $\mathbf{y}$ and updating them at each input timestep.

The Transducer beam search algorithm can be found in the original paper—though it is somewhat gnarlier than the simple attention model beam search, and I confess I haven’t implemented it myself yet. (Check out the soon-to-be-released SpeechBrain toolkit for my colleagues’ implementation.)

Code

Finally, the Colab notebook for the Transducer can be found here. The notebook implements a Transducer model in PyTorch for a toy sequence transduction problem (filling in missing vowels in a sentence: “hll wrld” –> “hello world”), including the loss function, the greedy search, and a function for computing the probability of a single alignment. Enjoy!

Citation

If you found this tutorial helpful and would like to cite it, you can use the following BibTeX entry:

@misc{
	lugosch_2020, 
	title={Sequence-to-sequence learning with Transducers}, 
	url={https://lorenlugosch.github.io/posts/2020/11/transducer/}, 
	author={Lugosch, Loren}, 
	year={2020}, 
	month={Nov}
}

It always seems to take a few years between Alex Graves publishing a good idea and the research community fully recognizing it. There was an 8 year gap between CTC (2006) and Baidu’s Deep Speech (2014), and an 8 year gap between the Transducer (2012) and Google’s latest result (2020). This suggests a simple algorithm for achieving state-of-the-art results: select a paper written by Alex Graves from 8 years ago, and reimplement it using whatever advances in deep learning have been made since then. Maybe 2022 will be the year Neural Turing Machines really shine! ↩
There’s been some interesting work developing attention models that do not have these three issues, like monotonic chunkwise attention (MoChA). ↩
As do their older cousins, Hidden Markov Models, Graph Transformer Networks, and the more recent AutoSegCriterion. See Awni Hannun’s excellent introduction if you want to learn more about CTC. ↩
Alternately, you can use a very big and deep network like Jasper. Jasper was a CTC model proposed by NVIDIA researchers that, astonishingly, achieved nearly state-of-the-art performance using only a greedy search. If the model is big and deep, it can intelligently coordinate its outputs so as to not produce dumb predictions like “I eight food” instead of “I ate food”. Still, it seems to be more parameter-efficient to use a model that explicitly assumes that outputs are not independent, like attention models and Transducer models. ↩
In the original paper, there was no joiner; the encoder vector and predictor vector were simply added together. Graves and his co-authors added the joiner in a subsequent paper, finding that it reduced the number of deletion errors. ↩
If you’re familiar with the powerful gadgets known as finite state transducers (FSTs), you may recognize that the Transducer graph is a weighted FST, where an alignment forms the input labels, $\mathbf{y}$ forms the output labels, and the weight for each edge is dynamically generated by the joiner network. ↩
The true alignment of neural networks is known to be Chaotic Good. ↩

My research goals

2020-08-14T00:00:00-07:00

I wanted to clarify to myself and others what some of my research goals are, and why I’m working on certain problems. The hope is that putting this online for the world to see will help challenge me to keep focused and working towards those goals—sort of like telling your friends that you’re going to quit smoking, or something like that.

My broad long-term goal is to build reliable, competent domestic robots: in other words, robots that can help you around the house with tidying up, folding laundry, and the kind of things you might now ask Alexa/Siri/Google Home for, like playing music and setting timers.

In addition to the interesting technical challenge of it, domestic robots are just something I personally would love to have. They could also make life a lot better for elderly people and people who need long-term care—if you can’t pay for a caregiver, and if your loved ones aren’t able to take on the role of caregiver, a robot might be an affordable alternative.

An important aspect of my goal is: I don’t want these robots to rely on the Internet—I want their AI to live on the robot, offline. There’s a few reasons for this.

1) Privacy. Yes, yes, I know there’s a lot of people working on things like computation on encrypted data (“some of my best friends work on data privacy!”), but barring some big breakthroughs, I’d prefer to just cut the Gordian Knot and avoid sending my data to the cloud altogether.

2) Latency/reliability. Suppose a robot lives with an elderly person, and the person is about to slip and fall. The robot needs to detect this and quickly move to keep the person from getting hurt. We might use a neural network to map the robot’s camera feed to a sequence of physical actions to take. If we store that neural network in the cloud, and the Internet cuts out, the robot won’t be able to act.

3) The Internet shouldn’t even be necessary. If it’s not something that inherently requires the Internet, like checking the weather forecast or buying groceries, I shouldn’t have to use the Internet to do it. My brain takes up the space of about a hard drive and runs on 20 Watts—I don’t have to store it in the cloud and run it on a supercomputer. It’s annoying that I need the Internet just to tell my phone to set a reminder.

And this last one admittedly isn’t a very good reason, but I want to be honest with myself:

4) I want to own all my stuff. I’m annoyed that less and less of my stuff consists of physical things that I definitely own, and instead is more virtual things in the cloud that I maybe kind of own or maybe am just renting. I want to own my data, for instance; I don’t want to have to worry about where it lives and who can see it. (But maybe this is just a bit of nostalgia, or a bit of the tin-foil-hat-wearing libertarian’s instinct to convert his digital money into a pile of gold and hide it under the bed.)

industry: would you like to move all your code and data to a rich man's personal computer?
programmer: what no that sounds like a bad idea
industry: ok would you like to run everything in "the cloud"?
programmer: oh that sounds fluffy and nice yes please
— Computer Facts (@computerfact) November 8, 2019

I suspect that, to make really good and reliable domestic robots happen, solving the AI problems—true spoken language understanding, fine-grained motor control, long-term planning—will require extremely large neural networks. I’m talking hundreds of trillions of parameters. For reference, the current biggest neural net reported in the literature has only 600 billion parameters.

Why do I say that? There’s lots of evidence. GPT-3 has 175 billion parameters—there’s theoretical and empirical reasons to believe that big models really are necessary—and that’s just for language understanding/generation: our robot will also need motor control, audio processing, vision, and haptics. The way we do AI now, each of these modalities is handled separately; I think ultimately we will need the modalities to be handled together, so that your robot doesn’t think a unicorn might have 4 horns—and so our models will need even more capacity, for understanding how the different modalities fit together. And if you like taking inspiration from human brains: ours have 100 trillion synapses—assuming that 1 synapse = 1 weight, and assuming that Nature is efficient, then we’ll need 100 trillion parameters to make AI that can do the range of things human brains can do.

Running 100-trillion-parameter networks on a domestic robot is going to be hard. Just running a 600-billion-parameter network on a supercomputer was a significant engineering challenge for Google. Will Moore’s Law alone get us there? Could we just wait until 2040, when a Raspberry Pi has 1 PetaFLOPS of processing and 100 TB of storage? Or do we need to consider fundamentally different hardware designs?

No, I’m not talking about matrix multiplication accelerators, which have been optimized to death already. I’m talking about data storage—way less sexy, but ripe for neural-net-specific optimizations. Right now, a 100 TB drive costs $40,000. The alternative, connecting up dozens of cheaper, smaller drives, might not be energy-efficient or space-efficient enough for a robot. Could we make denser, cheaper storage by taking advantage of the fact that—unlike the hard drives for storing your OS and bank account information—neural nets can withstand a little noise?

Whatever the hardware for domestic robot AI looks like, conditional computation—that is, not using all 100 trillion parameters of the neural network every time the clock ticks—is certainly going to be a part of the solution. That’s the focus of my PhD right now. My current hunch is that the hard mixture-of-experts is a good starting point, but not the final word in conditional computation. Just as inductive biases like translational invariance can make computer vision easier to learn, there are simple inductive biases from computer architecture and psychology which I think can make the hard mixture-of-experts easier to train. The fun part of AI is figuring out how little inductive bias we can get away with—and pilfering that bias from human brains and other sources of inspiration when we do need it.

Predictive coding in machines and brains

2020-07-11T00:00:00-07:00

The name “predictive coding” has been applied to a number of engineering techniques and scientific theories. All these techniques and theories involve predicting future observations from past observations, but what exactly is meant by “coding” differs in each case. Here is a quick tour of some flavors of “predictive coding” and how they’re related.

What is “coding”?

In signal processing and related fields, the term “coding” generally means putting a signal into some format where the signal will be easier to handle for some task.

In any coding scheme, there is an encoder, which puts the input signal into the new format, and a decoder, which puts the encoded signal back into the original format (or as close as possible to the original). The “code” is the space of possible encoded signals. For example, the American Standard Code for Information Interchange (ASCII) consists of a set of 8-bit representations of commonly used characters. Your computer doesn’t understand characters, but it does understand bits, so we use a bit-based coding scheme to represent characters in a computer.

Some other common types of coding are:

Source coding (also known as “data compression”). The encoder compresses the input into a small bitstream; the decoder decompresses that bitstream to get back the input. Examples: arithmetic coding (used in JPEG), Huffman coding (used to compress neural networks).
Channel coding (also known as “error correction”). The encoder adds redundancy to protect a message from noise in the channel through which it will be transmitted; the decoder maps the noisy received signal back to the most likely original message, with the help of the added redundancy. Examples: low-density parity-check codes (used in your cell phone and your hard drive), Reed-Solomon codes (used in CDs, for those old enough to remember this technology).
Encryption. (Encryption could also be called “receiver coding” because it considers possible receivers, but nobody uses this terminology.) The encoder encrypts the message into a format such that only a receiver with the appropriate keys can open the message; the decoder decrypts the encrypted message into the readable original. Examples: RSA (used in HTTPS), SHA-1 (once used, but now broken).

Aside: The “source coding” and “channel coding” lingo come from Claude Shannon’s wonderful 1948 paper “A mathematical theory of communication”.

Predictive coding for data compression

OK, that’s coding in general. So what’s predictive coding?

The term “predictive coding” was coined in 1955 by Peter Elias. Specifically, what Elias proposed was a method called linear predictive coding (LPC) for communication systems.

In LPC, the next sample of a signal is predicted using a linear function of the previous $n$ samples. Then the error between the predicted sample and the actual sample is transmitted, along with the coefficients of the linear predictor. Predicting a sample from nearby previous samples works because in signals like speech, nearby samples are strongly correlated with each other.

The idea behind transmitting the error in LPC is that if we have a good predictor, the error will be small; thus it will require less bandwidth to transmit than the original signal. (So here “coding” specifically refers to “source coding”, or compression.)

LPC has had a long history of successes for audio compression. Speak ‘n’ Spell used LPC to store and synthesize speech sounds. The old Game Boy soundchip mostly used simple square wave beeps ‘n’ boops to make music, but sometimes used a special case of LPC known as DPCM to store certain sounds, like Pikachu’s voice in Pokémon Yellow. (See this video for a great overview of this and other old-school soundchips. Audio compression was crucial back when game cartridges had limited data storage.) The speech codec used in Skype uses LPC, combined with a bunch of other gadgets.

Predictive coding is also used in video compression, under the name “motion compensation”. Like adjacent audio samples, adjacent video frames are strongly correlated, and so can be predicted from each other. If you haven’t already, it’s good to take a moment to have your mind blown by H.264 video compression—the digital equivalent of shrinking a 3000-pound car to 0.4 pounds.

And of course, linear models are not the only way to do predictive coding; nonlinear models like neural networks can be used as well. A speech codec using WaveNet has been reported to get lower bitrate for the same quality as some traditional speech codecs.

Predictive coding for representation learning

Linear predictive coding and friends can be thought of as special cases of something called an autoregressive model. An autoregressive model is a model that cleverly splits a complicated probability distribution over sequences into a number of chunks that are easier to handle.

Let $\mathbf{x} = \{x_1, x_2, \dots\}$ denote our sequence of interest. In an autoregressive model, the joint distribution $p(\mathbf{x})$ is defined as

$$\begin{eqnarray*} p(\mathbf{x}) = \prod_t p(x_t|x_{t-1}, x_{t-2}, \dots) \end{eqnarray*}$$

The $p(x_t|x_{t-1}, x_{t-2}, \dots)$ term can be implemented in different ways. In something simple like an $n$-gram model for text, $p(x_t|x_{t-1}, x_{t-2}, \dots)$ is just a lookup table containing the probability of the next letter being $x_t$, given that the previous letters were $x_{t-1}, x_{t-2}, \dots$. Another way to implement $p(x_t|x_{t-1}, x_{t-2}, \dots)$ is to feed $x_{t-1}, x_{t-2}, \dots$ into a neural network, which outputs a feature vector $h$, and then predict $x_t$ using a linear model on top of $h$, like a softmax classifier (for discrete $x_t$) or a linear regression model (for real-valued $x_t$).

Such a model is called “autoregressive” because if you want to sample from the distribution it defines, you must feed back the model’s own outputs (= “auto”) to predict the next output (= “regressive”).

It turns out that if you train a neural network as an autoregressive model, the internal representations learned by the network will work really well for supervised downstream tasks. Work like ULMFiT and World Models showed that this trick can make neural nets a lot more data-efficient.

Notice that using the networks’ internal representations is somewhat different from what we did in LPC: whereas in LPC the outputs are the errors (because the purpose is compression), in these autoregressive feature extractors the outputs are the features (because the purpose is extracting discriminative features). So in either case, we are predicting future observations, but the “coding scheme” and purpose of predicting the future is different.

A note about audio. Autoregressive modeling with big neural nets operating on the raw audio signal works—WaveNet did this for generative modeling, and the representations could be used on a downstream task (speech recognition)—but it is expensive to do because audio signals are high-dimensional.

Two methods have been developed to make it easier to do autoregressive modeling for audio: contrastive predictive coding and autoregressive predictive coding.

The first method, contrastive predictive coding (CPC), works by first encoding the input signal into a much lower-dimensional sequence of feature vectors using a convolutional neural network, and then training an autoregressive model on top of this sequence. Since the encoder could simply learn to output all 0s to make the objective function as easy as possible to optimize, the autoregressive component is instead trained using a contrastive loss: it must guess whether a sample is actually the next sample in the sequence, or a fake, thus preventing the encoder from collapsing to a trivial representation. The technique also works well for other high-dimensional signals, like images (represented as a sequence of pixels). Note that strictly speaking CPC is not a true autoregressive model because we can’t draw samples from it.

Autoregressive predictive coding (APC) takes a slightly different approach. Instead of modeling the raw audio, it extracts low-dimensional frequency domain features, and then does plain old autoregressive modeling on top of those features. The disadvantage is that 1) maybe some low-level information in the original signal gets thrown out, and 2) now you need to hand-craft some feature extraction, since what works for audio will not necessarily work for other modalities. (Incidentally, I think “autoregressive predictive coding” is not a very good name, because WaveNet is already an “autoregressive” “predictive coding” model. Engineers are not that great at naming things. Oh well.)

A nice paper comparing CPC and APC was recently published at the ICML 2020 Workshop on Self-supervision in Audio and Speech—check it out here, if you’re interested in speech models.

The drawback of using these predictive coding models for representation learning is that the representations we get are “unidirectional”: that is, they extract information only from the past, and not the future, to represent the current input. That’s a problem because the future is often very informative for interpreting the present. If you think of a phrase like “milk the cow”, we know that “milk” is a verb, and not a noun, from the words that follow it.

One way to overcome this problem is to do autoregression in both directions, and concatenate the representations from the forward and backward models, as is done in ELMo. Alternately, models like BERT use a bidirectional context and minimize a contrastive or denoising loss instead. But for tasks in which observations need to be processed in “real-time”, as is usually the case in control problems, a unidirectional context makes more sense.

Predictive coding for computational efficiency

Another really neat thing forward predictive models can do is tell you roughly whether an observation is “difficult” or not. The idea is this: if an input is surprising—if, for example, your autoregressive model assigns low probability to it—it contains more information, and it is therefore probably worth more attention.

This observation suggests yet another use for predictive coding: we can allocate less computation to more predictable inputs, a type of “conditional computation”. Jürgen Schmidhuber’s “Neural Sequence Chunker” is an early instance of this idea, in which predictable inputs are ignored and not sent to a subsequent neural network, and (shameless plug!) recently I wrote a paper describing a slightly more general version of the idea.

While unsurprising inputs merit less computation, the inverse is not necessarily true: surprising inputs do not always merit more computation. Alex Graves in his Adaptive Computation Time paper gives an excellent example of unpredictable inputs—specifically, random ID numbers in Wikipedia metadata—which a neural network model does not bother to allocate more computation to, because the model realizes that throwing more processing time at the problem just won’t help. This implies that we can do better than just using predictability by learning when to use more or less computation. Still, predictability is a good inductive bias for conditional computation in neural nets—and it appears that human brains may use this inductive bias for similar purposes.

Predictive coding in the brain

Artificial neural networks in machine learning were inspired by scientific theories about the structure of the human brain. For predictive coding, it was the other way around: engineers (starting with Elias) developed predictive coding to solve certain problems in signal processing, and only afterwards did scientists realize that the brain might be doing something like what engineers had developed.

One such early inkling was described in Jeffrey Elman’s classic “Finding Structure in Time”. Elman trained a recurrent neural network to predict the next letter in a stream of letters. The streams were formed by taking sentences and removing the spaces between words. This is sort of analogous to the scenario of children learning language. Children are not told where the boundaries between words are; presumably they only hear a relatively unbroken stream of phonemes when adults speak.

What Elman found was that letters with high surprisal corresponded closely to the location of word boundaries. Predictive coding might therefore be one of the ingredients for language learning in humans: children could in theory infer word boundaries by noting which phonemes have high surprisal. (This method isn’t foolproof: my uncle reports that as a kid he thought “tractorworking” was a word because he so often heard “tractor” and “working” together, e.g. “there’s a tractor working in the field”.)

Another piece of evidence for predictive coding in humans comes from studies of reading time. van Schijndel and Linzen put it nicely: “One of the most robust findings in the reading literature is that more predictable words are read faster than less predictable words… Word predictability effects fit into a picture of human cognition in which humans constantly make predictions about upcoming events and test those predictions against their perceptual input.”

There is an even more general “predictive coding hypothesis” which claims that the brain does predictive coding at every level: sensory signals are predicted by the neurons that receive them, and the prediction errors become the input to other neurons, which also make predictions and errors, and so on. This hypothesis is somewhat controversial, though, as can be seen from the many responses to Andy Clark’s “Whatever next? Predictive brains, situated agents, and the future of cognitive science” (the responses go from pg. 24 onward—hear also Grace Lindsay’s podcast episode on predictive coding, which discusses this essay).

The future of predictive coding

Regardless of the extent to which it actually happens in human brains, predictive coding is a very powerful idea. As researchers from OpenAI put it recently, it is “a universal unsupervised learning algorithm”. Indeed, OpenAI’s GPT-3—which is trained solely using next-step-prediction—can do all kinds of things it was never explicitly taught to do, like word arithmetic and Tom Swifty puns.

I suspect that more and more AI systems will have something like a predictive coding component built in. Forward predictive models are already needed for things like model-based reinforcement learning, where a model of the environment is used to plan by simulating and optimizing over possible trajectories; so, why not take advantage of the rich representations learned by those predictive models, and use them as inputs to subsequent processing?

A contemplation of $\text{logsumexp}$

2020-06-30T00:00:00-07:00

$\text{logsumexp}$ is an interesting little function that shows up surprisingly often in machine learning. Join me in this post to shed some light on $\text{logsumexp}$: where it lives, how it behaves, and how to interpret it.

What is $\text{logsumexp}$?

Let $\mathbf{x} \in \mathbb{R}^n$. $\text{logsumexp}(\mathbf{x})$ is defined as:

$$\begin{eqnarray*} \text{logsumexp}(\mathbf{x}) = \text{log} \left( \sum_{i} \text{exp}(x_i) \right). \end{eqnarray*}$$

Numerically, $\text{logsumexp}$ is similar to $\text{max}$: in fact, it’s sometimes called the “smooth maximum” function. For example:

$$\begin{eqnarray*} \text{max}([1,2,3]) = 3 \end{eqnarray*}$$ $$\begin{eqnarray*} \text{logsumexp}([1,2,3]) = 3.4076 \end{eqnarray*}$$

Examples of $\text{logsumexp}$

Here are some places in machine learning where $\text{logsumexp}$ is used.

Softmax classifiers

In a softmax classifier, the likelihood of label $i$ is defined as:

$$\begin{eqnarray*} p_{\theta}(i | \mathbf{l}) = \text{softmax}(\mathbf{l})_i, \end{eqnarray*}$$

where $\mathbf{l}$ is the vector of logits (unnormalized scores for each label).

Softmax classifiers are trained by minimizing the negative log-likelihood loss:

$$\begin{align*} -\text{log } p_{\theta}(i | \mathbf{l}) &= -\text{log} \left( \text{softmax}(\mathbf{l})_i \right) \\ &= -\text{log} \left( \text{exp}(l_i) / \sum_j \text{exp}(l_j) \right) \\ &= -\text{log} \left( \text{exp}(l_i)\right) +\text{log} \left(\sum_j \text{exp}(l_j) \right) \\ &= -l_i + \text{logsumexp}(\mathbf{l}). \end{align*}$$

Our friend $\text{logsumexp}$ appears in the last line.

Global pooling

For sequence classification tasks, it is usually necessary to map a variable length sequence of feature vectors to a single feature vector to be able to use something like a softmax classifier. To obtain a single vector, global pooling operations like max pooling or mean pooling can be used. Another aggregation method, which is less commonly used but has some of the advantages of both mean and max pooling, is $\text{logsumexp}$ pooling. (See this paper for an example.)

Latent alignment models

In latent alignment models, like connectionist temporal classification (CTC), dynamic programming is used to add up the probability of all possible alignments of an input sequence $\mathbf{x}$ to an output sequence $\mathbf{y}$ to train the model.

The dynamic programming algorithm in CTC uses the following recursion:

$$\begin{eqnarray*} \alpha_{s,t} = (\alpha_{s,t-1} + \alpha_{s-1,t-1} + \alpha_{s-2,t-1}) \cdot p_{\theta}(y_s | x_t)\end{eqnarray*}$$

This algorithm multiplies a long chain of probabilities, and so will underflow when aligning long sequences (like speech). Instead, we can run the algorithm in the log domain, in which case the recursion becomes:

$$\begin{align*} \text{log}(\alpha_{s,t}) &= \text{log}(\alpha_{s,t-1} + \alpha_{s-1,t-1} + \alpha_{s-2,t-1}) + \text{log } p_{\theta}(y_s | x_t)\\ &= \text{log}(\text{exp}(\text{log }\alpha_{s,t-1}) + \text{exp}(\text{log }\alpha_{s-1,t-1}) + \text{exp}(\text{log }\alpha_{s-2,t-1})) + \text{log } p_{\theta}(y_s | x_t)\\ &= \text{logsumexp}([\text{log }\alpha_{s,t-1},\text{log }\alpha_{s-1,t-1}, \text{log }\alpha_{s-2,t-1}]) + \text{log } p_{\theta}(y_s | x_t)\\ \end{align*}$$

Similar recursions using $\text{logsumexp}$ can be derived for the forward-backward algorithm used in Hidden Markov Models and Transducer models.

Fun fact: if we replace the $\text{logsumexp}$ with $\text{max}$, we get the Viterbi algorithm, which gives us the score of the single most likely alignment (and if we backpropagate, the alignment itself).

Some properties of $\text{logsumexp}$

$\text{logsumexp}$ has some useful properties. It is:

Convex. If you can pose your machine learning problem as a convex optimization problem, you can solve it quickly and reliably.
Differentiable everywhere. This is nice to have if your optimization algorithm is picky and doesn’t like functions with non-differentiable points, like $\text{max}$. For those picky algorithms, we can approximate $\text{max}$ using $\text{logsumexp}$.
Associative. So:

$$\begin{align*} \text{logsumexp}([a,b,c,d]) &= \text{logsumexp}([ \\ & \text{logsumexp}([a,b]), \\ & \text{logsumexp}([c,d]) \\ ]). \\ \end{align*}$$

(Hence, it can be computed in just $\text{log}_2(n)$ timesteps using a parallel reduction, where $n$ is the length of the vector we’re $\text{logsumexp}$ing.)
Close to $\text{max}$. In what sense “close”? It is bounded as follows:

$$\begin{eqnarray*} \text{max}(\mathbf{x}) \leq \text{logsumexp}(\mathbf{x}) \leq \text{max}(\mathbf{x}) + \text{log}(n) \end{eqnarray*}$$

The $\text{logsumexp}$ trick

Computing $\text{log} \left( \sum_{i} \text{exp}(x_i) \right)$ directly is numerically unstable because of the $\text{exp}$. Try the following in numpy:

x = np.array([7000,8000,9000])
np.log(np.sum(np.exp(x)))

From the approximation $\text{max}(\mathbf{x}) \approx \text{logsumexp}(\mathbf{x})$, we know that the result should be a little over 9000—but if you run this code, the result will be infinity because of overflow.

Instead, to compute $\text{logsumexp}$, use the following trick:

$$\begin{eqnarray*} \text{logsumexp}(\mathbf{x}) = \text{log} \left( \sum_{i} \text{exp}(x_i - \text{max}(\mathbf{x})) \right) + \text{max}(\mathbf{x}). \end{eqnarray*}$$

(See this post for the proof that the trick works.)

Now we won’t get an overflow because we’re taking the $\text{exp}$ of $[-2000,-1000,0]$ instead of $[7000,8000,9000]$. If we now run this instead:

x = np.array([7000,8000,9000])
np.log(np.sum(np.exp(x - x.max()))) + x.max()

we’ll get what we expect.

Takeaways

$\text{logsumexp}$ is everywhere!
You can make certain dense equations easier to digest by identifying instances of $\text{logsumexp}$ and mentally replacing them with $\text{max}$.

For example, from the discussion of softmax classifiers above, you now know that the loss for a classifier is just the difference between the maximum of the logits (roughly) and the logit for the right answer!
When computing $\text{logsumexp}$, use the “logsumexp trick”. (Or just check if your library already has a numerically stable $\text{logsumexp}$ function, as PyTorch does.)

Notebook: Fun with Hidden Markov Models

2020-01-28T00:00:00-08:00

I’ve written a notebook introducing Hidden Markov Models (HMMs) with a PyTorch implementation of the forward algorithm, the Viterbi algorithm, and training a model on a text dataset—check it out here!

An introduction to sequence-to-sequence learning

2019-02-19T00:00:00-08:00

Many interesting problems in artificial intelligence can be described in the following way:

Map a sequence of inputs $\mathbf{x}$ to the correct sequence of outputs $\mathbf{y}$.

Speech recognition is one example: the goal is to map an audio signal $\mathbf{x}$ (a sequence of real-valued audio samples) to the correct text transcript $\mathbf{y}$ (a sequence of letters). Other examples are machine translation, image captioning, and speech synthesis.

This post is a tutorial introduction to sequence-to-sequence learning, a method for using neural networks to solve these “sequence transduction” problems. In this post, I will:

show you why these problems are interesting and challenging
give a detailed description of sequence-to-sequence learning—or “seq2seq”, as the cool kids call it
walk through an example seq2seq application (with PyTorch code)

The only background you need to read this is basic knowledge of how to train and use neural networks for classification problems.

The problem

Let’s start by defining the general problem of sequence transduction a little more carefully.

Let $\mathbf{x} = \{x_1, x_2, \dots x_T\}$ be the input sequence and $\mathbf{y} = \{y_1, y_2, \dots, y_U\}$ be the output sequence, where $x_t \in \mathcal{S}_x$, $y_u \in \mathcal{S}_y$, and $\mathcal{S}_x$ and $\mathcal{S}_y$ are the sets of possible things that each $x_t$ and $y_u$ can be, respectively.

In speech recognition, $\mathcal{S}_x$ would be the set of real numbers, $\mathcal{S}_y$ would be the set of letters, $T$ might be on the order of thousands, and $U$ might be on the order of tens.

We’ll assume that the inputs and outputs are random variables, and the value of $T$ and $U$ may vary between examples of $\mathbf{x}$ and $\mathbf{y}$—for example, not all sentences have the same length. The fact that $U$ may vary is part of what makes these problems challenging: we need to guess how long the output should be, and often this can’t simply be inferred from the input length.

The goal is to find the best function $f(\mathbf{x})$ for mapping $\mathbf{x}$ to $\mathbf{y}$. But what does “best” mean here?

For a classification problem, the “best” $f(\mathbf{x})$ is the one with the highest accuracy, i.e. the lowest probability of guessing an output $\mathbf{\hat{y}}$ which is not equal to the true $\mathbf{y}$.

Similarly, we can use “accuracy” as a performance measure if the output is a sequence: if the guess $\mathbf{\hat{y}}$ is at all different from the true output $\mathbf{y}$, the output is incorrect. That is, if $\mathbf{y} = \text{“hello”}$ and $\mathbf{\hat{y}} = \text{“helo”}$, then $\mathbf{\hat{y}}$ is incorrect. This is also called the “$0$-$1$ loss”: $0$ if $\mathbf{\hat{y}} = \mathbf{y}$, $1$ otherwise.

The simple $0$-$1$ loss is not a realistic performance measure: an output like $\text{“lskdjfl”}$ should really be regarded as less accurate than $\text{“helo”}$ if the true output is $\text{“hello”}$.

In practice, we probably care more about some other performance measure, such as the word error rate (speech recognition) or the BLEU score (machine translation, image captioning). But the $0$-$1$ loss is often a good approximation or surrogate for other performance measures, and it doesn’t require any domain-specific knowledge to apply, so let’s assume that the $0$-$1$ loss is what we’re trying to optimize.

The ideal solution

Suppose that we have access to a magical genie who can tell us $p(\mathbf{y}|\mathbf{x})$, the probability that the correct output sequence is $\mathbf{y}$ given that the input is $\mathbf{x}$, for any $\mathbf{x}$ and $\mathbf{y}$. In that case, what would be the best function $f(\mathbf{x})$ to use?

Since we’re trying to minimize the probability of making an error (guessing a $\mathbf{\hat{y}}$ which is not equal to the correct $\mathbf{y}$), the following function is the best choice:

$$\begin{eqnarray*}\mathbf{\hat{y}} = f(\mathbf{x}) = \underset{\mathbf{y}}{\text{argmax }} p(\mathbf{y}|\mathbf{x})\end{eqnarray*}$$

Alas! We usually don’t have a magical genie who can tell us $p(\mathbf{y}|\mathbf{x})$. What we can do instead is fit a model $p_{\theta}(\mathbf{y}|\mathbf{x})$ to some training data, and if we’re lucky, that model will be close to the true distribution $p(\mathbf{y}|\mathbf{x})$.

Modeling $p(\mathbf{y})$

Before we consider implementing the model $p_{\theta}(\mathbf{y}|\mathbf{x})$, let’s start with a slightly easier but closely related problem: implementing a model $p_{\theta}(\mathbf{y})$ that is close to $p(\mathbf{y})$.

What $p(\mathbf{y})$ represents is the probability of observing a particular output sequence $\mathbf{y}$, independent of whatever the input sequence might be.

Intuitively, what does this mean? Well, consider the two following sequences of words:

$$\begin{eqnarray*}\mathbf{y}_1 = \text{I ate food}\end{eqnarray*}$$ $$\begin{eqnarray*}\mathbf{y}_2 = \text{I eight food}\end{eqnarray*}$$

The probability of observing $\mathbf{y}_1$ should be higher than the probability of observing $\mathbf{y}_2$, since $\mathbf{y}_1$ is a meaningful English sentence, and $\mathbf{y}_2$ is nonsensical. This fact could be useful in speech recognition to figure out which of the two phrases the person actually said, since both of these phrases sound the same if you say them out loud.

Likewise, consider the two following sequences of letters:

$$\begin{eqnarray*}\mathbf{y}_1 = \text{florpy}\end{eqnarray*}$$ $$\begin{eqnarray*}\mathbf{y}_2 = \text{fhqhwgads}\end{eqnarray*}$$

Although these two sequences are both not real English words, the first one $\mathbf{y}_1$ really could be an English word, whereas the second one $\mathbf{y}_2$ just looks like someone banging on a keyboard. If we were to train a model $p_{\theta}(\mathbf{y})$ on English text, we would probably find in this case that $p_{\theta}(\mathbf{y}_1) > p_{\theta}(\mathbf{y}_2)$.

To assign a probability to any sequence, we could just use a gigantic lookup table, with one entry for every possible $\mathbf{y}$. The problem is there are usually too many possible sequences for this to be feasible—in fact, there may be infinitely many possible sequences.

Instead, let’s define a model that computes the probability of a sequence by dividing it into a number of simpler probabilities using the chain rule of probability.

The chain rule of probability says that, for two random variables A and B, the probability of A and B is equal to the probability of A given B, multiplied by the probability of B:

$$\begin{eqnarray*} p(A,B) = p(A|B) \cdot p(B) \end{eqnarray*}$$

That’s one way to factorize $p(A,B)$—we could also factorize it like this:

$$\begin{eqnarray*} p(A,B) = p(B|A) \cdot p(A) \end{eqnarray*}$$

Moreover, you can apply the chain rule for as many random variables as you want:

$$\begin{align*} p(A,B,C) &= p(A|B,C) \cdot p(B,C) \\ &= p(A|B,C) \cdot p(B|C) \cdot p(C) \end{align*}$$

Remember that our sequence $\mathbf{y}$ is a collection of random variables $y_1, y_2, y_3, \dots, y_U$. So let’s use the chain rule to write out $p_{\theta}(\mathbf{y})$ as the product of the probability of each of these variables, given the previous ones:

$$\begin{eqnarray*} p_{\theta}(\mathbf{y}) &=& p_{\theta}(y_1, y_2, y_3, \dots, y_U)\\ &=& \overset{U}{\underset{u=1}{\prod}} p_{\theta}(y_u|y_{u-1},y_{u-2},\dots,y_{1}) \end{eqnarray*}$$

This $p_{\theta}(y_u|y_{u-1},y_{u-2},\dots,y_{1})$ term means “the probability of the next element, given what came before”. A model that implements $p_{\theta}(y_u|y_{u-1},y_{u-2},\dots,y_{1})$ is sometimes called a “next step predictor” or “language model”. When you are texting someone, and your phone suggests the next word for you as you are typing, it uses a model like this.

We can implement this “next step” probability $p_{\theta}(y_u|y_{u-1},y_{u-2},\dots,y_{1})$ using a neural network¹. The neural network takes as input the previous $y_u$’s and predicts the next $y_u$. This is shown in the figure below:

For example, if $\mathcal{S}_y$ is the 26 letters of the alphabet, then $y_{u-1},y_{u-2},\dots,y_{1}$ would be the previous letters in the sequence, and the neural network would have a softmax output of size 26 representing the probability of the next letter given the previous letters.

A common choice for the neural network is a recurrent neural network (RNN), which is what is shown in the diagram here.

An RNN, if you are not familiar, is a neural network with memory. At each timestep, the RNN takes in an input vector $i$ and its current state vector $h$, and outputs an updated state vector $h := f(i, h)$. Through the state $h$, the RNN can remember what inputs it has seen so far ($i_1, i_2, \dots$). The RNN can assign probabilities to different classes based on its state using a softmax classifier: $p(\text{class }c) = \text{softmax}(Wh + b)_c$

To get the probability of the entire sequence, the chain rule tells us to multiply each $p_{\theta}(y_u|y_{u-1},y_{u-2},\dots,y_{1})$ term together. In other words, just multiply the softmax outputs together.

For example, let’s say that $\mathcal{S}_y$ is the three letters $\{a,b,c\}$, and we want to calculate the probability of the sequence $ba$. In this case, the neural network would have a softmax output of size 3. If we have:

$p_{\theta}(y_1) = [0.3, 0.4, 0.3]$, and
$p_{\theta}(y_2|y_1=b) = [0.5, 0.3, 0.2]$,

then $p_{\theta}(ba) = p(a|b) p(b) = 0.5 \cdot 0.4 = 0.2$.

One final ingredient we need for a complete model $p_{\theta}(\mathbf{y})$ is a special element called the “end-of-sequence” element. Each sequence needs to end with “end-of-sequence”. Predicting “end-of-sequence” along with all the other elements of $\mathcal{S}_y$ allows the model to implicitly predict the length of the sequence.

Modeling $p(\mathbf{y}|\mathbf{x})$

We’re not just interested in $p(\mathbf{y})$: what we really want to model is $p(\mathbf{y}|\mathbf{x})$, the probability of an output sequence given a particular input sequence.

The simplest way to condition the output on the input is to split the model into an encoder RNN and a decoder RNN, where the encoder RNN converts the input sequence into a single vector that is used to “program” the decoder RNN.²

The encoder RNN reads the input sequence element-by-element. As the encoder reads each input element, it updates its state. The final state of the encoder after it has read the entire input sequence represents a fixed-length encoding of the input sequence. This encoding then becomes the initial state of the decoder RNN:

Hopefully, the encoding (a single vector) will contain all the information in the input needed for the decoder to accurately model the correct output. Then we just proceed to calculate $p_{\theta}(\mathbf{y}|\mathbf{x})$ as we would calculate $p_{\theta}(\mathbf{y})$, by multiplying the probabilities of all the $y_u$’s together:

$$\begin{eqnarray*} p_{\theta}(\mathbf{y}|\mathbf{x}) &=& p_{\theta}(y_1, y_2, y_3, \dots, y_U|\mathbf{x})\\ &=& \overset{U}{\underset{u=1}{\prod}} p_{\theta}(y_u|y_{u-1},y_{u-2},\dots,y_{1},\mathbf{x}) \end{eqnarray*}$$

Here’s a question you might ask, looking at the diagram of the model: Given that an RNN has state, and can remember what it has already outputted, why we do we need to feed $y_1, y_2, \dots$ into the decoder to compute the probability of the next output $y_u$?

There are two good reasons.

First, if we don’t make the output explicitly conditioned on the previous outputs, we are implicitly saying that the outputs are independent, which may not be a good assumption. Consider transcribing a recording of someone saying “triple A”. There are two valid transcriptions: $\text{AAA}$ and $\text{triple A}$. If the first output $y_1$ is $\text{A}$, then we can be certain that the second output $y_2$ will be $\text{A}$.³

Second, feeding in previous outputs allows us to use a feedforward model (which does not have state) for the decoder. A feedforward model can be much faster to train, since each $p_{\theta}(y_u|y_{u-1},y_{u-2},\dots,y_{1},\mathbf{x})$ term can be computed in parallel.⁴

Although we’ve been writing $p_{\theta}(\mathbf{y}|\mathbf{x})$, it’s usually better to work with the log probability $\text{log } p_{\theta}(\mathbf{y}|\mathbf{x})$ instead, for a few reasons. First, it is often easier to work with sums than it is to work with products. Recall that $\text{log }(a \cdot b) = \text{log }a + \text{log }b$. Thus, if you take the log of $p_{\theta}(\mathbf{y}|\mathbf{x})$, it becomes a sum instead of a product:

$$\begin{eqnarray*} \text{log } p_{\theta}(\mathbf{y}|\mathbf{x}) &=& \text{log } p_{\theta}(y_1, y_2, y_3, \dots, y_U|\mathbf{x})\\ &=& \overset{U}{\underset{u=1}{\sum}} \text{log } p_{\theta}(y_u|y_{u-1},y_{u-2},\dots,y_{1},\mathbf{x}) \end{eqnarray*}$$

This prevents multiplying together a bunch of numbers smaller than 1, which could cause an underflow. It also makes training the model using gradient descent easier, since minimizing a sum of terms is easier than minimizing a product of terms.⁵ And finally, since many useful probability distributions have an $\text{exp}(\cdot)$ term (including softmax, Gaussian, and Poisson), and $\text{log }\text{exp}(a) = a$, taking the log may transform $p_{\theta}$ into a simpler form.

Learning

Now that we have a way of computing $p_{\theta}(\mathbf{y}|\mathbf{x})$ using a neural network, we need to train the model, i.e. find $\theta$ such that the model distribution $p_{\theta}(\mathbf{y}|\mathbf{x})$ matches the true distribution $p(\mathbf{y}|\mathbf{x})$. As usual with neural networks, we can do that by minimizing a loss function.

Many seq2seq papers don’t explicitly write out a loss function, though. Instead, they will just say something like “we use maximum likelihood” or “we minimize the negative log likelihood”. Here, we will see how this translates to a particular loss function that you can implement.

Suppose that you have a training set $\mathcal{T}$ composed of ($\mathbf{x}^i, \mathbf{y}^i$) pairs. If the training examples are considered to be fixed, and you think of $p_{\theta}(\mathbf{y}|\mathbf{x})$ as a function of the parameters $\theta$, then we call $p_{\theta}(\mathbf{y}^i|\mathbf{x}^i)$ the “likelihood” of the training example ($\mathbf{x}^i, \mathbf{y}^i$). In maximum likelihood estimation, the model parameters $\theta$ are learned by maximizing the likelihood of the entire training set, $L_{\theta}(\mathcal{T})$, which is the product of the likelihoods of all the training examples:

$$\begin{eqnarray*}L_{\theta}(\mathcal{T}) = \overset{|\mathcal{T}|}{\underset{i=1}{\prod}} p_{\theta}(\mathbf{y}^i|\mathbf{x}^i)\end{eqnarray*}$$

If these ($\mathbf{x}^i, \mathbf{y}^i$) samples are independent and identically distributed (i.i.d.), then as the number of samples increases, maximum likelihood estimation gives you a model that is closer and closer to the true distribution.

As mentioned earlier, it’s easier to maximize a sum than it is to maximize a product, so we take the log, and maximize that instead. (The log function is monotonic—that is, $\text{log }a > \text{log }b$ implies that $a > b$—so maximizing the log likelihood is equivalent to maximizing the likelihood.) Also, in machine learning it’s often more natural to think of minimizing a loss function, so we minimize the negative log likelihood:

$$\begin{eqnarray*}-\text{log } L_{\theta}(\mathcal{T}) = -\overset{|\mathcal{T}|}{\underset{i=1}{\sum}} \text{log } p_{\theta}(\mathbf{y}^i|\mathbf{x}^i)\end{eqnarray*}$$

If we expand the summed term, we get:

$$\begin{eqnarray*}-\text{log } L_{\theta}(\mathcal{T}) = -\overset{|\mathcal{T}|}{\underset{i=1}{\sum}} \overset{U^i}{\underset{u=1}{\sum}} \text{log } p_{\theta}(y_{u}^i|y_{u-1}^i,y_{u-2}^i,\dots,y_{1}^i,\mathbf{x}^i)\end{eqnarray*}$$

In other words, for every example in the dataset, we sum up the negative log probability of the correct output at each timestep, given the previous correct outputs.

Feeding the previous correct outputs into the model during training, as opposed to the model’s own predictions, is called “teacher forcing”. A long time ago (in deep learning years, which are like dog years), it was thought that teacher forcing is bad, and you should sometimes sample previous outputs from the model’s output distribution.⁶ Nowadays, this is less common, and big parallelizable seq2seq models like the Transformer⁷ rely on teacher forcing to go fast. Also, the name “teacher forcing” makes it sound like a hack, but really it’s the right way to apply maximum likelihood!

To minimize the negative log likelihood loss, you can use stochastic gradient descent (SGD), just like in a regular classification problem.

Inference

So far, we’ve described how you can use a neural network to compute $p_{\theta}(\mathbf{y}|\mathbf{x})$, and how to train the model using maximum likelihood.

The question remains: How do we generate an output? That is, how do we find $\underset{\mathbf{y}}{\text{argmax }} p_{\theta}(\mathbf{y}|\mathbf{x})$, given a new $\mathbf{x}$?

The brute force solution is an exhaustive search: just compute $p_{\theta}(\mathbf{y}|\mathbf{x})$ for every possible $\mathbf{y}$ and pick the $\mathbf{y}$ with the highest probability. That’s what exactly what we do for classification problems.

However, unlike a typical classification problem, where you might have a thousand classes, in any practical sequence prediction problem there will be astronomically many possible output sequences, so an exhaustive search is infeasible.

That means we need a new ingredient: an efficient search algorithm. The goal of the search is to approximately find $\mathbf{y}^* \approx \underset{\mathbf{y}}{\text{argmax }} p_{\theta}(\mathbf{y}|\mathbf{x})$.

(Note: for historical reasons⁸, the search process is also often called “decoding”. I’m not a big fan of this terminology because “decoding” already means several other things in machine learning.)

We will consider two search algorithms:

greedy search
beam search

1) Greedy search. A greedy search works as follows: at each step, pick the top output of the network, and feed this output back into the network. In other words, for each timestep $u$, pick $y_u^* = \underset{y_u}{\text{argmax }} p_{\theta}(y_u | y_{u-1}^*, y_{u-2}^*, \dots, y_{1}^*, \mathbf{x})$.

The search can continue until an “end-of-sequence” is predicted, or until a maximum number of steps has occurred. An example of a greedy search is shown in the diagram below:

Because inference requires making a prediction, and feeding it back in to make the next prediction, this type of model is called “autoregressive” (“auto”=”self”, “regress”=”predict”).

2) Beam search. Greedy searching is fast, but you can show that it will not always find the most likely output sequence. As an example, suppose that:

$p_{\theta}(a|\mathbf{x}) = 0.4$
$p_{\theta}(b|\mathbf{x}) = 0.6$

and

$p_{\theta}(aa|\mathbf{x}) = 0.4$
$p_{\theta}(ab|\mathbf{x}) = 0.0$
$p_{\theta}(ba|\mathbf{x}) = 0.35$
$p_{\theta}(bb|\mathbf{x}) = 0.25$

Here, a greedy search would pick $b$, then $a$, and thus return $ba$, which has probability $0.35$. But the most likely sequence is actually $aa$, which has a probability of $0.4$.

We can get better results if we delay decisions about keeping a particular output until we have considered some future outputs. Beam search is one way of doing this.

In a beam search, we maintain a list (“beam”) of $B$ likely sequences (“hypotheses”). At each step, for each hypothesis, we compute the top $B$ outputs, and append them to the hypothesis. Now we have $B^2$ hypotheses. Of these, we prune the beam down to the top $B$, and then we continue to the next step.

An example of a beam search with $B=3$ is shown below.⁹

The algorithm returns the top $B$ hypotheses found. If your goal is just to estimate $\mathbf{y}$, you would just keep the top hypothesis, but for some applications it may also be useful to keep the rest of the hypotheses.

Notice that if $B = 1$, the beam search is equivalent to a greedy search. Also, if $B = \infty$, the beam search becomes an exhaustive search.

Beam search, too, is not guaranteed to find the most likely output sequence, but the wider you make the beam, the smaller the chance of a search error.¹⁰ The tradeoff is that you must re-run the neural network $B$ times, since you need to feed outputs back in.

Attention

We now have a complete method for doing sequence-to-sequence learning.

Unfortunately, if you apply the method exactly as described above—using an encoder RNN to map the input to a fixed-length vector consumed by the decoder RNN—it will not work for long sequences. The problem is that it is difficult to compress the entire input sequence into a single fixed-length vector.¹¹

Attention¹² is a mechanism that removes this fixed-length bottleneck. With attention, the decoder does not rely on a single vector to represent the input; instead, at every decoding step, it “looks” at a different part of the input using a weighted sum.

A more detailed introduction to the various forms of attention can be found here.

A toy task

Let’s use a toy task to test out our method. Consider the following string:

“Mst ppl hv lttl dffclty rdng ths sntnc”

This string is an example¹³ of how natural language is highly redundant or predictable: it’s generated by removing all the vowels from a normal sentence, and you can still understand the meaning.

If human intelligence can infer the missing vowels, then maybe artificial intelligence can as well! We will train a sequence-to-sequence model to take as input a vowel-less sentence (“Mst ppl”) and output the sentence with the correct vowels re-inserted (“Most people”).

This toy task is a bit easier to work with than a task like speech recognition or translation, in which you need lots of labelled data and lots of tricks to get something to work. A tutorial which covers the actual useful task of translating from French to English using PyTorch can be found here.

Running the experiment

We can easily generate a dataset for the task of inferring missing vowels using existing text. I used the text of “War and Peace”¹⁴: the input sequences are lines from the text with all the vowels removed, and the target output sequences are just the original lines. I also added the Penn Treebank (PTB) dataset, a commonly used dataset of news articles for language modeling experiments, to give the training data a little variety.

To run the code for yourself, or to train a model for filling in missing vowels on a new dataset, the code and a pre-trained model for this experiment can be found here.

We will first try training the simple encoder-decoder described above, without an attention mechanism. Here’s the result when we run a new sentence through the model:

input: Mst ppl hv lttl dffclty rdng ths sntnc.
truth: Most people have little difficulty reading this sentence.
guess: Mostov played with Prince Andrew and strengthers and

Oh dear! The search starts off strong—it correctly outputs “Most”—but then it gets distracted and tries to fill in the name “Rostov” (the name of a character in “War and Peace”). The next word, “played”, at least starts with the right letter, the “p” in “people”, but after that, the output really goes off the rails.

If we let the simple encoder-decoder model train a lot longer, the results get a bit better, but we still find bizarre mistakes like this:

input: th dy bfr, nmly, tht th cmmndr-n-chf
truth: the day before, namely, that the commander-in-chief
guess: the day before, manling that the mimicinacying Freemason

So let’s add in that fancy attention mechanism I mentioned and see if that helps:

input: Mst ppl hv lttl dffclty rdng ths sntnc.
truth: Most people have little difficulty reading this sentence.
guess: Most people have little difficulty riding this sentence.

Much better! But still not completely correct.

Let’s look at the beam (the $B$ hypotheses found by the beam search) and the beam scores (the hypotheses’ log probabilities):

Most people have little difficulty riding this sentence.   | -2.98
Most people have little difficulty reading this sentence.  | -3.28
Most people have little difficulty roading this sentence.  | -3.79
Most people have little difficulty riding those sentence.  | -3.81
Most people have little difficulty reading those sentence. | -4.11
Most people have little difficulty riding these sentence.  | -4.16
Most people have little difficulty reading these sentence. | -4.45
Most people have little difficulty roading those sentence. | -4.60

The correct answer does in fact appear in the beam (the 2nd hypothesis), but the model incorrectly assigns a higher probability to the hypothesis with “riding” instead of “reading”. Maybe with more/better training data this error would not occur, since “reading this sentence” ought to be a lot more probable in the training data than “riding this sentence”.

Notice another peculiar aspect of the beam: it is ordered (roughly) from shortest to longest. Autoregressive models are biased towards shorter output sequences!

Why? At each timestep, to compute the probability of a sequence, we multiply it (or add it, in the log domain) by the probability of the next output, which is always less than 1 (less than 0, in the log domain), so the probability of the complete sequence keeps getting smaller. In fact, the only thing keeping the search from producing outputs of length 0 is the fact that the model needs to predict the “end-of-sequence” token, and from the training data the model learns to assign low probability to “end-of-sequence” until it makes sense. It’s easy for the model to learn that a sentence like “The.” is very unlikely, but comparing two plausible sequences of roughly the same length seems to be harder.

How do people deal with the short-sequence bias in practice? Google Translate uses a variation¹⁵ of beam search in which the log probability is divided by a “length penalty” (“lp”), with a hyperparameter $\alpha$, that is computed as follows:

$$\begin{eqnarray*} \text{lp}(\mathbf{y}) = \frac{(5 + |\mathbf{y}|)^{\alpha}}{(5 + 1)^{\alpha}}\end{eqnarray*}$$

Yikes. How many TPU hours did they burn finding that formula? I hope we find a better way to mitigate the short-sequence bias!

The End

If you have any questions or if you find something wrong with this tutorial, please let me know.

Check out the code and try it out! It’s fun to feed the model random inputs, like your name, and see what stuff it comes up with trying to fill in the gaps. The code is also written in such a way that it should not be too hard to adapt it to a new task.

Thanks to Christoph Conrads and Mirco Ravanelli for their feedback on the draft of this post.

See: Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin, “A Neural Probabilistic Language Model”, Journal of Machine Learning Research 3 (2003), 1137–1155. ↩
Like many great ideas, the encoder-decoder model was invented independently and simultaneously by multiple groups. The paper that is usually cited, which invented the name “sequence-to-sequence learning”, is this one: Ilya Sutskever, Oriol Vinyals, and Quoc V. Le, “Sequence to sequence learning with neural networks”, NeurIPS 2014. ↩
The “triple A” example comes from: William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition”, ICASSP 2016. ↩
For an example of this, see: Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. “Convolutional sequence to sequence learning”, ICML 2017. ↩
Why? Consider minimizing $y_{sum} = x_1 + x_2$ and $y_{prod} = x_1 \cdot x_2$. The derivative $\frac{dy_{sum}}{dx_1}$ is just $1$ (independent of $x_2$), whereas the derivative $\frac{dy_{prod}}{dx_1}$ is equal to $x_2$. If $x_2$ is very small, $x_1$ will be “held back” from changing easily if you try to minimize $y_{prod}$—imagine running a race if you are tied to a slow person by a rope. ↩
See: Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer, “Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks”, NeurIPS 2015. ↩
See: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin, “Attention is all you need”, NeurIPS 2017. ↩
Some of the early pioneers in speech recognition and machine translation, like Fred Jelinek, originally worked on digital communications and error-correcting codes, where it really does make sense to refer to searching for the best output as “decoding”. They didn’t bother to find a better word and kept saying “decoding” when they started working on speech recognition. It may not be entirely surprising that Jelinek once said “every time I fire a linguist, the performance of my speech recognizer goes up.” ↩
Taken from: Awni Hannun, “Sequence Modeling with CTC”, Distill, 2017. ↩
A search error happens when $p_{\theta}(\mathbf{y}|\mathbf{x}) > p_{\theta}(\mathbf{\hat{y}}|\mathbf{x})$, but $\mathbf{y}$ wasn’t found during the search. In other words, the model correctly assigns more probability to the correct output sequence than an incorrect output sequence, but the search just didn’t get a chance to evaluate the correct sequence. Another type of error is when the model assigns more probability to an incorrect sequence and picks that sequence as a result. ↩
This paper discovered and diagnosed the problem: Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, Yoshua Bengio, “On the Properties of Neural Machine Translation: Encoder–Decoder Approaches”, SSST-8, 2014. ↩
See: Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate”, ICLR 2015. ↩
Supposedly given by Claude Shannon, although I haven’t been able to find the original reference where he wrote it. ↩
Provided by Andrej Karpathy in the code for The Unreasonable Effectiveness of Recurrent Neural Networks. ↩
See: Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, Jeffrey Dean, “Google’s neural machine translation system: Bridging the gap between human and machine translation”, arXiv preprint arXiv:1609.08144, 2016. ↩