Sure, there are a lot of potential things to do - either by enumerating towards the full "brute force" loss partway as I think you are hinting toward, or treating as a subsequences, or something else [0, 1, 2]. The point is, IMO RNNs
do need some kind of structured loss (more than per step likelihood) to be competitive with HMM approaches using Viterbi decoding, in addition to some sequence smarts (such as beam search) in decoding.
At least in NMT, enumerating the possible decisions as 3n + 1 is almost impossible since the softmax size is generally the memory bottleneck in training - and a bigger vocabulary is typically a huge win. It is more feasible in speech, but often your labels are themselves triphones and you end up with a pretty large vocabulary too.
Figuring out how to get RNNs closer to on par with DNN-HMMs with Viterbi decoding (or full on sequence training [3]) through something like "deep fusion" with a language model (or something else) is something I am very interested in.
[0] http://arxiv.org/abs/1312.6082
[1] http://arxiv.org/abs/1511.06456
[2] https://arxiv.org/abs/1511.04868
[3] http://www.danielpovey.com/files/2013_interspeech_dnn.pdf
At least in NMT, enumerating the possible decisions as 3n + 1 is almost impossible since the softmax size is generally the memory bottleneck in training - and a bigger vocabulary is typically a huge win. It is more feasible in speech, but often your labels are themselves triphones and you end up with a pretty large vocabulary too.
Figuring out how to get RNNs closer to on par with DNN-HMMs with Viterbi decoding (or full on sequence training [3]) through something like "deep fusion" with a language model (or something else) is something I am very interested in.
[0] http://arxiv.org/abs/1312.6082
[1] http://arxiv.org/abs/1511.06456
[2] https://arxiv.org/abs/1511.04868
[3] http://www.danielpovey.com/files/2013_interspeech_dnn.pdf