Understanding Recurrent Neural Networks (RNNs)

Neural Networks (NNs) that depend on past history are called Recurrent Neural Networks (RNNs). The technical term is networks that depend on temporal dependencies or dependencies that change based on time are called RNNs.

This class of NN’s are distinct than the ones that, for example, do image recognition which do not have dependencies on time. RNN’s have wider applications because most applications have dependencies that depend on time. A key challenge in RNNs is the vanishing gradient problem in which the contribution of information or memory (in the next paragraph) decays geometrically over time.

rnn-cell
Basic Cell in a RNN

The basic idea is that you take a feed forward NN (FFNN) and introduce internal memory state that takes the output from the internal training layer. Thus, our FFNN now remembers what happened before.  For completeness, prediction can be 1..n.

The next step is to chain each of these cells together.

Thus, a RNN is a combination of multiple such cells where multiple inputs are fed into different cells (think prediction of next word is dependent on the last few words), each cell has it’s own memory coming in from its previous iteration and each cell goes ahead and makes its own prediction.

rnns.png
RNNs

The beauty of RNNs is that they can be stacked as lego blocks. To imagine this, think about the above picture as 1 RNN lego block and make the prediction layer feed into another RNN block stacked above it.

Applications

There are some interesting use cases for RNNs:

  • Sentiment analysis
  • Speech recognition
  • Time series prediction
  • Natural language processing
  • Gesture Recognition

Amazon Lex provides a framework to build conversational interfaces using voice and text with .

Leave a Reply