RNN - information cycles through a loop, network remembers output from previous time step(s). RNNs have two common issues: vanishing or exploding gradients during backpropagation, making it difficult to train model.
Used for sequence labeling (POS), NER (IOB), classification w/softmax. Stacked & bidirectional RNNs
LSTM
LSTMs - used for seq2seq tasks (translation, text generation, etc.), are more efficient than RNNs at handling sequence data, and are designed to address the vanishing gradient problem of RNNs by using a series of gates to retain relevant and forget irrelevant info which remember long-term dependencies in data and control the flow of info across time steps:
Input gate: selects info to add to current context to be stored in the cell state - sigmoid activation:. (values from 0 (retain nothing) to 1 (retain everything)).
Forget gate: deletes unneeded info from the context (decides how much of prev cell state should be forgotten) by a weighted sum of the previous state’s hidden layer and the current input, passes that through a sigmoid, multiplies by the context vector. Sigmoid activation, values from 0 (forget more) to 1 (forget less).
Output gate: decides what cell state info should be sent to the next layer - combination of sigmoid activation (deciding what information to pass) and tanh (to scale the values).
RNNs and LSTMs struggle with capturing long-range dependencies in input data due to their sequential nature - makes it difficult to learn complex patterns over long sequences, especially significantly long ones.
Gated recurrent units (GRUs)
Simplify calc by dispensing separate context vector, and reducing gates to 2 - reset & update gate.
Additional reading on LLM Fundamentals
Source: https://cameronrwolfe.substack.com/p/llama-2-from-the-ground-up
The transformer architecture [link]
Decoder-only transformer architecture [link]
Language modeling (next-token prediction) objective [link]
Prompting language models [link]
Specialized language models [link]
Decoding (or inference) with a language model [link]
Beam search
Used for seq2seq models like RNNs or LSTMs, especially in translation; based on conditional probabilities
Example: breadth-first-search w/heuristic filter. A variant of Viterbi decoding that maintains only a fraction of high scoring states rather than all states during decoding - prunes the search space to stay within a fixed-size memory footprint
Beam search considers multiple best options based on beamwidth using conditional probability, which is better than the Greedy search (considers only one best option).
Illustrative example:
Seq2seq model for machine translation. “Voy a visitar mis padres” => translation
We will select the beam width =3; our vocabulary of English words is 10,000.
| Step 1: Find the top 3 words with the highest probability given the input sentence: decoder applies softmax f(x) to all 10,000 words in vocab and selects top 3 most probable words (I, My, We), store them in memory. |  |
|---|---|
Step 2: Find the three best pairs for the first and second words based on conditional probability. Take the first word “I”, apply the softmax to all 10,000 words in vocabulary. Same for the other two words: “My: and “We” Run 30,000 different combinations to choose the top 3 pairs of the first and the second words => “I am”, “My parents”, “I will” (dropped “We” – no bigram with high probability there) |
 |
| Step 3. Find the three best pairs for the first, second and third word based on the input sentence and the chosen first and the second word and so on |  |
| Finally, pick three sentences with the highest probability. We finally pick the output of the decoder as the sentence with the highest probability | |
| A higher beam width will give a better translation, but would use a lot more memory and computational power |