Temperature Explained
Default=1.0. Higher temperature - more creative, diverse outputs, lower - more focused and deterministic generated text - hyperparameter affecting proba distribution of generated tokens generated by the GPT model.
Mathematically, the temperature is incorporated into the softmax function (converts raw logits produced by the GPT model into a probability distribution) which becomes temperature-scaled softmax:
 |
 |
 |
Formulas 2 and 3 are equivalent: β=1/τ. Here logits r scores. If they r energy (Boltzman), use “–β” |
Parameter tau - temperature - controls softness of proba distribution. When tau gets lower, the biggest value in x gets more probability, when tau gets larger - the proba will be split more evenly on different elements.
Function Calling
Developers can now describe functions to LLM, and have the model intelligently choose to output a JSON object containing arguments to call those functions. Example: define function for weather as json output = { ‘city’: city, ‘state’: state, ‘country’: country, ‘temperature’: degrees Farenheit}. By providing schemas for "functions", the LLM will choose one and attempt to output a response matching that schema.
LSTM vs. Transformer
LSTM models consist of RNN cells designed to store and manipulate info across time steps -address the vanishing gradient problem, captures somewhat long dependencies in sequences. Transformers contain a stack of encoder and decoder layers, each consisting of self-attention and feed-forward neural network components. Self-attention - the most notable difference is the absence of RNN cells in Transformer architecture, which allows it to process inputs in parallel, resulting in faster computation. This architecture allows Transformers to process sequence data without the need for recurrent connections or cells, enabling:
parallel processing (self-attention mechanism processes multiple input tokens simultaneously ) for machine translation, text summarization, natural language understanding, etc.
more efficient long-range dependency handling due to the self-attention mechanism that weighs importance of tokens in input seq to help learn relations between words, whereas LSTMs might struggle with retaining info from distant positions in longer sequences. Attention mechanism lets one focus on different parts of input sequence simultaneously and without regard to distance (not sequentially). RNN / LSTM - sequential processing when the word’s encoding strongly affects only the next word’s representation => its influence fades after a few time steps (may be even lost in the final context vector). Bi-directional models - tokens in the end may have more importance, but it’s still a workaround not a real solution for very long dependencies.
LSTMs can still perform well on shorter sequences in tasks not requiring the full power and complexity of a Transformer model, e.g. sentiment or time-series prediction.
Transformer disadvantages:
memory constraints - requires more memory and computation power vs. RNNs and LSTMs due to extensive use of self-attention mechanisms => using Transformers on edge devices w/limited CPU and memory can be challenging + for deep Transformers, training requires significant computational resources – may not always be feasible. Choice of model should be based on specific reqs and constraints at hand.
not a disadvantage, but a limitation - transformers can capture only dependencies within the fixed input size max_len (if 50 tokens, then only 50). Transformer-XL tries to overcome this by kinda re-introducing recursion by storing hidden states of encoded sentences.