PEFT = (Q)LORA using LoraConfig() from peft library
Updates small subset of model's trainable params => much faster, memory-efficient.
How it works: a) identify layers to apply LoRA to (k,v,q,output proj layers), b) LoRA introduces and applies low-rank weights matrices to these layers – these m. are much smaller in size vs. original weight matrices, significantly reducing the number of trainable params. The original model weights are kept frozen, and only the low-rank matrices’ parameters are updated during fine-tuning.
We can have multiple lightweight portable LoRA models for various downstream tasks. As in LORAX (Lora Exchange) - serve large LLM once with many different adapters.
Matrix rank – max # linearly independent column vectors or row vectors in the matrix (=info). Found using some distance measure like the Frobenius norm. Good for resource-constrained env. like Google Colab
rank = 5; U, S, VT = np.linalg.svd(original_matrix, full_matrices=False) COPY EXAMPLES FROM GEMINI HERE
# Keep only the top 'rank' singular values
U_k = U[:, :rank]; S_k = np.diag(S[:rank]); VT_k = VT[:rank, :]
# Construct low-rank matrix
low_rank_matrix = np.dot(np.dot(U_k, S_k), VT_k)
DORA - weight-decomposed low-rank adaptation; decomposes weight updates into a) magnitude, b) direction. Direction handled by normal LoRA, whereas the magnitude is handled by separate learnable parameter.
Galore - a) gradients projected into a lower-dimensional (low-rank) subspace, b) projected gradients, model weights quantized from 32-bit float to 8-bit int, c) these gradients are used to update model weights, d) weights are then de-quantized and updated.
Quantization using BitsAndBytesConfig() from transformers
Reduces weights’ precision from full precision format (float32) to half precision (float16) or 8-bit precision (int8). Latter maps continuous range of weight values to a discrete set of values decreasing model’s memory footprint and inference time + speeding up computations; trade-off in accuracy - beneficial for deploying models on resource-constrained devices such as mobile phones or embedded systems. Example - your float32 weight = 0.34567 (range from -1 to 1); q. maps this continuous range to an 8-bit integer scale, e.g. from 0 to 255. => apply scaling and rounding, e.g. map -1 to 0 and 1 to 255 => 0.34567 becomes 172.
Pruning + PEFT QLORA - 1) up to 40%-50% of LLM layers pruned with min. impact on accu, 2) Identifying layers to prune: via similarity score to find redundant or less important layers (lowest angular distance), 3) pruning strategy: progressively delete layers that showed min. change in output when compared to adjacent layers, 4) fine-tune post-pruning (small amount of PEFT QLoRA) to recover lost performance. Deep layers removed with negligible effect (not shallow ones) - inefficient training of deep layers?