Embeddings in DL
Translate high-dimensional vectors for items (e.g. movies, text,...) into low-dimensional real vectors in a way that [semantically] similar items are close to each other. Joint embeddings of diverse data types (e.g. text, images, audio, ...) define a similarity between them.
When learning embeddings, Embedding layer = hidden layer with one unit per dimension. Intuitively the hidden units discover how to organize the items in the d-dimensional space in a way to best optimizes the final objective (using backpropagation).
Num dimensions - hyperparameter tuning problem. Too few - insufficient representation, too many - overfitting and slower training. Rule of thumb: num dimensions = fourth root (num possible values)
Embedding Quantization & Truncation: substantial cost and latency reductions in retrieval and similarity search + fewer bits for storage. Example: [12, 1, -100, 0.3,] => [1,1,0,1,] (0 if negative):
Binary Quantization: 45x lower latency, 32x less memory, 96% of retrieval performance.
Scalar (int8) Quantization: 4x lower latency, 4x less memory, 99.6% of retrieval performance.
Combine both: binary search for min. latency & memory + scalar rescore for high performance = save costs.
Embedding Truncation: w/min. performance loss, faster retrieval, clustering, etc. Train model on domain.
Word2Vec
Continuous Bag-of-Words - predicting the middle word based on surrounding context words (a few words before and after; order not important => BoW).
Continuous Skip-gram - predicting words in a range before and after current word in the same sentence. Skip-gram = n-gram with tokens skipped. Context can be represented as skip-gram pairs (target_word, context_word) where context_word = any word within a range (2, 3, etc.) from target_word. Negative sampling - the same skip-gram pairs, but the target_word is not from the context (used for training the model)
GloVe (Global Vectors for Word Representation): trained on non-zero entries of a global word-word co-occurrence matrix (tabulates how frequently words co-occur with one another in the corpus). Model combines global matrix factorization & local context window methods, leverages statistical information by training only on nonzero elements in a word-word cooccurrence matrix (not the entire sparse matrix or individual context windows)