DL Math: Gradients & Derivatives · Deep Learning and Machine Learning

DL Math

Gradient

Cartesian basis

A basis a, b, c - a Cartesian rectangular basis if:

1),and (orthogonal)

2) (normalized)

Extended definition:

Def. 1. Vectors,, … ,is called orthogonal if the dot product of every pair of distinct vectors equals zero. The orthogonality condition can equivalently be stated as pairwise

perpendicularity of these vectors^,.

Def. 2. Vectors,, … ,is called normalized if each of the vectors is a unit vector.

Def. 3. Vectors,, … ,are called orthonormal if they are orthogonal and normalized. The orthonormality condition in terms of the dot product can be written as follows:.

Def. 4. A Cartesian basis of a vector space is a set of vectors

,, … , that are orthonormal and whose count n equals the dimension of the vector space.

Gradient (vector)

Gradient of a cost function C(x_1, x_2, …, x_m) in point x is a vector of the partial derivatives of C in x.
The derivative of a function C measures the sensitivity to change of the function value (output value) with respect to a change in its argument x (input value). In other words, the derivative tells us the direction C is going.
The gradient shows how much the parameter x needs to change (in positive or negative direction) to minimize C.

The gradient at a specific point in space is a vector whose direction indicates the direction of steepest increase of a function ɸ whose value varies from one point of space to another (a scalar field), and whose magnitude (modulus) equals the rate of growth of that function in that direction.

For example, if ɸ is the height of the earth's surface above sea level, its gradient at every point of the surface points in the “direction of the steepest ascent”, and its magnitude characterizes the steepness of the slope.

Below is how to find the general equation for the entire vector field of all gradients of a single function — the sum of the derivatives with respect to each variable, each multiplied by its basis vector: 2*i + 6y*j - cos(z)*k. To compute the specific vector at a given point, substitute the coordinates of that point: the gradient (the vector pointing in the direction of maximum growth of the function) at the point (1, 2, 3) = 2*i + 12*j - cos(3)*k:

The gradient vector can be interpreted as the "direction and rate of fastest increase". If the gradient of a function is non-zero at a point p, the direction of the gradient is the direction in which the function increases most quickly from p, and the magnitude of the gradient is the rate of increase in that direction.

Further, the gradient is the zero vector at a point if and only if it is a stationary point (where the derivative vanishes). The gradient thus plays a fundamental role in optimization theory, where it is used to maximize a function by gradient ascent

Relation to derivative

The gradient is closely related to the (total) derivative ((total) differential) df: they are transpose (dual) to each other. Using the convention that vectors in R^n are represented by column vectors, and that covectors (linear maps) are represented by row vectors, the gradient and the derivative are expressed as a column and row vector, respectively, with the same components, but transpose of each other:

However, they represent different mathematical objects: at each point, derivative = cotangent vector = linear form [covector] expressing change in [scalar] output for a given infinitesimal* change in [vector] input, while at each point, the gradient = tangent vector expressing an infinitesimal change in [vector] input.

Gradient - element of tangent space at a point, while derivative - map from tangent space to real numbers. The tangent spaces at each point of R^n can be identified with the vector space itself, and similarly the cotangent space at each point can be identified with the dual vector space of convectors.

Computationally, matrix multiplication of tangent vector and derivative = dot product of tangent vector and gradient: