Gradient, Jacobian, and Hessian

The gradient, Jacobian, and Hessian are fundamental tools from calculus used to analyze and optimize functions, especially in the context of machine learning. The gradient of a scalar-valued function of multiple variables is a vector containing the partial derivatives with respect to each variable, indicating the direction of the steepest ascent. The Jacobian matrix generalizes the gradient to vector-valued functions of multiple variables, containing all the partial derivatives of each output component with respect to each input variable. The Hessian matrix, on the other hand, is the square matrix of second-order partial derivatives of a scalar-valued function, providing information about the local curvature of the function.