Optimization | Mathematics - Wyatt's Notes

Theorem 4.1 (First Derivative Test). If $f$ has a local extremum at an interior point $\mathbf{a}$ And $\nabla f(\mathbf{a})$ exists, then $\nabla f(\mathbf{a}) = \mathbf{0}$.

Points where $\nabla f = \mathbf{0}$ are called critical points (or stationary points).

Remark. Not all critical points are extrema. A critical point can be a local minimum, local maximum, Or saddle point. The second derivative test (Section 4.2) distinguishes these cases.

Theorem 4.2 (Second Derivative Test). Let $f$ have continuous second partial derivatives near a Critical point $(a,b)$ with $f_x(a,b) = f_y(a,b) = 0$. Let

$D = f_{xx}(a,b) f_{yy}(a,b) - [f_{xy}(a,b)]^2$

Be the Hessian determinant. Then:

• If $D \gt 0$ and $f_{xx}(a,b) \gt 0$: local minimum.
• If $D \gt 0$ and $f_{xx}(a,b) \lt 0$: local maximum.
• If $D \lt 0$: saddle point.
• If $D = 0$: the test is inconclusive.

Proof. By Taylor”s theorem to second order, for small $h, k$:

$f(a+h, b+k) - f(a,b) = \frac{1}{2}\left[f_{xx} h^2 + 2f_{xy} hk + f_{yy} k^2\right] + R_2$

Where the remainder $R_2 = o(h^2 + k^2)$ and all partials are evaluated at $(a,b)$. The sign of the Right-hand side is determined by the quadratic form

$Q(h,k) = f_{xx} h^2 + 2f_{xy} hk + f_{yy} k^2 = \begin{pmatrix} h & k \end{pmatrix} H \begin{pmatrix} h \\ k \end{pmatrix}$

Where $H = \begin{pmatrix} f_{xx} & f_{xy} \\ f_{xy} & f_{yy} \end{pmatrix}$ is the Hessian matrix.

By Sylvester’s criterion for $2 \times 2$ symmetric matrices:

• If $\det(H) = D \gt 0$ and $f_{xx} \gt 0$ Then $H$ is positive definite, so $Q \gt 0$ for all $(h,k) \neq (0,0)$Giving a local minimum.
• If $\det(H) = D \gt 0$ and $f_{xx} \lt 0$ Then $H$ is negative definite, so $Q \lt 0$ for all $(h,k) \neq (0,0)$Giving a local maximum.
• If $\det(H) = D \lt 0$ Then $H$ is indefinite, so $Q$ takes both positive and negative values, giving a saddle point.

When $D = 0$The quadratic form is degenerate and the sign is determined by higher-order terms. $\blacksquare$

Theorem 4.3 (Method of Lagrange Multipliers). To find the extrema of $f(x,y,z)$ subject to the Constraint $g(x,y,z) = 0$Solve the system:

$\nabla f = \lambda \nabla g, \quad g = 0$

More generally, for $k$ constraints $g_1 = 0, \ldots, g_k = 0$:

$\nabla f = \lambda_1 \nabla g_1 + \cdots + \lambda_k \nabla g_k$

Proof (single constraint, geometric justification). Let $M = \\{(x,y,z) : g(x,y,z) = 0\\}$ be the constraint surface. If $f$ has a local extremum on $M$ at $\mathbf{p}$ Then the directional derivative $D_{\mathbf{v}} f(\mathbf{p}) = 0$ for every tangent Vector $\mathbf{v}$ to $M$ at $\mathbf{p}$. Since $\nabla f(\mathbf{p}) \cdot \mathbf{v} = 0$ for all Such $\mathbf{v}$The gradient $\nabla f(\mathbf{p})$ must be orthogonal to the tangent space of $M$ At $\mathbf{p}$. But the tangent space of $M$ is orthogonal to $\nabla g(\mathbf{p})$ (by the implicit Function theorem). Therefore $\nabla f(\mathbf{p})$ must be parallel to $\nabla g(\mathbf{p})$I.e., $\nabla f(\mathbf{p}) = \lambda\, \nabla g(\mathbf{p})$ for some scalar $\lambda$. $\blacksquare$

Problem. Find the maximum of $f(x,y) = xy$ subject to $x^2 + y^2 = 1$.

Solution. Set $g(x,y) = x^2 + y^2 - 1$. The Lagrange multiplier equations:

$\nabla f = \lambda \nabla g \implies (y, x) = \lambda(2x, 2y)$

This gives $y = 2\lambda x$ and $x = 2\lambda y$. Multiplying: $xy = 4\lambda^2 xy$.

Case 1: $xy \neq 0$. Then $4\lambda^2 = 1$ So $\lambda = \pm 1/2$.

• $\lambda = 1/2$: $y = x$ And $x^2 + x^2 = 1$ So $x = \pm 1/\sqrt{2}$. Points: $(1/\sqrt{2}, 1/\sqrt{2})$ and $(-1/\sqrt{2}, -1/\sqrt{2})$ with $f = 1/2$.
• $\lambda = -1/2$: $y = -x$ And $x^2 + x^2 = 1$ So $x = \pm 1/\sqrt{2}$. Points: $(1/\sqrt{2}, -1/\sqrt{2})$ and $(-1/\sqrt{2}, 1/\sqrt{2})$ with $f = -1/2$.

Case 2: $xy = 0$. Then either $x = 0$ or $y = 0$. From the constraint: $(0, \pm 1)$ or $(\pm 1, 0)$ with $f = 0$.

Maximum: $f = 1/2$ at $(\pm 1/\sqrt{2}, \pm 1/\sqrt{2})$. Minimum: $f = -1/2$ at $(\pm 1/\sqrt{2}, \mp 1/\sqrt{2})$. $\blacksquare$

Problem. Find and classify all critical points of $f(x,y) = x^4 + y^4 - 4xy$.

Problem. Find and classify all critical points of $f(x,y) = x^3 + y^3 - 3xy$.

Problem. Find the point on the plane $x + 2y + 3z = 6$ closest to the origin.

Problem. Maximise $f(x,y,z) = xyz$ subject to $x + y + z = 1$ and $x^2 + y^2 + z^2 = 1/3$.

flowchart TD
    A[4_Optimization] --> B[Key Concepts]
    A --> C[Core Principles]
    A --> D[Practical Applications]
    B --> E[Fundamental definitions]
    C --> F[Design patterns]
    D --> G[Real-world usage]

Optimization asks: what is the best I can do under constraints? In unconstrained problems, the gradient points uphill — critical points are where it vanishes, meaning all first-order effects balance. The Hessian (matrix of second derivatives) tells you whether the critical point is a minimum, maximum, or saddle, like feeling the curvature of a bowl. Lagrange multipliers handle constraints by saying: at an optimum, the gradient of the objective must be parallel to the gradient of the constraint. Geometrically, this means the level curves of the objective are tangent to the constraint surface. The multiplier $\lambda$ measures the sensitivity — how much the optimum changes if you relax the constraint slightly.

• Partial Derivatives: The gradient and Hessian matrices used in optimisation are constructed from first and second partial derivatives.
• Vector Calculus: Lagrange multipliers use the geometric relationship between gradients and tangent spaces, which is formalised in the study of conservative fields.
• Sequences and Limits: The extreme value theorem and convergence of iterative optimisation methods rely on completeness and compactness from real analysis.
• Multiple Integrals: Computing volumes and moments over constraint surfaces uses the multiple integration techniques developed in this chapter.
• Quantum Mechanics
• Graph Theory
• Classical Mechanics
• Electromagnetism