Optimization

By the end, you can

• Explain what an objective function is and what it means to minimize it.
• State the first-order condition for a minimum and explain why it is not sufficient on its own.
• Use the Hessian to classify a stationary point as a minimum, maximum, or saddle point.
• Define convexity and explain why convex problems are reliably solvable.
• Distinguish a local minimum from a global minimum and predict where gradient descent lands on non-convex landscapes.
• Trace one step of gradient descent given f(x), its gradient, a starting point, and a learning rate.
• Explain the effect of a learning rate that is too small or too large.
• Describe how momentum and SGD differ from vanilla gradient descent.
• Write the Lagrange condition ∇f = λ∇g and interpret λ as a shadow price.
• Describe KKT complementary slackness and explain why KKT conditions certify a global optimum for convex problems.