What is the difference between first-order and second-order optimization methods in machine learning?

Question

Machine Learning — Hard

What is the difference between first-order and second-order optimization methods in machine learning?

Accepted Answer

First-order optimization methods in machine learning only use gradient information for updates, while second-order methods utilize curvature information via the Hessian matrix. This allows second-order methods to calculate more accurate step sizes but at a higher computational cost.