ABSTRACT
Least-squares problems turn out to be the most common optimisation problems in many fields, and this means that they have been very well studied and, fortunately, they have special structure in the problem that makes solving them easier than other problems. This chapter begins by trying to derive a better understanding of gradient descent, and seeing the algorithms that can be used for finding local optima for general problems. It looks at the specific case of solving least-squares optimisation problems, which are the most common examples in machine learning. The Levenberg–Marquardt algorithm itself is very general, but it needs to have the function to be minimised, along with its gradient and Jacobian passed into it. The basic idea of the hill climbing algorithm is to perform local search around the current solution, choosing any option that improves the result.
