ABSTRACT
The search for optimality can be thought of as a search through the space formed by the union of all free parameters and some objective function. This can be easily envisioned in three dimensions, and easily extended conceptually to higher dimensions. Consider a situation where we have only two free parameters and an objective function, which of course will vary depending on the value of the free parameters. Typically, we are trying to fit a model by adjusting these free parameters in such a way as to optimize an objective function: minimize an error or maximize some other modeled performance (profit, revenue). The model designer has complete freedom in selecting both the model structure as well as formulating an appropriate objective function that best suits the goals of the problem.
