ABSTRACT
Tree-based models are a class of nonparametric algorithms that work by partitioning the feature space into a number of smaller regions with similar response values using a set of splitting rules. This chapter discusses powerful ensemble algorithms— like random forests and gradient boosting machines—which are constructed by combining together many decision trees in a clever way. There are many methodologies for constructing decision trees but the most well-known is the classification and regression tree algorithm proposed in Leo Breiman. Early stopping explicitly restricts the growth of the tree. An alternative to explicitly specifying the depth of a decision tree is to grow a very large, complex tree and then prune it back to find an optimal subtree. Decision trees have a number of advantages. Trees require very little preprocessing. Gini index and cross-entropy are the two most commonly applied loss functions used for decision trees.
