ABSTRACT
Occam's razor is a classical argument for Bayesian model selection (BMS). Penalizing complexity leads to choosing a parsimonious model, whereas in frequentist tests one cannot prove the simpler model. This argument is correct but care is needed. Good model selection requires balancing parsimony and power, which depends on three elements: the priors, model, and whether it approximates well the data-generating mechanism (misspecification). Standard finite-dimensional BMS does not penalize complexity sufficiently, whereas high-dimensional formulations impose stronger penalties that ensure consistency, but may reduce power. Regarding misspecification, BMS still targets a reasonable problem when (inevitably) assuming the wrong model, and the effect on false positives vanishes asymptotically, but power is exponentially reduced. To increase power one may enrich the model, but having more parameters can reduce power. We discuss these issues, practical strategies to improve sparsity-power tradeoffs, and offer our views on a long-standing debate on BMS sensitivity to prior dispersion.
