Because activation functions play a crucial role in the training of such deep learners as the Convolutional Neural Network (CNN), the development of more efficient and better-performing activation functions has been the focus of much recent research. Most activation functions introduce different nonlinearities when mapping the outputs of one layer to the inputs of another. This diversity makes activation functions an excellent candidate for generating ensembles of deep learners. In this study, we not only develop an ensemble of CNNs trained using different activation functions, but also introduce a novel activation function. The objective is to improve the performance of CNNs on small to medium-sized biomedical datasets. Experimental results demonstrate that the best ensemble proposed in this chapter outperforms CNNs trained with the standard ReLU as well as with each of the other tested stand-alone activation functions (P-value of 0.01). For a more comprehensive evaluation, we test our approach on 13 different biomedical datasets using three CNN architectures: Vgg16, ResNet50, and DenseNet. The MATLAB code used in this study will be available at https://github.com/LorisNanni">https://github.com/LorisNanni.