The continuous demand for performance fed by emerging machine learning (ML) and Big-Data (BD) applications, as well as the growing technological limiting factors are pushing system architects to ever more exploit “heterogeneity” in their solutions. Besides performance, energy consumption and programmability represent hard constraints which limit the complexity (and thus possible functionalities) achievable by modern systems. To overcome such obstacles, in recent years, architectural specialization has been massively explored. Such architectural diversity will emerge at different levels and scales in designing new computing systems, posing new challenges to be addressed. From how to pack together ever more functionalities to make systems more intelligent and aware of the surrounding physical environment, to how ease the programming of such systems to express the “what to do” rather than the “how to do”, still ensuring non-functional properties of the system (security, energy saving, etc.) are just few of the challenges that designers and developers will face in the future. An overview of the recent approaches to hardware design and integration, technology limitations, and programming challenges are presented.