ABSTRACT
Detection and mitigation of malware continue to pose challenges as potential attackers find new and advanced ways to escape from detection methods, especially with the advent of code obfuscation, metamorphic malware, oligomorphic malware, packers, and newly released (zero-day) malware. Conventional security systems that depend on static analysis and signatures are largely ineffective against new emerging attacks, and the development of more innovative and effective malware defense mechanisms has been identified as an urgent requirement in the cyber-security community. In this chapter, we focus on this critical issue by providing a systematic and detailed overview of existing malware analysis and detection techniques, their findings and limitations, latest technological trends in malware detection, and open problems and challenges in this area. This chapter aims to help the readers to have a comprehensive view of the malware detection field and the new developments and directions of research explored by the scientific community to tackle the problem.
