One of the most appealing functions of music is that it can convey emotion and modulate a listener’s mood [90]. It is generally believed that music cannot be composed, performed, or listened to without affection involvement [159]. Music can bring us to tears, console us when we are grieving, and drive us to love. Music information behavior studies have also identified emotion as an important criterion used by people in music searching and organization. In this chapter, we describe the importance of music emotion recognition, with examples that illustrate the application of emotion-based music retrieval. After briefly describing the system components of a typical music emotion recognition system, we move on to discuss the issues one needs to consider when building the music emotion recognition system.