Recognizing scene information in images or videos, such as locating the objects and answering “Where am I?” has attracted much attention in the computer vision research field. Many existing scene recognition methods focus on static images and cannot achieve satisfactory results on videos that contain more complex scene features than images. In this chapter, we propose a robust movie scene recognition approach based on panoramic frame and representative feature patches. When our proposed approach is implemented to recognize scenes in realistic movies, the experimental results show that it can achieve satisfactory performance.