Scene “gist” recognition is rapid holistic semantic processing of a scene within a single eye fixation. Our own work has shown that aerial view gist recognition is slower and less accurate than for terrestrial views, and that this is due, in part, to aerial views missing constraints on the layout of scenes that are intrinsic to terrestrial views. The lack of these constraints has been shown to reduce the speed and accuracy of search for objects in aerial views. Our work also shows that eye-movement patterns during attempts to categorize aerial views are less efficient than for terrestrial views. Together, these findings suggest that normal viewers are expert at rapidly categorizing terrestrial views but far from expert for aerial views. Our research shows that despite the difficulty that people have in categorizing aerial scenes in comparison to terrestrial views, the types of errors made between views are remarkably similar, suggesting at least some level of similarity across viewpoints. The extent to which we can say that aerial and terrestrial views are the same may be a product of perceptual and semantic information that is view invariant, based on image statistical analyses and human psychophysical and physiological data. Thus, the processing of view-invariant information should be the foundation for adapting human and machine vision to aerial scene gist, and an attempt to elucidate this problem will therefore be an important topic in the chapter.