Although our visual perceptions of objects and of other people are more than just visual -- enactively involving motor, affective, and evaluative processes -- our visual perception of the other person who is gazing back at us is more complex. I review two approaches to this complexity (Sartre, Levinas) and show how they both remain relatively abstract. I argue that visual perception of the other's gaze is always situated and involves complex interactive behavioral and response patterns that arise out of an active engagement which is always more than a simple recognition. I cite empirical studies to show that such social perception is not a matter of me seeing the other's gaze or face simpliciter, but of seeing that the other sees me. The other's gaze is not something that can be subsumed into a strictly visual representation of eye direction, for example. It has an affective impact on my own system that sets me up for further response. On concepts that acknowledge the involvement of embodied and worldly contexts, there is something immanent in the gaze; something that elicits an elementary responsivity that characterizes our intercorporeal encounters.