A community demonstrates social trust when most members believe that others in their community are contributing to the goals and projects of one another by complying with mutually beneficial shared social norms. Scholars have argued that social trust is a good thing because it plays an essential stabilizing role in cooperative systems and provides many important benefits. Much of the relevant literature on social trust has assumed that individuals are epistemically justified in believing that others will comply with mutually beneficial norms. This paper examines circumstances when such epistemic justification is not present. Focusing on the example of anti-Black racism in the United States, we offer an account of epistemically justified social distrust in which individuals are justified in believing that others in their community are irrelevant or harmful for achieving one another's goals either by upholding harmful social norms or by failing to uphold beneficial social norms in an inclusive manner. We also explore potential pathways for building trust when social distrust is epistemically justified. Because justified social trust requires evidence that society is in fact trustworthy, the first step to building justified social trust must be building a society that is in fact trustworthy.