One of the (vanishingly few) real things that the race baiters in academia have noticed is that when you’re in a group of like people, everyone gets along pretty well with everybody else, and things go pretty smoothly. Add somebody “different” – and friction appears. If you don’t look/behave in whatever is the local “normal” fashion, you’re going to attract attention, and given humanity’s instinct for tribalism, that attention is going to be skewed toward “suspicious” (unless you’re an attractive female; but on the other hand, the positive attention may not be very agreeable to you).
So yes, if you’re the only black person in an enclave of white people – you’re going to feel awkward, because your instincts are whispering “these people are Not Like Me” in your ear – at least until you get to know them and maybe find out that they are “Like Me,” at least in the ways most important. White people in enclaves of black people feel the same way. (And usually justifiably more frightened for their safety. White kids don’t go around in feral packs hunting black people for sport. Black kids are known to engage in such criminal activity, an artifact not of their skin color, but of a culture that glorifies violence as “authentic” and law-abiding, rule-following behavior as “acting white.” Who’re the racists, now?)
The thing is, normality privilege holds true for groups defined by any characteristic, not just skin color. Catholics and Protestants would feel awkward in each other’s worship spaces. Computer geeks are generally uncomfortable if they get plopped into a group of football players. The tendency to group around Like Characteristics is part of the human hardware. The Bill of Rights even recognizes this in the First Amendment – freedom of association. And researchers find that when they build models of group dynamics, even if each individual merely wants to live surrounded by half Like Me and is perfectly okay with half Not Like Me, the groups will organically segregate over time. Like calls to like; people are just more comfortable in tribes.
And tribes can wind up being stifling, especially if, to receive your normality privilege and get an easier path in life, you have to behave in a certain way as well as look a certain way.