Xenophobia is a word derived from Greek. So let's start there, shall we? The Greek "xenos" was used towards both "foreigners" and "strangers." Of course, if you know anything about classical Greek societies, you would know that even those born in a city-state were considered "foreigners" for many, many generations. For example, the great-great-great-grandsom of a Corinthian would not usually be recognized as a "citizen" of Athens. He would be considered a "metic" (a form of "resident alien").
From the start, the term has always meant to mean fear of those who were "strange" or "foreign" to the dominant culture of reference. Although in American English (though not, I should note, in non-American English) it has been used simply to refer to immigrants who are, quite literally, from a foreign place.
But in practice, xenophobia (in the US) often assumes that those we find to be "different" must be also "foreign." Exhibit A: the "birther" controversy.
In a sense of civic rights and social inclusion, minorities (regardless of their point of origin) are "foreigners" and treated as such. Hence, all the ranting on PSJR about Asians, women, gays, etc. is, well, xenophobic in nature.