McIRProf is mostly correct. But I do think there is an important bias against people with less prestigious affiliations, regardless of your publishing record. All else equal, you are less likely to get the big grant or the book contract from the elite press. Not that it can’t happen, but institutional prestige matters for these things. You will rarely get asked to review promotion files from good places because they will want letterhead from a place that is at least as prestigious as an institution. While these reviews are a pain in the ****, there is probably some value to developing ties to and doing such a favor for the rising stars who you get to review if you are at a prestigious place. You also get the reputation as a leading senior scholar in your field through word of mouth.
Two other issues. Time-sucking mediocre grad students at less prestigious places, as well as a likelihood of smaller departmental/college budgets, and fewer resources to pass around. On the other hand, if there are fewer productive people to spread resources among, you can actually come out ahead. Don’t underestimate the “big fish in small pond” advantage, at least for campus-based resources.
Having said all that, I am senior and I don’t really worry too much about institutional prestige in my job market searches. I am looking for an R1 with at least a few productive people in my field as my minimum threshold. All the better if they seem motivated to get better. For me, it is all about general lifestyle now – where to live, campus environment, etc.