I've never heard of UPenn... honestly... or Caltech... I am aware of most of the Invy League schools like Harvard, Princeton, Brown, Cornell, Dartmouth, Old Dominion, etc.
Its all about perception... go to any SEC campus in the Deep South and you'll likely get the same answer.
Saint Leo is my alma mater and is a decent school that is renowned through the military around the world for their Veterans Programs, but if you ask a Big10 student or grad about them they'd offer the same answer... Saint Leo who?
Hell there are some people in Tampa and in Florida that don't know what Saint Leo is, or where it is... I can't tell you how many times I have heard "Isn't that a girls school" or "Isn't that located up north somewhere"...
Penn, CalTech, and every other school in question is in the national top 10. USF is #170 last I checked. Some of those schools have multi-billion dollar endowments (Harvard, for instance, is around 30 billion). The fact that the top 10 academic universities in the country can thrive without powerhouse football programs is completely irrelevant to our situation.