Why that order and not another
Because each hand beats the one below it by how seldom it shows up in a five-card deal, and by nothing else. Deal five cards at random a thousand times: you'll see a pair about four hundred times, a flush twice, and a straight flush not once. The poker hand ranking is that list of rarities, rarest on top, which is why it's the one thing worth learning by heart and why it only takes an afternoon.
Out of a thousand five-card deals
-
≈ 423
times you see one pair
-
≈ 2
times you see a flush
-
0
straight flushes, nearly always
With seven cards the count shifts at the very bottom. In Texas Hold'em you end up with high card in 17.4% of hands, less often than one pair (43.8%) or two pair (23.5%), and it still loses to both. The ranking was set with five cards, and seven don't change it.