A famous long-ago economist named Harold Hotelling proposed a classic explanation for this phenomenon back in a paper called "Stability in Competition," published in the March 1929 issue of the Economic Journal (39:153, pp. 41-57).
In one of his illustrations, Hotelling discussed the of two sellers of a product who are thinking about where to locate along Main Street. For simplicity, imagine that the addresses along the street are numbered from 1-100. The working assumption is that customers are spread evenly along Main Street, and the customers will go to whichever store is located closer to them. In this situation, if one store locates at, say, 10 Main Street, the other store will then choose to at 11 Main Street. The first store will then get all the customers from 0-10, and the second store will get all the customers from 11-100. The first store will then relocate to 12 Main Street, to snag the majority of customers, and the two stores will keep leap-frogging each other and relocating until they end up located side by side, right in the middle of Main Street.
As Hotelling pointed out back in 1929, this clustering is not ideal. From the consumers's point of view, it would be more useful to have the two stores located at 25 Main Street and 75 Main Street, because then no consumer along the street from 1 to 100 would be more than 25 away from a store. But the dynamics of competition can lead to excessive clustering.
Hotelling argued that this excessive sameness is apparent in many aspects of public competition, including competition between firms introducing new products, and competition between Republicans and Democrats. He wrote:
"Buyers are confronted everywhere with an excessive sameness. When a new merchant or manufacturer sets up shop he must not produce something exactly like what is already on the market or he will risk a price war ... But there is an incentive to make the new product very much like the old, applying some slight change which will seem an improvement to as many buyers as possible without ever going far in this direction. The tremendous standardisation of our furniture, our houses, our clothing, our automobiles and our education are due in part to the economies of large-scale production, in part·to fashion and imitation. But over and above these forces is the effect we have been discussing, the tendency to make only slight deviations in order to have for the new commodity as many buyers of the old as possible, to get, so to speak, between·one's competitors and a mass of customers.
So general is this tendency that it appears in the most diverse fields of competitive activity, even quite apart from what is called economic life. In politics it is strikingly exemplified. The competition for votes between the Republican and Democratic parties does not lead to a clear drawing of issues, an adoption of two strongly contrasted positions between which the voter may choose. Instead, each party strives, to make its platform as much like the other's as possible. Any radical departure would lose many votes, even though it might lead to stronger commendation of the party by some who would vote for it anyhow. Each candidate " pussyfoots," replies ambiguously to questions, refuses to take a definite stand in any controversy for fear of losing votes. Real differences, if they ever exist, fade gradually with time though the issues may be as important as ever. The Democratic party, once opposed to protective tariffs, moves gradually to a position almost, but not quite, identical with that of the Republicans. It need have no fear of fanatical free-traders, since they will still prefer it to the Republican party, and its advocacy of a continued high tariff will bring it the money and votes of some intermediate groups.Of course, it's not literally true that Republican and Democratic politicians both locate exactly in the middle of the political spectrum. Hotelling was describing a tendency to push to the middle, but in politics, there is also a need to assure your voters that you share their beliefs. Thus, there's a saying that American politics is a battle fought between the 40-yard lines. (For those unfamiliar with the line markers on an American football field, the statement suggests that the political battle is fought between the addresses of 40 and 60 on a Hotelling-style Main Street.) Mainstream politicians thus face a continual dynamic where they seek to reassure their more ardent partisans that they are on their side, while shading and tacking as needed to pick up voters in the middle. At an intuitive level, politicians recognize that offering "a choice, not an echo" is part of what led Barry Goldwater to a loss of historic magnitude in the 1964 US presidential election.
Political competition that is usually between centrists, whether right-of-center or left-of-center, does have some benefits. Extremists are much less likely to win high office. And even when the other side wins, it's reassuring to think that the person who won is at least closer to the center than the true believers at the extreme of that side. But every now and then, many of us yearn for a few more conviction politicians, who say what they mean and mean what they say, who play a greater role in driving the public debate, and who are OK with the possibility that doing so might end up costing them an election.
[For the record, using the metaphor of a football field to describe the range of political choice seems to have originated with the 1970 best-seller The Real Majority: An Extraordinary Examination of the American Electorate, by Ben Wattenberg and Richard M. Scammon. But they used the image to discuss how political conflict might sometimes be between those near the middle and sometimes between those at with more extreme positions. The claim that American politics usually happens between the 40 yard-lines is one of those statements that seems to have evolved afterwards, without a clear single author.]