It's perhaps useful to begin with a brief review of how the U.S. crime rate has evolved over time. Perhaps the best-known source is the Uniform Crime Reports data from the FBI, but there is also a National Crime Victimization Survey. The broad trends in all of these data sources are the same. For example, in the UCR data, the number of violent crimes doubled from about 300,000 at the start of the 1960s to 600,000 by 1968, then doubled again to 1.2 million in 1979, and continued rising to peak at 1.9 million in the early 1990s. However, since then the numbers of violent crimes have been falling and was down to 1.3 million by 2009. In other words, rates of crime have been falling for two decades, and have fallen substantially.
Nonetheless, here's Richard Posner writing in "Incarceration Blues" in the November 17 issue of the New Republic: "For the percentage of our [U.S.] population that is incarcerated is the highest of any country in the world; and at a shade under 3 percent (though it has been falling recently because of the states’ fiscal distress), it exceeds by factors of 4 to 7 the percentage incarcerated in any of our peer countries. And yet our crime rates are generally no lower than in those countries, and our murder rate is much higher. Our higher rate of incarceration is attributable primarily to more arrests and convictions rather than to our criminalizing more activities or imposing longer prison sentences, though those are factors as well."
Many of the basic facts can be illustrated with figures from the Bureau of Justice Statistics at the U.S. Department of Justice website. As a starting point, the rate of imprisonment per 100,000 people has quadrupled over the last 30 years, and then number of people on parole has quintupled. I'm certainly open to the argument that imprisonment rates needed to rise in response to the higher crime rates from the 1960s through to the 1980s. I believe that the higher imprisonment rate helped to bring down crime rates in the 1990s. For example, economist Steven Levitt makes this case in a Winter 2004 article in my own Journal of Economic Perspectives. But did the rate of imprisonment need to quadruple--and then to remain at that level? At a cost (including prison construction costs) of about $50,000 per inmate per year, we may well have taken a defensible public policy and pushed it too far.
Unsurprisingly, the costs of the criminal justice system have been rising dramatically, as well. Here are graphs again from the Bureau of Justice Statistics. The numbers are adjusted for inflation (into year 2007 dollars). Although this area of spending has leveled out the last few years, it has leveled out at levels double or more that of the early 1980s.
Given the pressures on state and local budgets, if nothing else, what might be done about this? Here are some charts that suggest some possible directions. First, Posner noted in the quotation above how more arrests are a large part of the reason for the higher U.S. imprisonment rate. A large share of the increases in arrests are for drug-related offenses: these have tripled in the last three decades. Moreover, if one looks at the number of those imprisoned for non-violent drug offenses, the total has risen from near-zero three decades ago to about 250,000 today.
Another issue is that spending in the criminal justice system is tilting more heavily toward prisons, and less toward police: in practice, this means more toward punishing crime after it has happened and less toward keeping the peace beforehand. Notice in the graph below that "corrections" spending was about half of police spending in 1982, but has risen to about 3/4 of police spending. 
On pure crime-reduction grounds, one can make a plausible case for acting to reduce the number of people in prison for non-violent crimes like a number of drug-related offenses, and transferring the funds saved to less expensive community-supervision and more active programs for reducing crime in communities--including more police on the streets and drug treatment programs. I also suspect that when our society decided to de-institutionalize its mental patients, we have instead ended up re-institutionalizing some of that same population more expensively in prisons.
But in addition, it is not healthy for the land of the free to have reached a situation where almost one male out of every 20 in the entire country--including those on probation and parole-- is under the supervision of the criminal justice system. For African-Americans, the rate of those under the supervision of the criminal justice system is nearly one out of ten. (For details, here is a 2009 Pew Foundation report.)
People--and especially young men--will commit crimes. Many of them, especially repeat violent offenders, should do some time. But for the crime rate, and for our communities, and for the offenders themselves, we need to find more productive answers than imprisonment at $50,000 per year for nonviolent criminals. Earlier this week, a police officer named Daniel Horan wrote an op-ed in the Wall Street Journal, "The Just-Say-No Crimebusters," about how using the threat of punishment to ask petty criminals to stop can be remarkably effective.