I admit that this topic may be of more absorbing interest to me than to most other humans on planet Earth. I'v been Managing Editor of the JEP since the start of the journal in 1987, and the bulk of my job is to edit the articles that appear in the journal. The length of JEP articles hasn't risen much at all during the last 27 years, while the length of articles in other journals has roughly doubled in that time. Am I doing something wrong? In an impressionistic way, what are some of the plausible reasons for additional length?
Ultimately, longer papers in academic research journals reflect an evolving consensus about what constitutes a necessary and useful presentation of research results. Over time, it is plausible that journal editors and paper referees have become more aggressive in requesting that additional materials should presented, additional hypotheses considered, additional statistical tests run, and the like.
An economics research paper back in the 1960s often made a point, and then stopped. An academic research paper in the second decade of the 21st century is more likely to spend a few pages setting the stage for their argument, setting the stage for the big question, give some sense in the introduction of the paper of main results, have a section discussing previous research, have a section giving a background theory, and so on.
Changes in information and computing technology have pushed economics papers to become longer. There is vastly more data available than in 1970, so academic papers need to spend additional space discussing data. There has been a movement in the last couple of decades toward "experimental economics," in which economists vary certain parameters--either in a laboratory with a bunch of students, or often in a real-world setting--which also means reporting in the research paper what was done and what data was collected. With cheaper computing power and better software, it is vastly easier to run a wide array of statistical tests, which means that space is need to explain which tests were run, the differing results of the tests, and which results the author finds most persuasive.
In the past, the ultimate constraint on length of academic journals was the cost of printing and postage. But in web-world, where we live today, distribution of academic research can have a near-zero cost. Editors of journals that are primarily distributed on-line have less incentive to require short articles.
Finally, one should mention the theoretical possibility that academic writing has become bloated over time, filled with loose sloppiness, with unneeded and length excursions into technical jargon, and occasion bouts of unrestrained pompousness.
Whatever the underlying cause of the added length of articles in economics journals, it creates a conflict between the underlying purposes of research publications. One purpose of such publications is to create a record of what was done, so that the data, theory, and arguments are spelled out in detail. However, another purpose is to allow findings to be disseminated among other researchers, as well as students and policy-makers, so that the results can be more broadly considered and understood. Longer articles probably do a better job of creating a record of what was done, and why. But given that time limits are real for us all, it now takes more time to read an economics article than it did four decades ago. The added length of journal articles means that many more pages of economics research articles are published, and a smaller proportion of those pages are read. I skim many economics articles, but having the time and space to read an article from beginning to end feels like a rare luxury, and I suspect I'm not alone.
The challenge is how to strike the right balance between the competing purposes of full documentation of research (which if unrestrained could easily run to hundreds of pages of data, statistics, and alternative theoretical models for a typical research paper), and the time limits faced by consumers of that research. Many modern research papers are organized in a way that allows or even encourages skimming to hit the high spots: for example, if you need to know right now about the details of the data collection, or the details of the theoretical model, or the details of statistics, you can skip past those sections.
Another option mentioned by Card and DellaVigna is the role of academic journals that go back to the old days, with a focus on presentation of key results, with all details available elsewhere. They write: "There may be an interesting parallel in the field of social psychology. The top journal in this field, the Journal of Personality and Social Psychology, publishes relatively long articles, as do other influential journals in the discipline. In 1988, however, a new journal, Psychological Science, was created to mirror the format of Science. Research papers submitted to Psychological Science can be no longer than 4,000 words. ... Psychological Science has quickly emerged as a leading journal in its area. In social psychology, journals publishing longer articles coexist with journals specializing in shorter, high-impact articles."
My own journal, the Journal of Economic Perspectives, offers articles that are meant as informed essays on a subject, and thus typically meant to be read from beginning to end. We hold to a constraint of about 1,000 published pages per year. (But even in JEP, we are becoming more likely to have added on-line appendices with details about data, additional statistical tests, and the like.) I sometimes say that JEP articles are a little like giving someone a tour of a house by walking around and looking in all the windows. You can get a good overview of the house in that way. But if you really want to know the place, you need to go into all the rooms and take a closer look.
The growing length of articles in economic research journals means that the profession has been giving greater priority to full presentation of the back-story of research, at the expense of readers. In one way or another, the pendulum is likely to swing back, in ways that make it easier for consumers of academic research to obtain a somewhat nuanced view of a range of research, without necessarily being buried in an avalanche of detail--but while still having that avalanche of detail available when desired.