Here's a chart from Macrotrends showing the inflation-adjusted price of oil going back to the 1940s. If one had to characterize the pattern of oil prices since about 1970, it would be fair to say that there are some sharp rises and falls, but not much long-term trend either up or down.
At a political economy level, it seems to me that we are often astonished because we tell ourselves a story about the previous change in oil prices, and then we have a hard time shifting our story. For example, I'm old enough to remember stories from the 1970s about how the iron grip of the OPEC cartel meant that oil prices would never fall again; and stories from the 1980s and 1990s about how deregulation of oil prices had broken the back of OPEC so that oil prices would be low; and stories from the mid-2000s about how the world was approaching "peak oil" production and oil prices would inevitably stay high. It's useful to remember that every story about price shifts comes with assumptions about other underlying conditions not changing--which means that such stories come with an expiration date.
Baumeister and Kilian go over oil prices since about 1970 in a much more systematic way. Here are a few of the points I took away from their article.
1) Even if economists have a pretty good understanding of how a market works in hindsight, that doesn't mean that economists or anyone else can predict what shocks will disrupt the market. For example, the long run-up in oil prices from the late 1990s up to the start of the Great Recession in 2008 is usually attributed at least in part to China's voracious hunger for energy products. In retrospect, that explanation looks fine. But China's economic growth started around 1980, and had proceeded for two decades already without spiking global oil prices. It wasn't obvious circa 1995 or 2000 that China's growth would kick into a higher and more energy-hungry gear. Economists aren't good at predicting when the global economy will surge or stagnate, so it shouldn't be a surprise that shifts in oil prices come as a surprise.
2) A rise in risks in the oil market, with concerns about higher future prices, will affect demand for inventories of oil--that is, if you think the price of oil is likely to rise in the future, you buy additional oil now before the price goes up. But when demand for inventories rises, this also pushes up the price of oil right away. Baumeister and Kilian note a number of historical cases (like 1979-80) where the actual production of oil doesn't change a lot, but concerns that the price might rise drove inventory demands that helped bring an actual price rise. It's perhaps worth saying that from an economic point of view, this dynamic is a useful one. If there's a real danger that the price of oil is going to be rising, it's useful for that information to be reflected in current prices, so that we can all start conserving on oil right away. The rise in inventory demand is the economic mechanism to translate some of the risk of higher future oil prices into current prices.
3) High prices for oil at one point in time will set off search for ways of conserving energy and for new energy resources. However, the outcomes of these activities are not fully predictable, and thus tend to surprise us. The oil price drop of the 1980s, for example, is in part traceable to energy conservation and new energy sources that made sense given the high energy prices of the 1970s. The hydraulic fracturing technologies for extracting oil and gas were pushed forward in response to the high energy prices of the early 2000s. These steps toward greater conservation and new technologies take an unpredictable amount of time, and when their effects arrive, it feels like a surprise.
3) High prices for oil at one point in time will set off search for ways of conserving energy and for new energy resources. However, the outcomes of these activities are not fully predictable, and thus tend to surprise us. The oil price drop of the 1980s, for example, is in part traceable to energy conservation and new energy sources that made sense given the high energy prices of the 1970s. The hydraulic fracturing technologies for extracting oil and gas were pushed forward in response to the high energy prices of the early 2000s. These steps toward greater conservation and new technologies take an unpredictable amount of time, and when their effects arrive, it feels like a surprise.
4) Baumeister and Kilian emphasize that different groups will often have different expectations about oil prices. For example, household are likely to assume that oil prices will stay more or less where they are: high will stay high, low will stay low. Governments often look to financial markets for the futures price of oil. Analysts of oil markets run more complex calculations that seek to capture underlying patterns of supply and demand. When expectations differ, whatever happens is going to come as a shock to someone.
I've argued in earlier posts that I don't expect a shortage of fossil fuels will drive the global price systematically much higher during the next few decades. But the history of oil prices in recent decades strongly suggests that even if the inflation-adjusted before-tax price of oil in, say, 2030 or 2040 isn't dramatically different from today, there will be some dramatic climbs and plunges in oil prices along the way.
I've argued in earlier posts that I don't expect a shortage of fossil fuels will drive the global price systematically much higher during the next few decades. But the history of oil prices in recent decades strongly suggests that even if the inflation-adjusted before-tax price of oil in, say, 2030 or 2040 isn't dramatically different from today, there will be some dramatic climbs and plunges in oil prices along the way.