Natural Language Processing and Forecasting of Energy Prices-A Massively Missed Opportunity for our Industry

Boyd, Head of Commercial and CX Data Sciences, The AES

Natural language processing-while you may not know it by its official name, this field of artificial intelligence is no doubt an inescapable part of your daily life. Whether it be asking Alexa the temperature, dictating to Siri a text message, verbally providing your four-digit pin after stating the reason you’re calling the bank, all before ever speaking with a live agent, even your spam filter and spell check, these are but a handful of the countless examples of natural language processing’s pervasiveness in everyone’s everyday life.

Natural language processing, or NLP, is a field of artificial intelligence that enables computers to understand, interpret and manipulate human language. Combining the power of computer science and computational linguistics to make human language intelligible to machines, NLP enables your computer to extract meaning from text and speech and respond or react appropriately. The power and potential use of NLP areimmense. Large-scale analysis can be automatically performed to glean insights and meaning from massive amounts of unstructured text data - think social media comments, customer support tickets, online reviews, news reports, etc. Routine processes can be automated in real-term to efficiently and accurately sort and route information 24 hours a day and without the need to ever interact with a human. The sentiment of your customers about your company and their interactions with it can be quickly gleaned.

For an industry ripe with complexity and industry-specific language, NLP is a boon as it can be tailored to the energy industry’s many intricacies and acronyms. This powerful capability, however, is not being fully advantaged by us. Rather, our use of NLP in the industry, and particularly utilities, is in sticking with the obvious and, dare I say it, mundane—interactive voice response in call centers, capturing customer satisfaction, automating the routing of certain requests or documents.

While these are all undoubtedly valuable, we are missing a massive trick in one of the most foundational elements of our business – forecasting prices. For an industry so dependent on managing energy price risk, we are woefully behind. NLP’s capability to forecast prices has already been discovered and deployed by the financial industry, with this success being chronicled in academic journals and via financial blog gurus alike.

That NLP is a natural transition in the mainstream forecasting journey is not surprising. To illustrate why, allow a brief sidetrack in the form of a history lesson.

Prior to the 1980s, economic theory, and with it finance, a subfield of economics, was dependent on the fundamental psychological assumption that individuals and firms, or what we’ll refer to as economic agents, are rational. When new information arrives, economic agents immediately update beliefs about future outcomes and do so correctly. Choices are thus always “perfect” in that they maximize utility, a theoretical concept encapsulating “desired” concepts such as value, happiness, preferences, profit, etc.

This rationalism results in prices that optimally balance via the laws of supply and demand. While frictions exist (e.g., it takes time to adjust production), the fundamentals of supply and demand combined with rationality will always prevail.

In the 1980s, this concept of rationality was turned on its head with the acknowledgment that economic agents are not always rational - beliefs are not fully updated in a rational manner. Correspondingly, economic agents have cognitive limits in that they are unable to process all of the information relevant to their decisions.

Choices are thus always ‘perfect’ in that they maximize utility, a theoretical concept encapsulating ‘desired’ concepts, such as value, happiness, preferences, profit, etc.

Fast forward four decades to 2021, where information, whether relevant or not, is ever-present and bombards us at every moment. According to the latest research by We are Social & Hootsuite, the average internet user now spends 6 hours and 43 minutes online everyday - the world’s 4.5 billion internet users (nearly 60 percentof the world’s population) spent a cumulative 1.25 billion years online in 2020, with more than one-third of that time spent using social media, a main source of information. Information now available at every turn includes company earnings reports, news about firms, political news, calls to boycott or increase use, etc.

With the cognitive limits surely met by this plethora of information, that which we do have access to is correspondingly biased thanks to algorithmic decision-making, which translates our “views” of one piece into what we see going forward. With an ever integrated data world, this bias extends to what we’ve bought, watched, commented on, had our friends say about us in posts, and even verbal asked (i.e., think Siri and Alexa, all of which are gathering data about you and your choices). Our information space thus becomes narrower despite the total universe being ever larger. That we are making decisions freely and with complete rationality thus is highly unlikely—the volume of information and the portion which we are exposed to make this impossible.

That prices are immune from this is thus also impossible. This is precisely what we’ve seen in finance, with sentiment analysis being a clear winner in predicting price movements. NLP has shown that sentiment influences abnormal movements in commodity prices such as oil significantly, with news sentiment’s influence vastly exceeding the influence of fundamentals. Interestingly, while abnormal movements are driven mostly by daily news, these short-term deviations are not a single dimension and flow through to index futures. Spot market sentiment and future’s market sentiment have more predictive power on short-term components of prices than long-term components of prices. Another major finding is that the influence of sentiment appears to be asymmetric such that returns are mostly driven by negative news rather than positive.

Given the vast amount of unstructured textual content available in today’s market, investors often struggle to separate noise from signal and determine which of these events has the potential to impact commodity prices. However, by utilizing the latest advances in Natural Language Processing (NLP), market participants have the ability to detect high-impact events and identify commodity price triggers in real-time. For an industry so dependent on energy prices and managing fluctuations, we would be well served to follow quickly in the footsteps of finance and incorporate NLP into our price forecasting approaches.