What can LLMs offer the Systematic Manager?

L-L-Mentary, my dear Watson– GAM Systematic’s Chris Longworth discusses the rise of Large Language Models in the investment world, their strengths and limitations, and assesses how systematic managers can maximise the opportunities they present.

21 December 2023

Artificial intelligence (AI) models based on large language models (LLMs), such as ChatGPT and others that followed on from predecessor technology like IBM’s Watson, truly captured the popular imagination in 2023. They have also consistently made headlines and sparked wider debate – often about ethics as much as the AI itself – to an extent that few technologies have previously, certainly since the advent of the Internet. A key factor behind surging levels of interest is that everyone – from novice users to AI experts – can interact with these systems in a much more human-like fashion, compared with previous AI techniques.

Models can already play an important role

These models can already assist with many supplementary areas of the investment process. For example, they can be used to support the production of market commentary while LLM-based code completion tools can also help with programming and development work. In addition, LLMs have many direct applications in the finance sector. For example, simply asking a general-purpose language model for an investment idea will often return a suggested portfolio. This proposition may also be accompanied by a convincing investment thesis.

AI and machine learning have historically been the domain of the systematic manager. These terms encompass a wide variety of related techniques and tools, each suited for different applications. For example, ‘reinforcement learning’ attempts to determine an optimal course of action, when the likely payoff for each action may be unknown until such time as it is tried. This turns out to be very relevant to the task of choosing an optimal strategy for executing a set of trades. For other problems – such as forecasting the return of an asset over the next year – a key challenge is the lack of historical data. Here, approaches that can explicitly capture this uncertainty, such as Bayesian probabilistic models – which essentially aim to update our existing beliefs on the back of new evidence – can be effective.

Models and managers: more in common than you might imagine

It is interesting to compare the strengths and limitations of newer language-based techniques to the more traditional machine learning-based approaches commonly used by systematic managers. Perhaps surprisingly, some of the features of language-based models turn out to be more like those of discretionary managers. There are many possible explanations for this, although one intriguing possibility is that recent advances have the potential to lead to greater convergence between systematic and discretionary investment styles.

Different approaches to the same problem

Although both systematic and discretionary managers attempt to solve the same underlying investment problem, the approaches they take often vary considerably, driven by the strength and limitations of each style. For example, traditionally, the most important input into systematic portfolios has been numerical data, such as market prices and volumes. There has been recent success from incorporating a broader range of alternative data inputs, such as earnings transcripts, into systematic models. However, this is normally achieved by pre-processing the data to extract numerical features, such as sentiment indicators. These can then be directly included into the underlying numerical models. In contrast, discretionary managers often work directly with textual data, such as company reports or economic forecasts, aggregating together a variety of sources – aiming, among other things, to help counter potential biases – to form a consistent view of the world from which an investment hypothesis can then be derived.

An important advantage of numerical models is that they can incrementally react as small changes in market conditions affect the expected pay-off of any given trade. Similarly, approaches based on probabilistic models can often express some degree of confidence in the forecasts they make. This can be directly incorporated into risk management of the portfolio.

Counterintuitively, for LLMs words do come easy, numbers less so

While LLMs are very effective at integrating textual data to form a consistent view of the world, it turns out that, like humans, it can often be challenging for LLMs to process numerical data and perform accurate mathematical computation. For example, OpenAI’s GPT4 model has been shown to deliver human-level exam performance across a wide variety of subjects, including art history and wine tasting. However, when applied to the AMC, a US high school mathematics exam, it was only able to perform as well as the bottom 10% of students. One consequence of how they work is that LLMs often struggle to capture some of the benefits of current systematic approaches that are reliant on numerical computation, such as incremental reaction to new information. This is key to effective risk management but is also much harder for language-based techniques to deliver.

Balancing diversification and risk factors

Another common feature of systematic investment styles tends to be broad portfolios holding diversified positions across many markets. This is because once a model is developed, it is often relatively straightforward to reapply that same model to other assets for which similar data is available. This enables the manager to benefit from portfolio diversification without a significant increase in workload. In contrast, discretionary investment decisions are often made on the basis of a detailed study of an individual market, with the result that the process can then be harder to transfer to other securities in a different market. This generally leads to portfolios containing fewer overall positions, but with a higher degree of conviction in each. In this case, LLMs often operate more like the discretionary manager, as they can be effective at sifting through large amounts of related information to suggest a trading hypothesis that is highly specific to each market.

Show your working – tracing decisions and handling ‘what-if’ scenarios

Another important requirement for managers is the ability to explain the rationale behind trading decisions. Despite systematic investment’s 'black box' reputation, one advantage is that all trades and adjustments to positions can ultimately be traced back to the data fed into the model. Similarly, systematic models are generally well-suited to answer questions about ‘what-if’ scenarios. By feeding hypothetical future data into a model, a manager can assess how it may react across a range of different potential market environments.

Keeping it real – the need for robust reasoning from models

There has been significant research effort aimed at improving the interpretability of LLMs. For example, ‘chain-of-thought’ approaches encourage the model to provide a step-by-step breakdown of its answers. This has been shown to both improve the quality of the answer, as well as provide a mechanism for a user to understand the model’s reasoning. However, in many cases it is not clear that this breakdown actually corresponds to the internal process originally used to generate the answer, instead acting as a retrospective justification. The presence of ‘hallucinations’ in model answers – seemingly plausible statements that are in fact entirely erroneous – also remains an ongoing concern.

Despite these limitations, the rise of new language-based techniques represents a clear opportunity for the systematic manager. Systematic and discretionary investment styles have historically been highly complementary, with the broadly diversified and incremental approach of the systematic manager contrasting with the more focused and conviction-led approach of the discretionary investor. Similarly, models that have the potential to capture some of the unique advantages of the discretionary approach are likely to combine more effectively with current numerical techniques. Ultimately, this offers the potential of a unified approach that can incorporate the strengths of both for the benefit of our clients.