Three steps to meaningful, actionable performance reviews
We’ve seen hundreds of investment performance reviews over the years. Most were a waste of time.
The typical investment performance review to an insurance company board is a perfunctory, boring recap of statistics; everyone politely waits until it’s time for something interesting — like the investment manager‘s favorite stock picks or market forecasts.
Consider the equity manager assessment of a recent performance review: The manager underperformed the S&P 500 Index by 3.3 percent in the latest year, outperformed by 0.3 percent annualized over the three-year time period, and underperformed by 1.4 percent annualized over the latest five-year time period, but with lower volatility than the index. What conclusions can be drawn from this type performance recap? None. 1
So why review investment performance in the first place?
The primary function of prudent oversight — and the first responsibility of every board member — is to continually ask the question: “Is everything still OK ?”
Effective performance evaluation is the only means to answer that most critical oversight question. An effective performance review is actionable. It provides the necessary data and context in sufficient detail to determine either 1) everything is still OK, or 2) there is a problem – and we’ve identified it.
Problem: Insufficient Context
Every investment strategy, portfolio and investment manager is subject to randomness and will underperform the relevant benchmark index during some time periods. The question then will be: “Is this underperformance an indication of a problem, or is it simply randomness?” If that question cannot be answered, the review was meaningless.
The unspoken difficulty in evaluating investment performance is that the distribution of randomness is significantly greater than the distribution of investment skill. The signal is small, the noise is large. The result is that it is very difficult to distinguish skill – positive or negative – from randomness. In fact, when evaluating an equity manager’s performance relative to a market index it takes between 40 and 70 years until there is sufficient data to infer a statistically meaningful conclusion regarding manager skill (see, for example: Rosenberg, Kritzman or Beebower 2). That is likely too late.
How much would your manager need to underperform the index to cause you to make a change?
Because the typical insurance company review compares performance solely to market indexes, results are neither meaningful nor actionable. Board members are bored for a good reason.
Step 1: Additional Data
Peer Universes – distributions of returns of “like-type” portfolios – responds to the lack of data context helps distinguish between a potential problem and normal randomness. For example, consider an equity manager whose annual return is six percentage points below his benchmark index.
In some years six percentage points below the market index equates to the bottom six percent of all similar equity managers, but in other years that same performance relative to the index is above the 35th percentile. Whether a manager is in the bottom 50 percent or the bottom five percent of other equity managers suggests very different conclusions. Without the added context of peer universes, the distinction is lost.
Survivor Bias
While peer universe data is a critical component of a useful performance review, it is also important that peer universes are comprised of truly “like-type” portfolios. Property-casualty company investment returns should be compared to those of other property-casualty company portfolios and not to distributions of mutual fund or pension fund returns. Pension funds are not subject to taxes and both pension and mutual funds are subject to very different constraints than insurance companies. Neither represent apples-to-apples comparisons.
But a more significant problem is survivor bias. Investment management firms often discontinue funds with poor performance. The returns of these funds cannot be included in multiyear return universes, resulting in an upward bias of return distributions over time. Numerous studies of the impact of survivor bias on peer return distributions show a bias in median returns of two to five percentage points over three- and five-year time periods. Survivor bias can be significant enough to eliminate the value of peer universe comparisons for multi-year periods.
A distinct advantage of property-casualty, health, or life, peer universes is that insurance companies are not discontinued due to poor relative investment performance and, as a result, return distributions calculated at the insurer level do not suffer from survivor bias. Insurance companies are in an ideal position to profit from peer universes in their performance review.
Step 2: A Question of Style
When performance is too low (or too high) relative to the peer universe, the next level of analysis is required.
A deeper analysis of performance is required only for periods of extreme relative performance.
A manager in the bottom decile in a particular time period may or may not be a problem, and if there is a problem it may be the manager or it may be a problem with understanding between the manager and the board. While there is no magic cutoff, in general, performance in the bottom ten percent (or in the extreme top of the distribution, by the way 3) suggests more investigation is necessary.
Performance evaluation should be done with a hierarchical approach. If the manager in all time periods reviewed is within some range within the distribution of peer returns – say above the tenth percentile and below the 95th – deeper analysis is less critical. But if performance is below (or above) a certain threshold in a particular time period, the next level of investigation should be undertaken.
Equity managers, for example, tend to focus on different sectors of the overall equity market — large capitalization stocks, small cap stocks, growth stocks, value stocks, core (the entire equity market), sector rotator (aka, a core manager with extra alpha risk) and international stocks (more separate asset class than different y style). Poor relative performance may be a reflection of an equity style that is “out-of-favor” rather than a problem manager. The next step in the review process is to define the manager’s style and then evaluate his performance within a distribution of returns of managers with the same style. Equity manager style is defined quantitatively either by regressing the manager’s returns over time against market style indexes or, for a fixed point in time, by analyzing the style of individual portfolio holdings.
A manager in the bottom ten percent of all equity managers when his particular style is out-of-favor may be well above median when considered within a distribution of managers with the same equity style. A manager performing well within his style universe is clearly not a problem unless the manager’s style is other than expected, or has changed substantially over time, in which case there may be a manager problem or a communication problem.
A manager in the bottom (or extreme top) of his style distribution may indicate a problem manager or it may be a manager with a very concentrated portfolio (with the associated increased exposure to randomness and resulting high volatility of relative performance). The latter manager would be appropriate within some portfolio structures, but problematic within others.
Step 3: Identify Manager “Bets”
The final level in the review process of a potential problem manager is a detailed factor analysis of portfolio holdings at multiple points in time over the period of poor performance. Factor analysis is a statistical modeling method in which incremental return relative to a neutral benchmark is explained by decomposing residual portfolio variance into contributions from individual positions. This analysis identifies the precise exposure bets that lead to the poor results. With this information in advance of a manager meeting, the board can determine whether the bets were intentional, with a satisfactory explanation, or unintentional — suggesting a change is in order.
The only way board members can fulfill their oversight responsibility by adequately answering the question “Is everything still OK?” is with effective performance reviews. It’s time for insurance company boards to insure that their investment performance review is worth their attention.
- See www.peeranalytics.com/index-limitations/
- FARQUHAR, T., ROSENBERG, B. and RUDD, A. (1982), Factor-Related and Specific Returns of Common Stocks: Serial Correlation and Market Inefficiency. The Journal of Finance
- Performance in the top tail of the distribution may indicate a manager exposing the portfolio to unacceptable risk.