A really good academic paper from guys (and girl) behind Quantopian:

Authors: Wiecki, Campbell, Lent, Stauth

Title: All that Glitters Is Not Gold: Comparing Backtest and Out-of-Sample Performance on a Large Cohort of Trading Algorithms

Link: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2745220

Abstract:

When automated trading strategies are developed and evaluated using backtests on historical pricing data, there exists a tendency to overfit to the past. Using a unique dataset of 888 algorithmic trading strategies developed and backtested on the Quantopian platform with at least 6 months of out-of-sample performance, we study the prevalence and impact of backtest overfitting. Specifically, we find that commonly reported backtest evaluation metrics like the Sharpe ratio offer little value in predicting out of sample performance (R² < 0.025). In contrast, higher order moments, like volatility and maximum drawdown, as well as portfolio construction features, like hedging, show significant predictive value of relevance to quantitative finance practitioners. Moreover, in line with prior theoretical considerations, we find empirical evidence of overfitting – the more backtesting a quant has done for a strategy, the larger the discrepancy between backtest and out-of-sample performance. Finally, we show that by training non-linear machine learning classifiers on a variety of features that describe backtest behavior, out-of-sample performance can be predicted at a much higher accuracy (R² = 0.17) on hold-out data compared to using linear, univariate features. A portfolio constructed on predictions on hold-out data performed significantly better out-of-sample than one constructed from algorithms with the highest backtest Sharpe ratios.

Notable quotations from the academic research paper:

"For the first time, to the best of our knowledge, we present empirical data that can be used to validate theoretical and anecdotal claims about the ubiquity of backtest overfitting and its impact on algorithm selection. This was possible by having access to a unique data set of 888 trading algorithms developed and tested by quants on the Quantopian platform. Analysis revealed several results relevant to the quantitative finance community at large – practitioners and academics alike.

Most strikingly, we find very weak correlations between IS and OOS performance in most common finance metrics including Sharpe ratio, information ratio, alpha. This result provides strong empirical support for the simulations carried out by Bailey et al. [2014]. More specifically, it supports the assumptions underlying their simulations without compensatory market forces to be present which would induce a negative correlation between IS and OOS Sharpe ratio. It is also interesting to compare different performance metrics in their predictability of OOS performance. Highest predictability was achieved by using the Sharpe ratio computed over the last IS year. This feature was also picked up by the random forest classifier as the most predictive feature.

Additionally, we find significant evidence that the more backtests a user ran, the bigger the difference between IS and OOS performance – a direct indication of the detrimental effect of backtest overfitting. This observed relationship is also consistent with Bailey et. al's [2014] prediction that increased backtesting of multiple strategy variations (parameter tuning) would increase overfitting. Thus, our results further support the notion that backtest overfitting is common and wide-spread. The observed significant positive relationship between amount of backtesting and Sharpe shortfall (IS Sharpe – OOS Sharpe) provides support for a Sharpe ratio penalized by the amount of backtesting
(e.g. the "deflated Sharpe ratio" by Bailey & Lopez de Prado [2014]). An attempt to calibrate such a backtesting penalty based on observed data is a promising direction for future research.

Together, these sobering results suggest that a reported Sharpe ratio (or related measure) based on backtest results alone can not be expected to prevail in future market environments with any reasonable confidence.

While the results described above are relevant by themselves, overall, predictability of OOS performance was low (R² < 0.25) suggesting that it is simply not possible to forecast profitability of a trading strategy based on its backtest data. However, we show that machine learning together with careful feature engineering can predict OOS performance far better than any of the individual measures alone. Using these predictions to construct a portfolio of strategies resulted in competitive cumulative OOS returns with a Sharpe ratio of 1.2 that is better than most portfolios constructed by randomly selecting strategies. While it is difficult to extract an intuition about how the Random Forest is deriving predictions, we have provided some indication of which features it deems important. It is interesting to note that among the most important features are those that quantify higher-order moments including skew and tail-behavior of returns (tail-ratio and kurtosis). Together, these results suggest that predictive information can indeed be extracted from a backtest, just not in a linear and univariate way. It is important to note that we cannot yet claim that this specific selection mechanism will work well on future data as the machine learning algorithm might learn to predict which strategy type worked well over the specific OOS time-period most of our algorithms were tested on (for a more detailed discussion of this point, see the limitations section). However, if these results are reproducible on an independent data set or the strategies identified continue to outperform the broad cohort over a much longer time frame, it should be of high relevance to quantitative finance professionals who now have a more accurate and automatic tool to evaluate the merit of a trading algorithm. As such, we believe our work highlights the potential of a data scientific approach to quantitative portfolio construction as an alternative to discretionary capital allocation."

Are you looking for more strategies to read about? Check http://quantpedia.com/Screener

Do you want to see performance of trading systems we described? Check http://quantpedia.com/Chart/Performance

Do you want to know more about us? Check http://quantpedia.com/Home/About

A related paper has been added to:

#21 – Momentum Effect in Commodities

Authors: Bianchi, Drew, Fan

Title: Microscopic Momentum in Commodity Futures

Link: https://www120.secure.griffith.edu.au/research/file/0a572b95-132b-419d-9a71-310420fad143/1/2015-10-microscopic-momentum-in-commodity-futures.pdf

Abstract:

Conventional  momentum strategies  rely  on 12 months of past returns for  portfolio formation. Novy-Marx  (2012)  shows that the intermediate  return  momentum strategy formed  using only twelve to  seven  months  of returns prior  to  portfolio  formation significantly outperforms the recent return momentum formed using six to two month returns  prior. This  paper proposes a more granular strategy  termed  ‘microscopic momentum’, which further decomposes the intermediate and recent return momentum into  single-month  momentum components. The  novel  decomposition  reveals that a microscopic  momentum  strategy  generates  persistent  economic profits  even  after controlling   for   sector-specific   or month-of-year   commodity   seasonality   effects. Moreover, we show that the intermediate return  momentum in the commodity  futures must  be  considered largely  illusory, and all 12  months of  past  returns play  important  roles in determining the conventional momentum profits.

Notable quotations from the academic research paper:

"In  this  study,  we  propose  a  third  type  of  momentum  strategy  termed Microscopic Momentum, which further decomposes the recent (6 to 2 months) and intermediate (12 to 7 months)  momentum  of  Novy-Marx  (2012)  into  12  single-month  individual momentum components. As a consequence of the decomposition, we are able to take a glimpse at momentum profits under a month-by-month, microscopic scale. For the first time,  this  novel  approach  not  only  reveals  a  striking  new  discovery  of  a  momentum based anomaly, but also allows us to pinpoint whether specific months in the past play a more significant role in determining conventional and echo momentum profits, hence it offers fresh insights into our understanding of momentum in commodity futures.

The proposed granular analysis of microscopic momentum makes four major contributions to the commodity futures literature. First, in the commodity futures markets, the ‘11,10 microscopic momentum strategy’, constructed using the 11 to 10-month return prior to formation, produces an annualised average return of 14.74% with strong statistical significance. The superiority of the 11,10 strategy is not driven by sector-specific nor month-of-year commodity seasonality effects and is robust across sub-periods  and  out-of-sample  analysis. 

Second,  when  the  RNM  echo  momentum  is regressed  against  its  microscopic  components,  RNM  intermediate  momentum  can  be completely   subsumed  by   the  11,10  microscopic  momentum.  Thus,  the  superior performance  of  intermediate  momentum  claimed by  RNM may  be  an  illusion  created by  the  11,10  microscopic  momentum.  This  implies  that  for  tactical  asset  allocation decisions,  CTAs  and  commodity  fund  managers  must  not  consider  intermediate momentum  as  a  viable  substitute  for  conventional  momentum  strategies. Instead,  the 11,10  microscopic  strategy,  which  offers  similar  profits  in  magnitude  but  unique dynamics of returns to conventional strategies, may be a feasible alternative.

Third, around 77% of the variation of returns in the JT conventional momentum strategy can be explained by its microscopic decomposition. However, since no dominance is found on any individual month, all past months are found to be important in determining the conventional commodity momentum profits.

Fourth, echo and microscopic  momentum  is  partially  related  to  the  U.S.  cross-sectional  equity momentum and the returns from broad commodity futures, but is not related to stocks, bonds,  foreign currency risks and macroeconomic  conditions. Consistent with  Asness et. al., (2013), this finding implies that there may  indeed be a  common component in momentum across asset classes."

Are you looking for more strategies to read about? Check http://quantpedia.com/Screener

Do you want to see performance of trading systems we described? Check http://quantpedia.com/Chart/Performance

Do you want to know more about us? Check http://quantpedia.com/Home/About

A related paper has been added to:

#129 – Dollar Carry Trade

Authors: Shehadeh, Erdos, Li, Moore

Title: US Dollar Carry Trades in the Era of 'Cheap Money'

Link: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2765552

Abstract:

In this paper, we employ a unique dataset of actual US dollar (USD) forward positions against a number of currencies taken by so-called Commodity Trading Advisors (CTAs). We investigate to what extent these positions exhibit a pattern of USD carry trading or other patterns of currency trading over the recent period of the ultra-loose US monetary policy. Our analysis indeed shows that USD positions against emerging market currencies are characterised by a pattern of carry trading. That is, the USD, as the lower yielding currency, is associated with short positions. The payoff distributions of these positions, moreover, are found to have positive Sharpe ratios, negative skewness and high kurtosis. On the other hand, we find that USD positions against other advanced country currencies have a pattern completely opposite to carry trading which is in line with uncovered interest parity trading; that is, the lower (higher) yielding currency is associated with long (short) positions.

Notable quotations from the academic research paper:

"In the wake of the 2007-2008 financial crisis, many countries, especially developed countries including the USA, have adopted unconventional loose monetary policies with the purpose of stimulating their sluggish and unstable economies. This period is termed in the financial press as “the era of cheap money”. On the other hand, other countries, especially emerging markets, have maintained relatively high interest rates over the same period. Because of the potential impact of these effects on the trading decisions of the FX traders, it is worthwhile to consider currency trading in general and USD carry trading in particular over the sample period of the paper.

In light of this, the crux of the paper is to analyse our dataset of the USD forward positions to find out to what extent they show characteristics of USD carry trading or another trading strategy over the recent period of record-low US interest rates. In other words, we investigate whether these positions exhibit a response to the very low US interest rates by having a pattern of USD carry trading or other patterns of trading strategies can be identified across different currency markets. The distinctive feature of this study is that we have access to a dataset of daily-aggregated USD forward positions against a number of advanced and emerging currencies. It is collected from a Swedish investment specialist, Risk & Portfolio Management AB (RPM) which is a fund of hedge funds investing in Managed Futures strategies which are also known as Commodity Trading Advisors (CTAs). CTAs engage in various strategies like trend-following, short-term trading, and global macro that often employs carry trading as a sub-strategy. By exploiting and analysing our private dataset we find significant long-run equilibrium relationships which directly relate the USD forward positions to its forward premium. The relationships point to different trading strategies for emerging and advanced market currencies. For emerging currencies, we find that these relationships are consistent with carry trading. That is the lower yielding currency (the USD) is associated with short positions and vice versa. This carry trading pattern of forward shorting lower-yielding currency is induced by the expectations that the lower-yielding currency will not actually appreciate on average as much as the forward rate implies, or even it will depreciate. This in turn implies a profit on average at maturity. On the other hand, we find that the reverse holds between the USD and advanced currencies. In other words, we find a pattern of “fundamentals-based” trading consistent with the uncovered interest parity condition. That is, the lower (higher) yielding currency is associated with more long (short) positions. These anti-carry positions can reflect the unattractiveness of the advanced currencies-USD carry trading due to the increased uncertainty and narrow interest differentials for these markets over the period following the recent crisis.

Given that our data set is collected from FX traders which are mainly trend-followers, these results of the different trading strategies for emerging and advanced market currencies shed some light on the trading behaviour of this group of the FX market participants. On the one hand, the characteristics of carry trades for EM currencies which involve long high-interest currency aginst the low-interest currency reflect a trend-following strategy which is based on
the expectations that high-interest currency is going to appreciate -i.e. based on the appreciation trend of the high-interest rate currency. On the other hand, the characteristics of “fundamentals-based” trades for AM currencies which involve long low-interest currency against high-interest currency reflect a trend-following strategy which is based on the expectations that low-interest rate currency is going to appreciate –i.e. based on the appreciation trend of the low-interest rate currency. This is in line with the heterogeneous agents model developed by Spronk et al. (2013). The model demonstrates that depending on the dominant trend in the market, FX trend-followers can be in the same line of either carry traders or fundamentalists. In this sense, our results provide some insights into these features of the FX trend-following traders."

Are you looking for more strategies to read about? Check http://quantpedia.com/Screener

Do you want to see performance of trading systems we described? Check http://quantpedia.com/Chart/Performance

Do you want to know more about us? Check http://quantpedia.com/Home/About

A related paper has been added to:

#52 – Asset Growth Effect

Authors: Prombutr, Phengpis, Lam

Title: Anatomy of the Mispricing Theory: Evidence from Growth Anomalies

Link: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2746998

Abstract:

This paper investigates corporate growth anomalies in asset pricing from behavioral perspectives. Cross-sectional analyses indicate that a long-term 3-year investment growth is statistically significant in explaining subsequent stock returns, but the first 1-year growth that is closest to the formation is priced by investors the most, followed by the second and third ones, monotonically. We find that the evidence is driven by myopic mispricing in that investors tend to put more weights on recent information since the evolution of the firm’s prospects around the formation year consistently shows that the growth closest (farthest) to the formation has the most (least) severe mispricing. Further investigations show that the mispricing evolution is directly amplified by limits to arbitrage and that benchmark-adjusted returns on short positions are affected more than those on long positions. However, the farther growth is less sensitive to the limit-to-arbitrage because of the extrapolation is myopic. The asset growth anomaly also shows the same pattern as the investment growth anomaly.

Notable quotations from the academic research paper:

"With respect to the extrapolative mispricing theory, we hypothesize that the growth effect cannot happen without mispricing. The growth premium coincides with the disappointment from extrapolating the firm’s prospects into the future. Along with the extrapolative mispricing theory, we investigate two additional behavioral theories: limits to arbitrage and arbitrage asymmetry. We realize that rational and behavioral theories cannot be differentiated (See Lin and Zhang, (2013)) because the characteristic-based factor models are linear approximations of investment returns. Hence, if the three behavioral theories (extrapolative mispricing, limits to arbitrage and arbitrage asymmetry) are connected and contemporaneously supported by the data, our paper will strengthen the validity of behavioral explanations without disapproving the rational q-theory.

Importantly, in this paper, we expand an investigation into the growth anomaly by decomposing the growth measures. We decompose a long-term 3-year growth measure into three consecutive short-term 1-year growth measures based on the concept of term structure. With this decomposition, we can do several anatomy tests on the corporate growth anomaly. The anatomy tests along with the evolution of mispricing and returns should be able to show more clearly if predictions of the above three behavioral theories are consistent with empirical evidence. Prior studies have not decomposed long-term growth measures into short-term components or investigated the three behavioral theories concurrently.

We find, based on cross-sectional regressions, that the explanatory power of the 3-year long-term growth (IG13) on stock returns is statistically significant and actually comes from its most recent 1-year growth component (IG1, one year closest to the formation period). The farther 1-year growth components (IG2 and IG3, two and three years away from the formation period, respectively) show monotonically diminishing explanatory power. These results indicate short memory of investors which are consistent with the myopic theory from the behavioral perspective. Cognitive biases such as representativeness state that investors tend to put too much weight on recent information (Kahneman and Tversky, 1974). Combining the myopic theory with the mispricing theory that investors extrapolate too much into the future about firms’ prospects and valuations for high and low growth firms, more severe extrapolative mispricing should exist closer to the portfolio formation year than farther distant years.

We empirically find that (1) long-term growth (IG13) demonstrates less extrapolative mispricing around the formation year than short-term growth that is measured right before the formation year (IG1), (2) compared to the IG1, the farther 1-year growth measures (IG2 and IG3, two and three years away from the formation period) show monotonic decreases in mispricing around the formation year, and (3) return performances associated with these growth measures show patterns that are consistent with the degrees of mispricing described in (1) and (2). Combined with the above cross-sectional regression findings, these results strongly suggest that the investment growth anomaly can be explained by the extrapolative myopic mispricing theory.

All of the above tests are redone using asset growth measures instead of investment growth measures. The results are similar and in fact even more pronounced. In sum, we conclude that the investment or asset growth effect is associated with mispricing. The mispricing is short-lived, so a long-term 3-year growth has the most severe mispricing near the formation year. Additionally, limits to arbitrage lead to more severe mispricing and a short position is affected more than a long position. However, the mispricing evolution tends to be less pronounced when the farther growth measures are used since investors are myopic."

Are you looking for more strategies to read about? Check http://quantpedia.com/Screener

Do you want to see performance of trading systems we described? Check http://quantpedia.com/Chart/Performance

Do you want to know more about us? Check http://quantpedia.com/Home/About

A new academic paper related to our previous blog-post (Benchmarking Commodity CTAs):

Authors: Blocher, Cooper, Molyboga

Title: Performance (and) Persistence in Commodity Funds

Link: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2658153

Abstract:

This study documents persistent, net-of-fees, alpha-generating commodity trading advisor funds focused on commodity investment (“Commodity Funds”). The baseline for performance measurement is a new benchmark model that includes factors established in the literature. A nonparametric bootstrap test establishes the existence of alpha that cannot be explained by luck. Performance persists 12 months out of sample and subsequently disappears. Such performance, without a reversal, indicates that persistent alpha is based in information about fundamentals, not fund flows or sentiment. These results are robust to data biases established in the literature.

Notable quotations from the academic research paper:

"To evaluate fund manager performance, we first implement and test a five-factor asset pricing model as a benchmark for commodity manager performance measurement. The model includes a market factor, a time series momentum factor, a spot basis factor, and high and low term premia factors. These factors are drawn from the extant literature, based in commodity fundamentals, and each has been shown separately to capture a risk premium embedded in commodity futures (e.g., Szymanowska et al. 2014, Bakshi, Gao Bakshi, and Rossi 2014, Moskowitz, Ooi, and Pedersen 2012). This factor model for commodity futures parallels Fama and French’s now ubiquitous model for publicly traded equities (now also with five factors, see Fama and French 2015).

We then use this five-factor model benchmark to identify commodity fund manager performance and persistence. First, we conduct a bootstrap analysis of the distribution of alpha t-statistics and find both top and bottom performers that cannot be explained by luck. Second, we find that both good and bad performance persists for approximately 12 months. Annualized alpha of the top performing quintile is 2.53%, while the same for the bottom performing  quintile is -1.94%. This performance persistence disappears after 12 months, but does not reverse. This nonreversal indicates that commodity fund manager performance is based on information and/or skill, rather than sentiment or other non-fundamental factor, which is often the case in mutual funds (Blocher 2015, Lou 2012).

"

Are you looking for more strategies to read about? Check http://quantpedia.com/Screener

Do you want to see performance of trading systems we described? Check http://quantpedia.com/Chart/Performance

Do you want to know more about us? Check http://quantpedia.com/Home/About

A related paper has been added to:

#21 – Momentum Effect in Commodities
#22 – Term Structure Effect in Commodities
#118 – Time Series Momentum Effect

Authors: Blocher, Cooper, Molyboga

Title: Benchmarking Commodity Investments

Link: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2744766

Abstract:

While much is known about the financialization of commodities, less is known about how to profitably invest in commodities. Existing studies of Commodity Trading Advisors (CTAs) do not adequately address this question because only 19% of CTAs invest solely in commodities, despite their name. We compare a novel four-factor asset pricing model to existing benchmarks used to evaluate CTAs. Only our four-factor model prices both commodity spot and term risk premia. Overall, our four-factor model prices commodity risk premia better than the Fama-French three-factor model prices equity risk premia, and thus is an appropriate benchmark to evaluate commodity investment vehicles.

Notable quotations from the academic research paper:

"The four factors in our model include a market factor, a time series momentum factor, and separate high and low term premia factors, sorted on commodity basis. These factors are drawn from the extant literature and based in commodity fundamentals, and each has been shown separately to capture a risk premium embedded in commodity futures, though never together in the form we propose.

We consider factors for each premium in turn, starting with the spot premium. We first include a market factor (MKT), which is an equally weighted average of all commodities’ one period spot return. Next, we include a momentum factor. We choose a time series momentum factor (TSMOM) as in Moskowitz, Ooi, and Pedersen (2012), which is the difference in return between an equally weighted portfolio of commodities with a positive return over the previous twelve months and one with a negative return over the previous twelve months. We next consider the term premium. To price the term premium, we choose two factors. First, we construct a high-term premium factor (Hterm) consisting of the average of the 2-month, 4-month, and 6-month realized term premia for the 10 commodities with above-median basis (as previously defined in the HML factor). We also construct a low-term premium factor (Lterm), computed the same way as Hterm, except using the 10 commodities with below-median basis.

Until now, benchmarking commodity investments has been inhibited by a lack of understanding of the drivers of risk premia. Recently, however, the literature has coalesced around a few key drivers of commodity risk premia, represented by the four factors in our model. Simultaneously, increased interest in commodity investment in the past decade combined with the poor performance of passive market indexes means sophisticated investors are more interested in evaluating the performance of active commodity fund managers. Financial advisors have even suggested that individuals include commodities in their personal asset allocation.5 Yet, to our knowledge, there is not a thoroughly tested and established benchmark to evaluate commodity fund managers or commodity ETFs.

While commodity investment often is included as a subset of the hedge fund/CTA literature, there are more similarities between commodity markets and equity markets than between commodities and hedge funds. Both commodities and equities are publicly traded with public closing prices, providing clear, end-of-day portfolio values. Both have a clearly identified regulatory body (the CFTC and SEC). Both represent a defined investment set within which a manager (Commodity Fund or Mutual Fund) must choose either long or short positions. Given these similarities, our paper can be seen as establishing a factor model benchmark for Commodity Funds in the same way that Fama and French (1992, 2015) have established a benchmark for Mutual Funds."

Are you looking for more strategies to read about? Check http://quantpedia.com/Screener

Do you want to see performance of trading systems we described? Check http://quantpedia.com/Chart/Performance

Do you want to know more about us? Check http://quantpedia.com/Home/About