Quantopian's Academic Paper About In vs. Out-of-Sample Performance of Trading Algorithms Wednesday, 4 May, 2016

A really good academic paper from guys (and girl) behind Quantopian:

Authors:
Wiecki, Campbell, Lent, Stauth

Title: All that Glitters Is Not Gold: Comparing Backtest and Out-of-Sample Performance on a Large Cohort of Trading Algorithms

Link: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2745220

Abstract:

When automated trading strategies are developed and evaluated using backtests on historical pricing data, there exists a tendency to overfit to the past. Using a unique dataset of 888 algorithmic trading strategies developed and backtested on the Quantopian platform with at least 6 months of out-of-sample performance, we study the prevalence and impact of backtest overfitting. Specifically, we find that commonly reported backtest evaluation metrics like the Sharpe ratio offer little value in predicting out of sample performance (R² < 0.025). In contrast, higher order moments, like volatility and maximum drawdown, as well as portfolio construction features, like hedging, show significant predictive value of relevance to quantitative finance practitioners. Moreover, in line with prior theoretical considerations, we find empirical evidence of overfitting – the more backtesting a quant has done for a strategy, the larger the discrepancy between backtest and out-of-sample performance. Finally, we show that by training non-linear machine learning classifiers on a variety of features that describe backtest behavior, out-of-sample performance can be predicted at a much higher accuracy (R² = 0.17) on hold-out data compared to using linear, univariate features. A portfolio constructed on predictions on hold-out data performed significantly better out-of-sample than one constructed from algorithms with the highest backtest Sharpe ratios.

Notable quotations from the academic research paper:

"For the first time, to the best of our knowledge, we present empirical data that can be used to validate theoretical and anecdotal claims about the ubiquity of backtest overfitting and its impact on algorithm selection. This was possible by having access to a unique data set of 888 trading algorithms developed and tested by quants on the Quantopian platform. Analysis revealed several results relevant to the quantitative finance community at large – practitioners and academics alike.

Most strikingly, we find very weak correlations between IS and OOS performance in most common finance metrics including Sharpe ratio, information ratio, alpha. This result provides strong empirical support for the simulations carried out by Bailey et al. [2014]. More specifically, it supports the assumptions underlying their simulations without compensatory market forces to be present which would induce a negative correlation between IS and OOS Sharpe ratio. It is also interesting to compare different performance metrics in their predictability of OOS performance. Highest predictability was achieved by using the Sharpe ratio computed over the last IS year. This feature was also picked up by the random forest classifier as the most predictive feature.

Additionally, we find significant evidence that the more backtests a user ran, the bigger the difference between IS and OOS performance – a direct indication of the detrimental effect of backtest overfitting. This observed relationship is also consistent with Bailey et. al's [2014] prediction that increased backtesting of multiple strategy variations (parameter tuning) would increase overfitting. Thus, our results further support the notion that backtest overfitting is common and wide-spread. The observed significant positive relationship between amount of backtesting and Sharpe shortfall (IS Sharpe - OOS Sharpe) provides support for a Sharpe ratio penalized by the amount of backtesting
(e.g. the "deflated Sharpe ratio" by Bailey & Lopez de Prado [2014]). An attempt to calibrate such a backtesting penalty based on observed data is a promising direction for future research.

Together, these sobering results suggest that a reported Sharpe ratio (or related measure) based on backtest results alone can not be expected to prevail in future market environments with any reasonable confidence.

While the results described above are relevant by themselves, overall, predictability of OOS performance was low (R² < 0.25) suggesting that it is simply not possible to forecast profitability of a trading strategy based on its backtest data. However, we show that machine learning together with careful feature engineering can predict OOS performance far better than any of the individual measures alone. Using these predictions to construct a portfolio of strategies resulted in competitive cumulative OOS returns with a Sharpe ratio of 1.2 that is better than most portfolios constructed by randomly selecting strategies. While it is difficult to extract an intuition about how the Random Forest is deriving predictions, we have provided some indication of which features it deems important. It is interesting to note that among the most important features are those that quantify higher-order moments including skew and tail-behavior of returns (tail-ratio and kurtosis). Together, these results suggest that predictive information can indeed be extracted from a backtest, just not in a linear and univariate way. It is important to note that we cannot yet claim that this specific selection mechanism will work well on future data as the machine learning algorithm might learn to predict which strategy type worked well over the specific OOS time-period most of our algorithms were tested on (for a more detailed discussion of this point, see the limitations section). However, if these results are reproducible on an independent data set or the strategies identified continue to outperform the broad cohort over a much longer time frame, it should be of high relevance to quantitative finance professionals who now have a more accurate and automatic tool to evaluate the merit of a trading algorithm. As such, we believe our work highlights the potential of a data scientific approach to quantitative portfolio construction as an alternative to discretionary capital allocation."


Are you looking for more strategies to read about? Check http://quantpedia.com/Screener

Do you want to see performance of trading systems we described? Check http://quantpedia.com/Chart/Performance

Do you want to know more about us? Check http://quantpedia.com/Home/About

A New Analysis of Commodity Momentum Strategy Tuesday, 26 April, 2016

A related paper has been added to:

#21 - Momentum Effect in Commodities

Authors: Bianchi, Drew, Fan

Title: Microscopic Momentum in Commodity Futures

Link: https://www120.secure.griffith.edu.au/research/file/0a572b95-132b-419d-9a71-310420fad143/1/2015-10-microscopic-momentum-in-commodity-futures.pdf

Abstract:

Conventional  momentum strategies  rely  on 12 months of past returns for  portfolio formation. Novy-Marx  (2012)  shows that the intermediate  return  momentum strategy formed  using only twelve to  seven  months  of returns prior  to  portfolio  formation significantly outperforms the recent return momentum formed using six to two month returns  prior. This  paper proposes a more granular strategy  termed  ‘microscopic momentum’, which further decomposes the intermediate and recent return momentum into  single-month  momentum components. The  novel  decomposition  reveals that a microscopic  momentum  strategy  generates  persistent  economic profits  even  after controlling   for   sector-specific   or month-of-year   commodity   seasonality   effects. Moreover, we show that the intermediate return  momentum in the commodity  futures must  be  considered largely  illusory, and all 12  months of  past  returns play  important  roles in determining the conventional momentum profits.

Notable quotations from the academic research paper:

"In  this  study,  we  propose  a  third  type  of  momentum  strategy  termed Microscopic Momentum, which further decomposes the recent (6 to 2 months) and intermediate (12 to 7 months)  momentum  of  Novy-Marx  (2012)  into  12  single-month  individual momentum components. As a consequence of the decomposition, we are able to take a glimpse at momentum profits under a month-by-month, microscopic scale. For the first time,  this  novel  approach  not  only  reveals  a  striking  new  discovery  of  a  momentum based anomaly, but also allows us to pinpoint whether specific months in the past play a more significant role in determining conventional and echo momentum profits, hence it offers fresh insights into our understanding of momentum in commodity futures.

The proposed granular analysis of microscopic momentum makes four major contributions to the commodity futures literature. First, in the commodity futures markets, the ‘11,10 microscopic momentum strategy’, constructed using the 11 to 10-month return prior to formation, produces an annualised average return of 14.74% with strong statistical significance. The superiority of the 11,10 strategy is not driven by sector-specific nor month-of-year commodity seasonality effects and is robust across sub-periods  and  out-of-sample  analysis. 

Second,  when  the  RNM  echo  momentum  is regressed  against  its  microscopic  components,  RNM  intermediate  momentum  can  be completely   subsumed  by   the  11,10  microscopic  momentum.  Thus,  the  superior performance  of  intermediate  momentum  claimed by  RNM may  be  an  illusion  created by  the  11,10  microscopic  momentum.  This  implies  that  for  tactical  asset  allocation decisions,  CTAs  and  commodity  fund  managers  must  not  consider  intermediate momentum  as  a  viable  substitute  for  conventional  momentum  strategies. Instead,  the 11,10  microscopic  strategy,  which  offers  similar  profits  in  magnitude  but  unique dynamics of returns to conventional strategies, may be a feasible alternative.

Third, around 77% of the variation of returns in the JT conventional momentum strategy can be explained by its microscopic decomposition. However, since no dominance is found on any individual month, all past months are found to be important in determining the conventional commodity momentum profits.

Fourth, echo and microscopic  momentum  is  partially  related  to  the  U.S.  cross-sectional  equity momentum and the returns from broad commodity futures, but is not related to stocks, bonds,  foreign currency risks and macroeconomic  conditions. Consistent with  Asness et. al., (2013), this finding implies that there may  indeed be a  common component in momentum across asset classes."


Are you looking for more strategies to read about? Check http://quantpedia.com/Screener

Do you want to see performance of trading systems we described? Check http://quantpedia.com/Chart/Performance

Do you want to know more about us? Check http://quantpedia.com/Home/About

Analysis of US Dollar Carry Trades in the Era of 'Cheap Money' Wednesday, 20 April, 2016

A related paper has been added to:

#129 - Dollar Carry Trade

Authors: Shehadeh, Erdos, Li, Moore

Title: US Dollar Carry Trades in the Era of 'Cheap Money'

Link: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2765552

Abstract:

In this paper, we employ a unique dataset of actual US dollar (USD) forward positions against a number of currencies taken by so-called Commodity Trading Advisors (CTAs). We investigate to what extent these positions exhibit a pattern of USD carry trading or other patterns of currency trading over the recent period of the ultra-loose US monetary policy. Our analysis indeed shows that USD positions against emerging market currencies are characterised by a pattern of carry trading. That is, the USD, as the lower yielding currency, is associated with short positions. The payoff distributions of these positions, moreover, are found to have positive Sharpe ratios, negative skewness and high kurtosis. On the other hand, we find that USD positions against other advanced country currencies have a pattern completely opposite to carry trading which is in line with uncovered interest parity trading; that is, the lower (higher) yielding currency is associated with long (short) positions.

Notable quotations from the academic research paper:

"In the wake of the 2007-2008 financial crisis, many countries, especially developed countries including the USA, have adopted unconventional loose monetary policies with the purpose of stimulating their sluggish and unstable economies. This period is termed in the financial press as “the era of cheap money”. On the other hand, other countries, especially emerging markets, have maintained relatively high interest rates over the same period. Because of the potential impact of these effects on the trading decisions of the FX traders, it is worthwhile to consider currency trading in general and USD carry trading in particular over the sample period of the paper.

In light of this, the crux of the paper is to analyse our dataset of the USD forward positions to find out to what extent they show characteristics of USD carry trading or another trading strategy over the recent period of record-low US interest rates. In other words, we investigate whether these positions exhibit a response to the very low US interest rates by having a pattern of USD carry trading or other patterns of trading strategies can be identified across different currency markets. The distinctive feature of this study is that we have access to a dataset of daily-aggregated USD forward positions against a number of advanced and emerging currencies. It is collected from a Swedish investment specialist, Risk & Portfolio Management AB (RPM) which is a fund of hedge funds investing in Managed Futures strategies which are also known as Commodity Trading Advisors (CTAs). CTAs engage in various strategies like trend-following, short-term trading, and global macro that often employs carry trading as a sub-strategy. By exploiting and analysing our private dataset we find significant long-run equilibrium relationships which directly relate the USD forward positions to its forward premium. The relationships point to different trading strategies for emerging and advanced market currencies. For emerging currencies, we find that these relationships are consistent with carry trading. That is the lower yielding currency (the USD) is associated with short positions and vice versa. This carry trading pattern of forward shorting lower-yielding currency is induced by the expectations that the lower-yielding currency will not actually appreciate on average as much as the forward rate implies, or even it will depreciate. This in turn implies a profit on average at maturity. On the other hand, we find that the reverse holds between the USD and advanced currencies. In other words, we find a pattern of “fundamentals-based” trading consistent with the uncovered interest parity condition. That is, the lower (higher) yielding currency is associated with more long (short) positions. These anti-carry positions can reflect the unattractiveness of the advanced currencies-USD carry trading due to the increased uncertainty and narrow interest differentials for these markets over the period following the recent crisis.

Given that our data set is collected from FX traders which are mainly trend-followers, these results of the different trading strategies for emerging and advanced market currencies shed some light on the trading behaviour of this group of the FX market participants. On the one hand, the characteristics of carry trades for EM currencies which involve long high-interest currency aginst the low-interest currency reflect a trend-following strategy which is based on
the expectations that high-interest currency is going to appreciate -i.e. based on the appreciation trend of the high-interest rate currency. On the other hand, the characteristics of “fundamentals-based” trades for AM currencies which involve long low-interest currency against high-interest currency reflect a trend-following strategy which is based on the expectations that low-interest rate currency is going to appreciate –i.e. based on the appreciation trend of the low-interest rate currency. This is in line with the heterogeneous agents model developed by Spronk et al. (2013). The model demonstrates that depending on the dominant trend in the market, FX trend-followers can be in the same line of either carry traders or fundamentalists. In this sense, our results provide some insights into these features of the FX trend-following traders."


Are you looking for more strategies to read about? Check http://quantpedia.com/Screener

Do you want to see performance of trading systems we described? Check http://quantpedia.com/Chart/Performance

Do you want to know more about us? Check http://quantpedia.com/Home/About