Position Sizing Tool Position sizing software for trading. Trade smarter. Maximize results.

The Breakout Bulletin The following article was originally published in the March 2006 issue of The Breakout Bulletin.   The Myth of Optimization In conversations with customers and with fellow traders, I've encountered a variety of opinions on the topic of trading system optimization. Some traders take it for granted as part of the system development process. Other traders won't go near a trading system that's been optimized. Some traders simply adopt the viewpoint expressed by whichever trading expert they've recently heard or read. If a well-known trading authority says "optimization is bad," it's tempting to accept that view without question. Unfortunately, even well meaning experts can be biased by a few strong experiences, either positive or negative. If my experiences are representative, it's harder to do optimization right than wrong, so biases probably tend towards the negative. Experience, however, is not the only way to understand something. We can also apply theory, as I've done in past articles when, for example, I discussed using a statistical test to identify system "over-fitting." A third possibility is simpler, if not rarer: applying commonsense logic. This is the approach I'd like to take here. What is Optimization? Let's start with the definition of optimization. Many traders probably think of optimization as a technical or mathematical method used to improve the back-tested performance of trading systems. In many cases, that's true. However, optimization is much broader than that. I first studied optimization in graduate school as an engineer, first in engineering classes then later in computer science and mathematics classes. In that context, optimization is about finding the minimum (or, alternatively, the maximum) value of a function. The function represents some feature or quality that you want to improve or a negative aspect that you want to eliminate or reduce. With regard to trading systems, the feature that we want to improve or maximize is the performance of the system. The complicating factor is that the only performance we can improve during the optimization process is historical performance. Past performance may not be indicative of future results, but, as the saying goes, that's all we have. The next part of the definition concerns what we modify or adjust to improve performance. Traditionally, trading system optimization means finding values for the adjustable parameters of the trading system. However, there is no reason to limit the definition of system optimization to finding the parameter values that maximize performance. If our goal is to maximize performance, we might want to change not only the parameter values but the system rules or even the system itself. With that in mind, we can say that trading system optimization is the process of developing a trading system to maximize its historical performance. Consider for a moment what this definition says and doesn't say about optimization. In particular, it says nothing about the type of optimization method. For example, it's not limited to standard numerical optimization methods, such as the built-in system optimizer of TradeStation. This definition includes manual optimization methods, such as choosing indicators by visually evaluating their performance on a price chart. This definition also includes selecting the best performing system from a set of prospective systems. Optimization is Hard to Avoid The point of defining optimization the way I have is to demonstrate that optimization is more inclusive than many traders realize and much harder to avoid. If you think you've never optimized a trading system, ask yourself if you've ever:		Chosen the length of a moving average or other indicator based on what you've used successfully before?
	Looked at an indicator on the screen and adjusted the parameter values until it worked the way you wanted it to?
	Developed multiple variations of your original system idea and picked the one that worked best?
	Refined and modified your system code until it gave acceptable performance?
	Purchased a trading system by reviewing the past performance of several systems and selecting the one with the best performance?

These are all examples of system optimization according to the definition above, and each of these is potentially just as harmful or helpful as running a trading system through TradeStation's built-in optimizer.

If you're starting to think that avoiding optimization when developing a trading system or trading plan is almost impossible, then you understand my point. Optimization is really just a fancy name for making decisions about a trading system's development or selection based on past performance. But who wouldn't want to select a trading system -- or a rule to add to a trading system or a parameter value for a rule -- that resulted in good past performance? Put another way, would you rather choose a system or rule or parameter value that resulted in good past performance or poor past performance? Commonsense suggests that we're increasing our odds of success if we choose the former. If we do, that means we're optimizing.

What's the problem with optimization? As I've discussed in past newsletters, the risk of optimizing a trading system is something I refer to as "over fitting." That means that we spend so much effort tailoring our system to a particular market or set of market conditions that the system has little chance of doing well when it encounters different conditions in the future. Since the future is never exactly the same as the past, a system that is fit too closely to the past may not do well in the future.

So optimization is common in system development, and there's a risk the optimized system may not hold up in real trading. If we really want to avoid optimization, is there a way to do so? Here are three possible methods I came up with to avoid optimizing the parameter values for an existing trading system:

1. Develop a trading system with no parameters.
2. Choose the parameter values randomly.
3. Choose the parameter values based on nonrandom, non-adjustable factors. For example, set the length of a moving average to the dominant cycle length or choose the breakout amount as one standard deviation of the average volatility.

On the surface, these seem like good methods to avoid optimization. However, even here, there are subtle forms of optimization. In #1, for example, how did we come up with the rules for the trading system? If we chose the rules based on what we knew from past experience would work well, then the system is optimized to some extent. Also, even if there are no adjustable parameters, there may be "hard-coded" parameters in the system, such as using 14 as the length for ADX. You might argue that you didn't select 14 as the parameter value because that's the default value. However, someone at some time had to choose that number, and whoever did may have chosen it because it worked well in testing. Just because someone else did the optimization doesn't mean you can ignore it.

Method #2 seems unassailable. However, even here, you have to choose a reasonable range from which to randomly select the parameter value. It wouldn't be sensible to randomly select the length of a moving average from the range 1 to 1,000,000. Most people would probably choose a more meaningful range, such as 5 to 100. As soon as you choose a range, you're doing a limited kind of optimization. More likely, though, the problem with #2 would be that most people wouldn't be able to resist the urge to re-select new parameters if the first set they found was unprofitable. In that case, you'd be employing a random-search optimization method.

Method #3 may be the most practical. Unfortunately, there probably are not many non-adjustable factors that can be used to set system parameter values. Even the ones I mentioned generally require some input. Most of the cycle calculations I've seen include parameters such as the look-back length or the range of cycle lengths to consider. If you want to base a breakout system on the standard deviation of volatility, how many standard deviations should you use? You could pick it randomly, but then you're back to #2.

This seems to suggest that all system development involves some form of optimization, if only to a limited extent. This tells me that it's not optimization that's the problem but the type of optimization and its extent. Considering all lookback lengths from 1 to 1000 may be overkill, but using the dominant cycle length -- even if the calculation requires some trial and error -- may be perfectly acceptable. Some optimizations are better than others, which brings us back to the problem of over fitting.

What Can Be Done?

A successful optimization matches the trading system to the internal dynamics of the market. By internal dynamics, I mean the technical features of the market that characterize its trading behavior. It's possible, of course, that the internal dynamics of the market will change over time, which can cause a system to fail, even despite proper system design and optimization. The likelihood of that can be reduced somewhat by optimizing over a long period of time. Over a long period of time, the market dynamics may change. If a trading system tests well over changing conditions, it must be flexible enough to adapt to the changing conditions or it's capturing market dynamics that are more fundamental than the ones that are changing.

By contrast, an unsuccessful optimization is capturing market noise rather than fundamental market behavior. It's usually possible to increase the likelihood that the optimization is capturing market fundamentals rather than noise by increasing the number of trades in the optimization sample. By definition, noise is random behavior, so by optimizing over a larger number of trades, the effect of the randomness is averaged out, and the likelihood that the system's profit could be coming from noise rather than internal market dynamics is reduced. Of course, this only works if the trading system logic is capable of capturing the market dynamics. If it isn't, increasing the number of trades should make this fact clear by reducing the profitability of the system. In fact, if the optimized average trade size (i.e., the size of the average trade on a one-contract or fixed share basis after optimization) of a trading system tends to decrease as more trades are added to the optimization, this might suggest that the profit is coming from market noise rather than fundamental market dynamics. At the extreme, a trading system that cannot be made profitable even with optimization is clearly not capturing market dynamics.

In general, then, the best way to avoid over fitting is to increase the number of trades in the optimization sample and/or increase the time frame over which the optimization is performed. For the reasons noted above, however, this doesn't guarantee that the system won't be over fit to the market. Consequently, it's necessary to have some way to test for over fitting. The standard methods for this are out-of-sample testing and real-time trading.

In out-of-sample testing, a portion of the market data is set aside as the "test segment." The optimization is performed on the remaining data, and the optimized system is then tested on the test segment. Since the optimized system didn't previously run on the test segment, the results on that data provide an objective assessment of the system's performance. Theoretically, this approach is as valid as testing the system in real-time trading.

One drawback to out-of-sample testing is that it's easy to cheat, even if you don't intend to. For example, let's say the results in the test segment are only mediocre, but you think you know why, so you make a change in your system and re-optimize. Now the "out-of-sample" test results are better. Unfortunately, by changing your system based on how it performed on the test data, you've just optimized over the test data, which no longer provides an objective test. This is why some people prefer real-time trading results when evaluating trading systems. Even this is not necessarily fool-proof, however. In principle, one could simultaneously trade multiple variations of the same system in real-time then select the version that produced the best real-time results. In effect, this would be optimizing the system over the real-time data, and there would be no guarantee that the selected version of the system wouldn't be over-fit to the market.

So what is the myth of optimization? I think the myth of optimization is that optimization is necessarily bad for trading systems and should be avoided at all costs. The reality is that virtually all trading system development involves optimization in some form and to some extent. Saying optimization is bad is like saying mathematics is bad. It's only bad if you do it poorly. Good optimization starts with trading system logic that's capable of capturing the market's fundamental behavior. Good optimization avoids over fitting by including a large number of trades over a long period of time. Good optimization ends with out-of-sample testing and/or real-time trading. Of course, none of this guarantees success. Markets change over time, and no testing method I know of can predict the future. After all, we're still talking about trading.