The following article was originally published in the March 2006 issue of
The Breakout Bulletin.
Myth of Optimization
In conversations with
customers and with fellow traders, I've encountered a variety of opinions on the
topic of trading system optimization. Some traders take it for granted as part
of the system development process. Other traders won't go near a trading system
that's been optimized. Some traders simply adopt the viewpoint expressed by
whichever trading expert they've recently heard or read. If a well-known trading
authority says "optimization is bad," it's tempting to accept that view without
question. Unfortunately, even well meaning experts can be biased by a few strong
experiences, either positive or negative. If my experiences are representative,
it's harder to do optimization right than wrong, so biases probably tend towards
Experience, however, is not the only way to understand something. We can also
apply theory, as I've done in past articles when, for example, I discussed using
a statistical test to identify system "over-fitting." A third possibility is
simpler, if not rarer: applying commonsense logic. This is the approach I'd like
to take here.
What is Optimization?
Let's start with the definition of optimization. Many
traders probably think of optimization as a technical or mathematical method
used to improve the back-tested performance of trading systems. In many cases,
that's true. However, optimization is much broader than that. I first studied
optimization in graduate school as an engineer, first in engineering classes
then later in computer science and mathematics classes. In that context,
optimization is about finding the minimum (or, alternatively, the maximum) value
of a function. The function represents some feature or quality that you want to
improve or a negative aspect that you want to eliminate or reduce.
With regard to trading systems, the
feature that we want to improve or maximize is the performance of the system.
The complicating factor is that the only performance we can improve during the
optimization process is historical performance. Past performance may not be
indicative of future results, but, as the saying goes, that's all we have.
The next part of the definition
concerns what we modify or adjust to improve performance. Traditionally, trading
system optimization means finding values for the adjustable parameters of the
trading system. However, there is no reason to limit the definition of system
optimization to finding the parameter values that maximize performance. If our
goal is to maximize performance, we might want to change not only the parameter
values but the system rules or even the system itself.
With that in mind, we can say that
trading system optimization is the process of developing a
trading system to maximize its historical performance.
Consider for a moment what this
definition says and doesn't say about optimization. In particular, it says
nothing about the type of optimization method. For example, it's not limited to
standard numerical optimization methods, such as the built-in system optimizer
of TradeStation. This definition includes manual optimization methods, such as
choosing indicators by visually evaluating their performance on a price chart.
This definition also includes selecting the best performing system from a set of
Optimization is Hard to Avoid
The point of defining optimization the way I have is to
demonstrate that optimization is more inclusive than many traders realize and
much harder to avoid. If you think you've never optimized a trading system, ask
yourself if you've ever:
These are all examples of system
optimization according to the definition above, and each of these is
potentially just as harmful or helpful as running a trading system through
TradeStation's built-in optimizer.
If you're starting to think that
avoiding optimization when developing a trading system or trading plan is almost
impossible, then you understand my point. Optimization is really just a fancy
name for making decisions about a trading system's development or selection
based on past performance. But who wouldn't want to select a trading system --
or a rule to add to a trading system or a parameter value for a rule -- that
resulted in good past performance? Put another way, would you rather choose a
system or rule or parameter value that resulted in good past performance or poor
past performance? Commonsense suggests that we're increasing our odds of success
if we choose the former. If we do, that means we're optimizing.
What's the problem with optimization?
As I've discussed in past newsletters, the risk of optimizing a trading system
is something I refer to as "over fitting." That means that we spend so much
effort tailoring our system to a particular market or set of market conditions
that the system has little chance of doing well when it encounters different
conditions in the future. Since the future is never exactly the same as the
past, a system that is fit too closely to the past may not do well in the
So optimization is common in system
development, and there's a risk the optimized system may not hold up in real
trading. If we really want to avoid optimization, is there a way to do so? Here
are three possible methods I came up with to avoid optimizing the parameter
values for an existing trading system:
Develop a trading system with no parameters.
Choose the parameter values randomly.
Choose the parameter values based on nonrandom,
non-adjustable factors. For example, set the length of a moving average
to the dominant cycle length or choose the breakout amount as one
standard deviation of the average volatility.
On the surface, these seem like good
methods to avoid optimization. However, even here, there are subtle forms of
optimization. In #1, for example, how did we come up with the rules for the
trading system? If we chose the rules based on what we knew from past experience
would work well, then the system is optimized to some extent. Also, even if
there are no adjustable parameters, there may be "hard-coded" parameters in the
system, such as using 14 as the length for ADX. You might argue that you didn't
select 14 as the parameter value because that's the default value. However,
someone at some time had to choose that number, and whoever did may have chosen
it because it worked well in testing. Just because someone else did the
optimization doesn't mean you can ignore it.
Method #2 seems unassailable. However,
even here, you have to choose a reasonable range from which to randomly select
the parameter value. It wouldn't be sensible to randomly select the length of a
moving average from the range 1 to 1,000,000. Most people would probably choose
a more meaningful range, such as 5 to 100. As soon as you choose a range, you're
doing a limited kind of optimization. More likely, though, the problem with #2
would be that most people wouldn't be able to resist the urge to re-select new
parameters if the first set they found was unprofitable. In that case, you'd
be employing a random-search optimization method.
Method #3 may be the most
practical. Unfortunately, there probably are not many non-adjustable factors
that can be used to set system parameter values. Even the ones I mentioned
generally require some input. Most of the cycle calculations I've seen include
parameters such as the look-back length or the range of cycle lengths to
consider. If you want to base a breakout system on the standard deviation of
volatility, how many standard deviations should you use? You could pick it
randomly, but then you're back to #2.
This seems to suggest that all system
development involves some form of optimization, if only to a limited
extent. This tells me that it's not optimization that's the problem but the type
of optimization and its extent. Considering all lookback lengths from 1 to 1000
may be overkill, but using the dominant cycle length -- even if the calculation
requires some trial and error -- may be perfectly acceptable. Some optimizations
are better than others, which brings us back to the problem of over fitting.
What Can Be Done?
A successful optimization matches the trading system to the
internal dynamics of the market. By internal dynamics, I mean the technical
features of the market that characterize its trading behavior. It's possible, of
course, that the internal dynamics of the market will change over time, which
can cause a system to fail, even despite proper system design and optimization.
The likelihood of that can be reduced somewhat by optimizing over a long period
of time. Over a long period of time, the market dynamics may change. If a
trading system tests well over changing conditions, it must be flexible enough
to adapt to the changing conditions or it's capturing market dynamics that are
more fundamental than the ones that are changing.
By contrast, an unsuccessful
optimization is capturing market noise rather than fundamental market behavior.
It's usually possible to increase the likelihood that the optimization is
capturing market fundamentals rather than noise by increasing the number of
trades in the optimization sample. By definition, noise is random behavior, so
by optimizing over a larger number of trades, the effect of the randomness is
averaged out, and the likelihood that the system's profit could be coming from
noise rather than internal market dynamics is reduced. Of course, this only
works if the trading system logic is capable of capturing the market dynamics.
If it isn't, increasing the number of trades should make this fact clear by
reducing the profitability of the system. In fact, if the optimized average
trade size (i.e., the size of the average trade on a one-contract or fixed share
basis after optimization) of a trading system tends to decrease as more trades
are added to the optimization, this might suggest that the profit is coming from
market noise rather than fundamental market dynamics. At the extreme, a trading
system that cannot be made profitable even with optimization is clearly not
capturing market dynamics.
In general, then, the best way to
avoid over fitting is to increase the number of trades in the optimization
sample and/or increase the time frame over which the optimization is performed.
For the reasons noted above, however, this doesn't guarantee that the system
won't be over fit to the market. Consequently, it's necessary to have some way
to test for over fitting. The standard methods for this are out-of-sample
testing and real-time trading.
In out-of-sample testing, a portion of
the market data is set aside as the "test segment." The optimization is
performed on the remaining data, and the optimized system is then tested on the
test segment. Since the optimized system didn't previously run on the test
segment, the results on that data provide an objective assessment of the
system's performance. Theoretically, this approach is as valid as testing the
system in real-time trading.
One drawback to out-of-sample testing
is that it's easy to cheat, even if you don't intend to. For example, let's say
the results in the test segment are only mediocre, but you think you know why,
so you make a change in your system and re-optimize. Now the "out-of-sample"
test results are better. Unfortunately, by changing your system based on how it
performed on the test data, you've just optimized over the test data, which no
longer provides an objective test. This is why some people prefer real-time
trading results when evaluating trading systems. Even this is not necessarily
fool-proof, however. In principle, one could simultaneously trade multiple
variations of the same system in real-time then select the version that produced
the best real-time results. In effect, this would be optimizing the system over
the real-time data, and there would be no guarantee that the selected version of
the system wouldn't be over-fit to the market.
So what is the myth of optimization? I
think the myth of optimization is that optimization is necessarily bad for
trading systems and should be avoided at all costs. The reality is that
virtually all trading system development involves optimization in some form and
to some extent. Saying optimization is bad is like saying mathematics is bad.
It's only bad if you do it poorly. Good optimization starts with trading system
logic that's capable of capturing the market's fundamental behavior. Good
optimization avoids over fitting by including a large number of trades over a
long period of time. Good optimization ends with out-of-sample testing and/or
real-time trading. Of course, none of this guarantees success. Markets change
over time, and no testing method I know of can predict the future. After all,
we're still talking about trading.
That's all for
now. Good luck
with your trading.