Table 24-1
Effect of Number of Indicators on Number of Models to Be Tested

Number of Indicators

You can see that a system with three indicators, each of which can have a range of 25 values, results in a universe of over 15,000 models. When a fourth indicator is added, the number of models increases to nearly 400,000. Adding a fifth indicator raises the number of models to nearly 10 million. Increasing the range of values over which the indicators vary also causes the number of models to increase, but less dramatically. For instance, a system with five indicators, which can each have 20 values, creates a total of over three million models. Keeping the number of indicators constant at five, but increasing their range of values from 20 to 25, increases the number of models to nearly 10 million.

The amount of time it takes to test a system varies in direct proportion to the number of models tested. Therefore, it would be impractical on a microcomputer to test a system on even one market using more than four or five indicators. The coarse testing and fine testing method would be required, and even then the increment sizes might need to be so large that only a small percentage of possible models could actually be tested. Under these circumstances, there is a tendency, after you pick a model to use in real time, to feel compelled to change to other models or to switch indicators each time you incur a string of back-to-back losing trades. This syndrome, known as paralysis of analysis, can become immobilizing.

Still, many traders believe that the more indicators a model comprises, the better the system's profitability. This may be true up to a point, but then diminishing returns set in. If you use a limited number of indicators in the modeling process, you will discern a general pattern to a market that objectively exists. By contrast, when you employ too many indicators, you merely superimpose a pattern, which actually does not exist, onto the historical data. This is particularly true of indicators that are highly correlated with one another. Then you risk using seemingly profitable models in real time trading. They look good on paper, but conform so closely to the historical data that they have little, if any, predictiveness. This serious pitfall is commonly known as "curve-fitting." Such models tend to be unstable, decay rapidly, and will therefore subject you to unnecessary losses.

Some software programs now give traders considerable flexibility to write their own trading rules. While this much flexibility may be useful to certain traders who have the time and resources to engage in extensive trading system design, most traders find this much flexibility to be a hindrance to their success as traders. Such software encourages traders to build complex systems with numerous indicators, in the search for the "Holy Grail." By doing so, traders risk picking "curve-fitted" models, or becoming immobilized by the paralysis-of-analysis syndrome. Most off-the-shelf systems already offer more than enough user flexibility. Unless you have considerable spare time to spend, or are a full-time professional technical analyst, it really isn't cost-effective to recreate the wheel by designing your own system from scratch. You're better off leaving that to professional trading system developers.

Know When to Reoptimize
Each market should be reoptimized routinely, on a preemptive basis. Many software programs can screen models to find those that meet your financial performance objectives, saving this information on a disk file. Then you can view the information on your monitor. With fast computers, such as those based on the 80286 or 80386 microprocessor, testing can be done on a rotational basis, one market each night, and several on a weekend. This way, even with a large diversified portfolio, you can easily reoptimize every market that you trade at least once a month.

To make this process virtually a turn-key operation for my clients, I provide a monthly report to them, called ProfitTuner, which tests eighteen actively traded markets on a mainframe computer. In some the models may change from one month to the next, while in other markets the models are quite stable. Since the object is to find models that are not curve-fitted, ProfitTuner lists the 50 best-performing models for each market, rather than the one model that produces the maximum profit.

Whenever a fundamental event takes place, affecting specific markets in a way that could not have been anticipated by the modeling process, you should retest. For instance, a weekend meeting of the G-7 trading partners, with an unexpected announcement of a revised trading range for the U.S. dollar, should immediately raise a red flag for you to perform retesting on those markets.

Check Your Models for Curve-Fitting
Perform blind-simulation testing to see whether your models are curve-fitted. To do this, test your models on a defined time period of historical prices, which I call a testing window. The best model selected is then applied to a trading window composed of entirely different historical prices that were not included in the original test. In this way, real-time trading conditions are simulated, with the performance of the model assessed for its stability, predictiveness, and profitability prior to actually risking capital.

Blind-simulation testing should actually be performed as part of your routine testing. Different size testing and trading windows for each market could be explored, since the window sizes affect the test results and choice of models. Finding the optimal window sizes for each market could substantially improve real-time trading performance.

Make Realistic Assumptions
You should also set realistic assumptions about overhead costs, particularly the amount of slippage. If your chosen model isn't derived under realistic conditions, its performance in real time may be both disappointing and costly. Table 24-2 illustrates the effects that various overhead costs have on a model's performance.

As you can see in this example, Model A has $10,000 total profit, 25 trades, and $2,000 maximum drawdown. Model B has $12,000 profit, 45 trades, and also $2,000 drawdown. At first glance, when overhead is excluded, Model B appears to be superior to Model A. The risk/ reward ratio of profit to drawdown is higher for Model B.

Then, when a $100 overhead for slippage and commissions is included, both models produce identical net profit and risk/reward ratio. The inclusion of $100 overhead costs is more damaging to Model B because of its higher frequency of trading. This pattern is further exacerbated as overhead costs are increased. At $250 per round turn, Model B's ratio of profit to drawdown is less than 1.00. At $300, it has a net loss of $1,500.

In effect, the impact of overhead is magnified on models that generate a substantial number of trades. Models that look good when overhead costs are not taken into consideration may prove to be big losers when realistic costs are factored in. Overhead assumptions regarding slippage should reflect differences in each market's price volatility.

You also need to set realistic values for the limit move in each market. Otherwise, your modeling results will not be based on realistic execution prices. It is incorrect to record the execution of a trade as occurring at the time that the signal is generated when that market is lock-limit. Limit conditions must be accounted for and executions deferred until feasible. The big move in the silver market in 1979-1980 is a classic case where hypothetical track records of trading systems overstate profits due to unrealistic lock-limit assumptions.

A major improvement in this area could be made if data suppliers were to report each market's daily limit value and the 90-day T-bill rate on a daily basis along with prices, volume, and open interest. Then modeling software could be programmed to check precisely for normal and expanded daily limits during testing to avoid unrealistic price executions, and compare a system's daily rate of return to the risk-free return.

MODEL A

MODEL B

Test the Right Contracts
The type of contract tested also affects the results and model selected. You can choose between actual contract months with rollovers, single contracts, or artificially derived continuous contracts. My suggestion is to use actual contracts with rollovers because this minimizes unnecessary distortions. Results are less valid when you test single contracts over periods of low volume and open interest. Distortions also occur when you test continuous contracts that don't realistically handle the price spread between two contract months, at a time when the model has a position on in the expiring contract.

Test the Right Data
It is better to err on the side of more rather than less data. Statistical validity is increased when the testing window size is larger. It would be unwise to select a model based on just a few trades that occurred in the past couple of months. Such a model would not work very well under different market conditions that you might be faced with in the immediate future. Also, you should know about the model's distribution of profits. If you discover a highly profitable-looking model, but most of its profits were made over a year ago on one or two good trades and it has lost money since, that's not a worthy model. One quick way to look at the distribution of profits over the test period is to calculate for each model its ratio of profits in the last half of the test period to total profits over the entire test period. Any model with a ratio of less than 50 percent should be rejected, regardless of how well it does on other performance measures.

Of course, you should also verify the accuracy of your data. If it is not clean, your test results will be distorted. Automatic daily updating using a modem will eliminate typo errors that would otherwise be introduced when prices are updated manually from daily newspapers. The cliché garbage-in, garbage-out must be respected when doing historical testing.

Know Your Own Risk Propensity
Experiment with a trading system before actually trading it in real-time. You should be comfortable with it, both in terms of its operation and the indicators that it employs to generate buy/sell signals. Table 24-3 lists a number of performance measures that you can look at when comparing models to one another.

Check the ratio of cumulative net profit to maximum drawdown. Examine the drawdown. Drawdown should not be calculated as the dollar value of back-to-back losing trades. It's the maximum equity drop from a high point in equity to the subsequent low point in equity until the high point in equity is breached to the upside.

Table 24-3 Performance Measures to Use When Evaluating a Trading Model

• Total number of closed out trades

• Percentage of winning trades

• Percentage of long winning trades

• Percentage of short winning trades

• Gross cumulative profit or loss

• Net cumulative profit (Gross profit less overhead)

• Maximum drawdown

• Ratio of net cumulative profit to drawdown

• Maximum winning trade

• Maximum losing trade

• Average winning trade

• Average losing trade

• Average profit or loss per trade

• Number of consecutive losing trades

• Unrealized profit or loss in open position

• Distribution of profits over time

This ratio gives you a good indication of your risk/reward payoff. Essentially, it tells you the number of dollars you can expect to win for every dollar you put at risk. You should know your own risk propensity. Some traders are willing to risk a dollar to make a dollar. Others need more payoff for each dollar put at risk. Ask yourself how many dollars you would need to make on a given trade to motivate you to risk a dollar by taking that trade. That's the minimum ratio of profit to drawdown that you should accept.

Make sure that the system's characteristics and performance are compatible with your temperament, style of trading, and the depth of your pockets. If you can't handle emotionally or financially the frequency of back-to-back losing trades or the short term drawdowns that a particular system incurs, what good is it if it's a big money maker in the long run?

Other Modeling Ideas to Explore
Instead of developing models based on recent price data, you might want to test data from previous time periods that closely resemble today's market conditions. For instance, if you want to find a model that works well during a current drought affecting the soybean market, you might go back and test prior data in which beans were bullish because of earlier drought conditions. The grain markets, in particular, lend themselves to this type of seasonal testing.

Another interesting approach is to incorporate economic or fundamental indicators into your trading strategy. For instance, you might use one or more fundamental indicators to identify a bias for each market as either bullish, bearish, or sideways. Then, for instance, you would only act on your technical system's buy signals if, at the same time, that market's fundamental "bias" is bullish. In effect, you are applying a fundamental filter to confirm the technical signal generated by your trading system. This approach can eliminate false signals and alert you to impending turning points in the markets. Recently, I developed a software program called Trader that lets the user do just this by correlating the effects of 50 economic indicators on the price direction in each market.

Conclusion
With system design and testing still more of an art than a science, you are not guaranteed a free lunch from the futures markets. Common sense and businesslike decision making are needed for sound money management and proper portfolio diversification. Costly mistakes in designing and testing trading systems must be avoided. The computer is only a tool. It can help improve your trading if you are already knowledgeable about market dynamics, technical analysis methods, and modeling pitfalls to avoid. But it can't turn a trader who has not done his homework into an instant winner.

Want to see how you can use VantagePoint
to give you an edge in your trading? Click Here >>>