By: Louis B. Mendelsohn
President, Market Technologies

Over the past several years there has been a proliferation of software programs which allow futures traders to develop trading models and perform simulated historical testing to find the best model to use for each Market. This testing process is widely known as "historical optimization." Too often, however, when these models are then used in real-time, traders are not able to achieve the expected profits.

This problem is primarily due to design limitations in the software programs which undermine the validity of the testing process itself. One of these design limitations involves the use of non-disclosed ("black box") or partially disclosed ("gray box") trading rules that comprise the models. These restrictions lessen the trader's confidence and discipline to act on the trading signals in real-time, and prevent him from understanding how to alter the models when necessary to improve their performance. Only programs which fully disclose their trading rules overcome these restrictions.

Another serious design limitation concerns the inclusion of too many entry and exit rules in the model. This approach is based on the erroneous premise that trading performance will be improved if the models have a large number of trading rules. The following hints on historical testing will help you avoid these costly pitfalls when designing and testing models for use in real time trading.

First, you should realize that historical modeling is like looking for a needle in a haystack. The bigger the haystack, the less likely it is that you will find the needle. Similarly, the larger the number of trading rules, the more difficult, time consuming, and frustrating it will be to find and stick with a model--without feeling compelled to "hopscotch" from one set of rules to another whenever you incur a string of back-to-back losing trades. This pitfall, often referred to as "paralysis of analysis," is especially dangerous for newcomers to trading.

When designing and testing models, keep it simple. Use a limited number of rules, preferably five or less, in the "modeling" process to discern a general tone or pattern in a market that objectively exists. By contrast, if you use too many rules, particularly highly correlated ones, you will merely superimpose non-existent patterns onto the historical data. This results in seemingly profitable models which conform so closely to the historical data that they have little if any predictiveness for the future. This serious pitfall is commonly called "curve-fitting."

Remember, the testing process should not be directed solely toward finding trading models which produce the most profit on historical prices. The real goal is to find models that have a high likelihood of producing profits in real time. Testing is a means to an end, not an end in itself.

One way to overcome these pitfalls is to perform "blind simulation" testing, in which a limited number of trading rules are applied to a specified time period of historical prices, which I call a "testing window." The best models are then applied to a "trading window" comprised of entirely different prices that were not included in the original test. In this way, real time trading conditions are simulated prior to actually risking capital.

The use of different size testing and trading windows for each market should be considered, however, since the window sizes that are selected affect the results. This could mean that the best model to use may be overlooked. Therefore, finding the "optimal" window sizes for each market could substantially improve real time trading performance.

"Blind simulation" testing is an ongoing process of moving a testing window of a specific size forward in time, in order to find the right model to use for each market. Then you would use that model to trade in real-time for a certain time period, before retesting and updating the model. As a practical matter, if you have any doubt about the correct window sizes to use, you should perform testing more frequently, to confirm that the model that you are presently using is still the best one.

Finally, you should understand that the brute force of using computers to "crunch numbers" is not a substitute for your intelligent involvement in the decision-making process. The computer is merely a tool. The old cliché of "garbage-in garbage-out" can have serious financial consequences on your bottom line -- namely whether or not you make or lose money as a trader.

Louis Mendelsohn, president Market Technologies, Wesley Chapel, Florida, designs and tests neural trading systems for the financial industry.

Want to see how you can use VantagePoint
to give you an edge in your trading? Click Here >>>