Designing and Testing Trading Systems: How to Avoid Costly Mistakes
By Louis B. Mendelsohn
WITH THE ADVENT OF MICROCOMPUTERS A DECADE AGO, the window of opportunity opened for individual futures traders to design and test technical trading systems. Initially, only those traders who were also proficient programmers could do so, since the early trading software was neither very elegant nor user-friendly. At first, two basic approaches were available to traders: "tool box" and "black box."
Early Approaches Lack Testing Capability
The tool-box approach was limited to charting prices and calculating individual technical indicators. The main drawback was that traders still had to analyze subjectively all of the information to decide what action to take. Since this approach did not generate actual trading signals, it cannot be considered a trading system.
By comparison, the black box approach did generate signals. However, they were based on secret indicators that had fixed, preset values. There was no way to verify whether a system's logic was even based on sound technical analysis principles. Therefore, all traders using a particular black box system, regardless of their differences in risk propensity, trading style, and financial goals, received the identical signal at the same time and were expected to act on it on blind faith.
Neither approach had a history tester. Traders could not design and test their own trading models on real price data to find the best model to use for each market.
Introduction to Historical Modeling
ProfitTaker, which I developed, was the first full-blown futures trading system with disclosed trading rules. It enabled traders to design and test customized trading models using actual contracts with rollovers under simulated real-time trading conditions. With its innovative modeling concept, ProfitTaker ushered in a new generation of software.
But historical modeling did not become practical until the replacement of the Apple computer (with its limited memory and disk capacity) by the much more powerful IBM and low-cost compatibles as the computer of choice among traders. Before long, a plethora of trading systems followed suit, adopting ProfitTaker's modeling concept. Then, as satellite technology became more cost effective and daily prices more volatile, traders turned increasingly to day-trading. Now, even traders with limited trading capital can analyze realtime tick-by-tick prices on their computers. Quotation services, which previously provided only price quotes and news information, now offer built-in technical analysis and system modeling software.
Costly Mistakes Must Be Avoided
Today's traders easily develop and test customized trading models, which can be retested and adjusted as market conditions change. All that traders have to do is pick the price data to be tested and specify the indicator values to use. Depending on the number of models involved and the computer's speed, this procedure may need to be performed in a two-step process: coarse testing followed by fine testing. First the increment size between values is set relatively large for each indicator. Once a narrow range of profitable models is found, the increment size is reduced. These models are then tested within this more limited range, until the best model is isolated.
This process, by which traders design trading models, test their profitability and search for the best model to use, is widely known as "historical optimization." Too often, however, after models founded on historical data are applied to current prices in real-time trading, the expected profits are not realized. With so many traders now using computers to design and test trading systems, it is important to examine the intricacies of the modeling process itself and to identify commonly made mistakes that could be costly to your bottom line.
Full Disclosure
First, you should avoid using trading systems that intentionally keep some or all of their trading rules secret. These unwarranted restrictions prevent you from understanding the rationale behind the signals and undermine the discipline and confidence needed for implementing a sound trading strategy.
Testing Is the Means to the End
Another mistake occurs if you become so involved looking for profitable models on historical data that you completely lose sight of the forest for the sake of the trees. Testing is only a means to an end, not an end in itself. You should not try to find the most profitable trading model on historical prices. Otherwise, you risk selecting an isolated model that is surrounded on both sides by poorly performing ones. I call this type of model a "profit island."
The goal of historical modeling is to find models that have a high likelihood of producing profits and remaining stable in the ensuing period when you are actually trading in real time. To do this, you should look for a broad band of profitable models with the best performing one located near the middle. To either side of the best model are other profitable models, with their performance dropping off gradually the further they are from the best one. A model chosen from this profit cluster is more likely to remain stable even with subsequent changes in price characteristics.
