Realization: Walk-Forward Out-of-Sample Testing
The Realization feature in Investor/RT refers to the process of walk-forward out-of-sample testing (or walk-forward optimization). In essence, Investor/RT optimizes (or backtests) over a given period, then tests using the optimized variables (or best performing symbols, or both) over a subsequent period immediately following the optimization period. In this way, we are seeing historically what we could have achieved in reality by trading our optimized results or best performing symbols. This process can give us a good gauge of whether the system should be trading going forward, or shelved/re-evaluated.
The Realization window can be seen below:
There are three primary methods in which Realization can be used:
The Realization window asks for an "Optimization" on which to operate. When performing SOTF realization, there is no need to optimize, however, an optimization must be created involving the trading system which the user wants tested, and that optimization should be specified in the realization window. In this scenario, the "Skip Optimization" checkbox should be checked as well.
When a Realization is in process, a "Status" area at the bottom of the Realization window provides the user with useful information regarding the progress and state of the Realization. The information provided includes which iteration is currently being processed, which process is currently running (pre-testing, post-testing, optimization, or real-testing), the status of the process currently running, the approximate amount of time left in realization, the Net Profit and APS overall, the Net Profit of Last iteration, and the date range of the data currently being tested.
When the Realization is competed, a report is provided which looks very similar to the :Day by Day" backtest report. It provides a variety of cumulative statistics for the realized testing, including Net Profit, Average Profit / Share, Total Trades, Best Day, Worst Day, etc. It also breaks down each day. If SOTF realization was performed, the "best" symbols used for each iteration, along with the statistical value (Net or APS) of each symbol, is listed for each iteration.
Taking a look at a specific example, let's assume we have a quotepage with 100 symbols (Nasdaq 100) and we have the 500 days of daily data ending 3/23/05 which can be used for testing. To keep things simple, let's look at a very "vanilla" system which buys at the open of each day and sells at the close of each day when the previous day was up (CL.1 > OP.1) and shorts at the open of each day and covers at the close when the previous day was down or flat (CL.1 <= OP.1). This particular system will make one trade (100 shares) every day for every symbol tested. The system exits each position at the close of the day.
If we simply test this system as is over the 500 day period, the Nasdaq 100 gives us a relatively marginal loss of $0.0019 / share (Net $-17,913.42). Based on these results, we certainly wouldn't be excited about trading this system. If anything, we would probably choose to trade the reverse (buy when previous day was down), but even that would result in a very marginal profit that would get sucked up by commissions and slippage. Before discarding this system however, we might want to put it through some Realization runs to see if there potential opportunities.
Using Method 1 as mentioned above, we might choose to trade only the top 50 symbols based on their performance over the most recent 40 days (~2 months), and change this list of 50 each 20 days (~1 month). This would require an optimization period of 40 days, and a realization period of 20 days. The realization window would be setup similar to what is seen below:
Here are a sample of results on the Vanilla system.
Looking at the results, it's clear that at least for this system in this environment, the symbols that did well in the recent past, seemed to continue to perform into the future. Using an optimization period of 20 days with a realization period of 5 days, and testing only the best 2 symbols, produced a strong 5.9 cents/share for a net profit of $11,873. This is compared with a $17,913 loss when testing the whole list of 100 stocks. This net profit was also produced trading only 2% of the shares.
Now, let's really give realization a test by turning the Vanilla system on it's head. Instead, we'll buys at the open of each day and sells at the close of each day when the previous day was down or flat (CL <= OP) and short at the open of each day and cover at the close when the previous day was up (CL > OP). Here are some realization results using the VanillaRev system.
Now, let's throw a variable into the system via a target so that we can involve optimization (on this variable target) in the realization. We'll start by testing with a target of 0.40. Going back to our original Vanilla system, we get a net profit of $3,398 and an APS of $0.0004. This is a pretty nice improvement over our "no target" original system already (Net: $-17,913.42).
Now, we'll setup a "per symbol" optimization on the target, going from 0.20 to 1.00 and stepping 0.20.
...More coming soon