Auto Trade System Back Testing

Introduction

When you are using the Spreadsheet System for Trading study or you have created an ACSIL (Advanced Custom Study and Interface Language) trading system study that uses the ACSIL Trading functions, then you can perform Back Testing by using Trade Simulation Mode and replaying the chart, or by performing a Bar Based Back Test.

In order to evaluate how your trading system has performed historically and to see the historical trades for past data, it is necessary to run a Back Test.

A Bar Based Back Test uses the Open, High, Low, Close data of the loaded chart bars themselves to evaluate the automated trading system. It is this data which is incrementally added to the chart during the Back Test. This method of Back Testing is faster compared to a Replay Back Test, however it is less precise than a Replay Back Test.

A Replay Back Test uses the underlying data the Intraday data file to evaluate the automated trading system. This underlying data has a timeframe per record anywhere from 1 Tick to 1 Minute (depends upon various factors, including Data/Trading service used, historical data available and Sierra Chart settings). It is this more detailed underlying data in the Intraday data file which is incrementally added to the chart during the Back Test. This is a slower method of back testing compared to a Bar Based Back Test, however it is more precise.

With either method of Back Testing, a complete Trade Statistics report can be viewed based upon all of the order fills generated during the Back Test. Refer to Viewing Back Test Results.

Bar Based Back Testing

Supported for Historical and Intraday charts.

Back testing on loaded bars is a back testing method using the Open, High, Low, and Close values of the loaded bars. The underlying data in the data file is not used directly, only the loaded bars themselves.

This usually does not perform as high accuracy of a Back Test compared to a Replay Back Test. However, it is faster and the recommended method if fast back testing is important.

1. Go to the chart you want to do the back test for and make sure there is a check mark by both Trade >> Auto Trading Enabled - Global and Trade >> Auto Trading Enabled - Chart if there is not one already. Otherwise, the automated trading system will not generate any trades.
2. Disconnect from the data and trade server if not already, by selecting File >> Disconnect. It is not possible to perform a Bar Based back test while connected to the data or trade server.
3. The speed of the back test is affected by the Chart Update Interval. For the fastest back test, it is recommended to reduce the Chart Update Interval directly on the chart itself. If you make this very low like under 100 ms and do not normally require that, it is recommended to change it back to a higher value when you no longer need to do back testing of the chart.
4. To perform a high-speed Back Test based on loaded bars, select Trade >> Auto Trade System Bar Based Back Test.
5. You will be prompted for the Bar Processing Increment . The default is 250. This specifies the number of bars that are processed at once before user interface input is processed. Which means if you specify 250, then 250 bars be processed at once and then another 250. Between each of these blocks of processing, you are able to interrupt the Back Test by pressing the Escape key on the keyboard. The larger the number that is specified, the longer the time that the user interface will remain busy.
6. To view detailed results of the Back Test, refer to Viewing Back Test Results.

Replay Back Testing - Automatic

Supported for Intraday charts only.

This section documents how to perform a Back Test by using the Trade >> Auto Trade System Replay Back Test command. This simplifies the process of performing a Replay Based Back Test.

1. Disconnect from the data feed by selecting File >> Disconnect on the menu.
2. Select Chart >> Chart Settings and set the Days to Load to the number of days you want to perform your Back Test over. Press OK. Once this is set, it does not have to be changed again.
3. Make sure there is a check mark by both Trade >> Auto Trading Enabled - Global and Trade >> Auto Trading Enabled - Chart if there is not one already. Otherwise, the automated trading system will not generate any trades.
4. Select Trade >> Auto Trade System Replay Back Test on the menu. This command clears all simulated trade data for the chart's symbol and replays the entire chart from the beginning at a high speed.
5. Sierra Chart will be busy during the Back Test. You can stop the Back Test though by pressing the Stop button on the Chart Replay window. You will have an opportunity to press this button every few seconds.
6. To view detailed results of the Back Test, refer to Viewing Back Test Results.
7. Using the command Trade >> Auto Trade System Replay Back Test cannot fully be replicated by using the Chart Replay window because the replay starts at the very first bar in the chart which is not possible to do manually. However, the settings on the Chart Replay window are as follows: For All Charts in Chartbook (Unchecked), Skip Empty Records (Checked), Accurate Trading System Back Test Mode (Selected), Speed (100000).

Replay Back Testing - Manual

Supported for Intraday charts only.

Manual back testing is used when you want to run the Back Test at a slower speed to observe what it is doing and it may be necessary to use this type of Back Test when you are back testing a trading system that involves multiple charts.

1. Make sure there is a check mark by Trade >> Trade Simulation Mode On if there is not one already. However, Sierra Chart will be placed into this mode anyway when you start a Back Test.
2. Make sure there is a check mark by both Trade >> Auto Trading Enabled - Global and Trade >> Auto Trading Enabled - Chart if there is not one already. Otherwise, the automated trading system will not generate any trades.
3. Select Chart >> Replay Chart >> Replay Chart (Control Panel) to display the replay window.
4. On the Replay Window select Replay Mode >> Accurate Trading System Back Test Mode.
5. If you want to replay multiple charts in a trading system that uses multiple charts, then select Charts to Replay >> For All Charts in Chartbook on the Replay Window. Otherwise, select Charts to Replay >> Single Chart. For additional information, refer to Performing Back Testing on a Trading System That Uses Multiple Charts.
6. Make sure the Use Start Date-Time option is disabled.
7. Scroll to the point in the chart where you want to begin the Back Test at and press the Play button on the Replay Window.
8. When the replay is started, the current simulated Trade Position for the symbol, if any, the simulated orders for the symbol, if any, and all of the simulated order fills for the symbol are all cleared.
9. For further details, refer to the Replaying Charts page.
10. To view detailed results of the Back Test, refer to Viewing Back Test Results.

Performing Back Testing on a Trading System That Uses Multiple Charts

When you have a trading system that involves multiple charts, the question is whether you need to replay or Back Test only the chart the trading system study is on, or all of the charts that the trading system makes reference to.

You will want to replay all charts that a trading system is referencing by using the Replay Back Testing - Manual method if you want to have your trading system use the price and study values from bars that are in the process of being formed rather than using study and price values in the source charts which have complete and finalized values.

For example, consider a 5 minute per bar chart. In the source chart that the trading system is referring to, if it is not replaying at the same time it will have complete 5 minute bars. Whereas if you are replaying it, gradually over the five-minute timeframe, a 5 minute bar will build and the study values will change. If it is important to have changing bar values and study values for your trading system, and only you can answer this question, then you do want to replay all charts the trading system uses.

You will want to not replay all the charts that a trading system is referencing and instead only replay or perform a Back Test on the particular chart that the trading system is on, if it is acceptable for you to have complete and finalized bar and study values in the source chart or charts.

When a trading system is making references to other charts, it needs to use one of two methods to do so. It will need to use the Study/Price Overlay study. Or in the case of an ACSIL trading system it can make references to other charts by using the method documented on the Referencing Other Time Frames and Symbols When Using the ACSIL page.

When using the Study/Price Overlay to overlay price or study data onto the chart that the trading system is on, and the chart bars are Number of Trades, Volume, Range, Reversal, Renko, Delta Volume, Price Change, Point and Figure Bars, then you need to set the Data Copy Mode study Input to Use Earliest Value from Corresponding Timeframe.

Otherwise, in the case of when the other chart being referenced is not replaying and is not synchronized timewise to the destination chart, data from the last bar in the chart being referenced will be used for the current bar being back tested. So the referenced data would not be correct.

When you are replaying multiple charts at the same time to perform a back test, then you need to use the Replay Back Testing - Manual method. On the Replay control panel window you need to select the For All Charts in Chartbook option in the Charts to Replay list if you have decided to replay all of the charts.

And on the Replay control panel window, select Accurate Trading System Back Test Mode or Calculate at Every Tick/Trade. When you do this, a synchronized Back Test will be performed. This is essential. For this type of Back Test, it is recommended that you disconnect from the data and trade servers by selecting File >> Disconnect.

With a synchronized Back Test, the dependency between the charts is determined and the calculations are done in the proper order. This ensures consistency and accuracy. You can use any Replay Speed.

When you start the synchronized Back Test replay you will be asked for the time step increment to perform the back test in (Enter Processing Step in Seconds). The default is 60 seconds. The smaller the timeframe, the more accurate the Back Test, however it will take longer to complete. The larger the timeframe, the faster the Back Test will be but it will be less accurate. However, the accuracy really depends upon the timeframe of the source and destination chart bars. You would not want to use a time step increment that is longer than the time frame of the bars being involved in the trading system. You may want to choose a timeframe which is about 25% of the average bar time length.

Viewing Back Test Results

You are able to view detailed results of your trading system by selecting Trade >> Trade Activity Log >> Trade Statistics from the menu.

At the top of the Trade Activity Log, in the list of symbols, select the symbol that the Back Test was run on. You need to choose the symbol that has [Sim] in front of it.

At the top of the Trade Activity Log, from the list of Trade Accounts, select the Trade Account that the Trade Window for the chart that the Back Test was run from, is set to. For more information, refer to Selecting Trade Account.

For complete instructions, refer to Trade Statistics Tab. There are many fields displayed which provide detailed results of the trading.

For a Trade by Trade log, select Trade >> Trade Activity Log >> Trades from the menu. For complete instructions, refer to Trades Tab.

To view trade order fills visually on the chart, enable Trade >> Show Order Fills on the menu of the main window or the chart window.

To know exactly when an order was filled, you need to look at the order fills on the chart rather than looking at the arrows placed by a trading system on the chart because those arrows may not necessarily represent actual orders accepted and filled. For additional information about this, refer to Ignored Signals With Spreadsheet Systems or Alerts (This applies to the Spreadsheet System for Trading study).

During a loaded Bar Based Back Test, only Fill entries are added to the Trade Activity Log. All the other order related Trade Activity is excluded for performance reasons.

Running Multiple Back Tests For Comparisons

It is supported to run a Back Test multiple times for a trading system and maintain the results of each of those tests separately. This is accomplished by using a different Trade Account before starting the Back Test. To do this, go to the Trade Window for the chart the trading system is on and select a different Trade Account before the start of the Back Test.

For instructions, refer to Selecting Trade Account.

The results of the back tests are held in the Trade Activity Log. Refer to Viewing Back Test Results.

One thing to be aware of is that when you are running multiple Back Tests for the same set of data and the same trading system, the data in the Trade Activity Log accumulates. There are going to be multiple order fills which will often occur at the same time. The Date-Time of each of these records in the Trade Activity Log for these order fills cannot be the same. Sierra Chart will assign a unique millisecond value to each of these records to keep them unique. So you will see order fills at the same time but have a slightly different millisecond value. This is important to understand if you are running a Back Test with identical settings but under a different Trade Account. This in no way affects the consistency of the trading system. This is nothing more than a Date-Time stamp for logging purposes.

Improving Back Test Performance

The following are items for improving Back Testing performance.

1. The fastest back testing is going to be when using the Bar Based Back Testing back testing method. Inherently, Replay Back Testing is going to be slower since individual Intraday data records within each chart bar going to be processed which takes more time.
2. In the case of when using the Spreadsheet System for Trading, refer to Improving Back Test Performance of Spreadsheet Trading Systems in addition to the items below.
3. Developing an automated trading system using the Advanced Custom Study Interface and Language is going to always give the very best performance compared to using the Spreadsheet System for Trading study.
4. When running a replay type of Back Test, if the data in the Intraday chart data file contains 1 Tick or 1 Second data records, back testing will be slower compared to higher timeframe data records in the Intraday data file.
   
   Therefore, increase the Global Settings >> Data/Trade Service Settings >> Intraday Data Storage Time Unit. After this is done, re-download the Intraday chart data by going to the Intraday chart and select Edit >> Delete All Data and Download. This only needs to be done once per symbol. This will help reduce the time for Back Testing.
5. The next thing to check to determine the source of slow Back Testing is to remove all of the studies from the chart or charts that are being replayed, or from the chart a bar based Back Test is being performed on. Run the Back Test again and see if you get better performance. If so, then it is one of those particular studies causing the problem. You can then add studies back gradually one by one to see which one is taking a longer time to calculate.
6. If a Custom Study is causing a performance problem, then the developer of that study needs to improve the performance of that study.
7. Make sure that Replay Mode on the Replay Window is not set to Calculate At Every Tick/Trade.
8. Disable Global Settings >> Chart Trade Settings >> Enable Estimated Position in Queue Tracking.

Improving Back Test Performance of Spreadsheet Trading Systems

In the case of when using the Spreadsheet System for Trading study, Back Testing may not be very fast. The reason for this is because all of the chart data is outputted to a separate Spreadsheet object.

Every time there is a new bar added to the chart during a Back Test, the existing data has to be shifted down in the Sheet. This can cause a lot of calculations. Inherently using the Advanced Custom Study Interface and Language is much faster.

To improve Back Test performance, follow the steps below.

1. Look at your formulas in cells K-Z (last formula column when using 16 formula columns). Determine how many rows from the current row 3, they refer back to. For example, if the formulas refer down no further than row 12, then the Spreadsheet Study only needs to output 10 rows of data to the Spreadsheet.
2. Reduce the Number of Rows input in the Study Settings window for the Spreadsheet System for Trading study to the minimum number of rows required.
3. Clear the data from the Sheet rows that are no longer used on the Spreadsheet. To do this, highlight those rows by selecting the row numbers on the left side of the Sheet of the Spreadsheet and press the Delete key on the keyboard or select Spreadsheet >> Clear Selection.
4. Clear the data from the Sheet columns that are no longer used on the Spreadsheet. To do this, highlight those columns by selecting the column letters on the top side of the Sheet of the Spreadsheet and press the Delete key on the keyboard or select Spreadsheet >> Clear Selection.
5. Clear any other unused formulas.
6. Remove any unused Sheets within the Spreadsheet. At the top left of the Spreadsheet window, select the particular Sheet# that is not used, and select Spreadsheet >> Delete Sheet.
7. Perform the Back Test using one of the methods described on this page. The fastest back test will be a Bar Based Back Test.
8. When using a Bar Based Back Test, the speed is affected by the Chart Update Interval setting. For the fastest Back Test, you will need to reduce the Chart Update Interval directly on the chart itself. Make it a small value like 100 to 200 milliseconds.

Differences Between Back Testing and Real-Time Auto Trade System Evaluation

The BuyEntry, BuyExit, SellEntry, SellExit Order Actions are evaluated when the trading study is calculated. During normal real-time chart updating this calculation happens at the Chart Update Interval set in Global Settings >> General Settings. Each individual chart can also specify their own independent Chart Update Interval.

However, when the Chart Update Interval elapses, the trading studies will only be calculated when there is data being received from the data feed, or there is new/updated trade orders or position data. During a Back Test, there are definite rules that control when trading studies are calculated.

For a Bar Based Back Test , the trading studies are calculated for each of the Open, High, Low, Close values in the bar.

During a Replay Back Test, the trading studies are calculated when there is a new bar added to the chart and any time there is a change with the High, Low, Close values of the last bar in the chart as the data is processed through the Intraday chart data file.

During real-time updating, between the times the trading studies are calculated which is at the Chart Update Interval, there potentially may be more than one data record added to the chart. This is especially true with tick by tick data.

Therefore, the study is not calculated at every single tick. If you want your Order Actions evaluated as fast as possible during real-time chart updating, then you may want to decrease the Chart Update Interval. We do not recommend going lower than 50 to 100 ms.

The Bid and Ask prices which are used to fill the orders, can be different between Back Testing and also during real-time updating. For more information, refer to Trade Simulation Accuracy and Bid/Ask Prices During Replays.

The point of this discussion is that there can be some inconsistencies between a trading system performing in real-time and during a Back Test due to the conditions and frequency for when the studies including the trading study are calculated.

You will have the greatest consistency between back testing and real-time auto trade system evaluation when you are performing a Replay Based Back Test on tick by tick data. To ensure you have tick by tick data in the Intraday data files, refer to Tick by Tick Data Configuration. However, this does not mean that back testing will match real-time auto trade system evaluation. It will not.

In the case of the Spreadsheet System for Trading study, if you want to evaluate your condition formulas only on a bar close, then set the corresponding Signal Only on a Bar Close inputs with the Spreadsheet System for Trading study to Yes. This will give your system greater stability and not be affected by the Chart Update Interval. In the case of an ACSIL trading system, you will want to use sc.GetBarHasClosedStatus().

In the case of a trading system which use multiple charts, during real-time updating of charts, to control the calculation order of the charts based upon the dependency between them you need to enable Global Settings >> General Settings >> Use Controlled Order Chart Updating. This provides higher consistency between back testing and real-time auto trade system evaluation for trading systems that use multiple charts.

When performing a Replay Back Test, try performing it at a slower speed to see if you notice more of a consistency. Whether you will or not, is uncertain.

If the trading system involves multiple charts, then make sure that all charts are replaying. This can be done with the Charts to Replay setting.

Another cause for inconsistencies between back testing and real-time auto trade system evaluation is that when you are using the Trade and Current Quote Symbol, real-time auto trade system evaluation is going to be based upon the market data for this Trade Symbol. When performing back testing, the automated trading system is evaluated using the data in the chart which is from the main symbol of the chart. Therefore, during back testing, change the main symbol of the chart to be the same as the Trade and Current Quote Symbol.

Back Testing and Continuous Futures Contract Charts

It is supported to perform Back Testing when using one of the Chart >> Chart Settings >> Symbol >> Continuous Contract options.

When performing back testing across a Continuous Futures contract chart, although the expired futures contracts are loaded into the chart, the chart only has one symbol which is used for trading even though you are Back Testing across historical data. So the trades are occurring on expired contract prices, but being specified with the current symbol. Therefore, you will only see one symbol when reviewing the trading results in the Trade Activity Log.

Consistency Between Back Tests

When performing Back Testing, you must use one of the methods documented on this page and follow the procedures as given.

When performing a Replay Based Back Test, on the Replay Window you must select Accurate Trading System Back Test Mode or Calculate at Every Tick/Trade in order to have consistency when Back Testing the same automated trading system under the same conditions.

During a Back Test, when Sierra Chart is processing through the underlying Intraday chart data records for a Replay Based Back Test or through the bar data for a Bar Based Back Test, the Bid and Ask prices are set from the underlying data records in the Intraday data file or are derived from the Open, High, Low, Close bar data. These Bid and Ask prices are used to fill the orders which are submitted by an automated trading system.

There is always a 100% consistency with the setting of the Bid and Ask prices between Back Tests on the same chart data as the data is processed, assuming the same back test method is used ( Replay or Bar Based).

If your trading system gives inconsistent results between Back Tests without making any changes to the chart data, the Chart Settings or to Study Settings, you must look at your own trading system code/formulas for a cause since your code is a major and primary portion of the overall functioning of an automated trading system.

When there are inconsistencies, this can only be attributed to your own code. It is very unlikely to be the result of how Sierra Chart is processing the back test. Inconsistencies reveals that your own trading system produces inconsistent results based on the way that it is designed and working.

Even if you do not get a consistent result between Back Tests, generally this does not necessarily mean very much because although you may want to see a consistent result every time you run a Back Test, assuming that your study is working consistently and Sierra Chart does not have any control over that, it does not change the fact that with live trading you will get yet another different result from your trading system.

Therefore, to insist that Sierra Chart gives a consistent result of your trading system which it cannot guarantee because you wrote your own trading system and it has its own behavior, will accomplish nothing. While it may seem better, it is not going to change the fact that your trading system itself is not stable if it gives inconsistent results, and is going to give a different result during live trading.

Try one of the Sierra Chart provided trading systems. Run it many times. You will get a consistent result every time. If you do get a consistent result every time, and with your own trading system you do not, then this proves the inconsistency problem is within your own trading system. If you do not get a consistent result with a Sierra Chart provided example trading system, then consult with Sierra Chart support for the reason.

The Sierra Chart example trading systems can be found in Analysis >> Studies >> Add Custom Study >> Sierra Chart Custom Studies and Examples >> Trading Example: *.

If you are Back Testing a trading system that involves replaying multiple charts at the same time, then this is something which involves a lot more complexity. This is something to consider if you have inconsistencies between Back Tests in particular with varying replay speeds. In this particular case inconsistencies can be possible. Also, make sure in this case the Charts to Replay option on the Replay Window is set to All Charts in Chartbook or Charts with Same Link Number. Greater consistency can be achieved in versions of 1863 or higher of Sierra Chart which support the Replay Modes Accurate Trading System Back Test Mode or Calculate at Every Tick/Trade when replaying multiple charts.

Comparing Bar Based Back Test and Replay Back Test Results

When Back Testing an automated trading system using either the Bar Based Back Testing or the Replay Back Testing - Automatic method, each method will give a different result.

The more accurate result would be from the Replay Back Testing method. However, it is more time-consuming using the replay method.

Using Actual Bid and Ask Prices During Back Test

Sierra Chart supports using the actual Bid and Ask prices during a Replay Based Back Test. These Bid and Ask prices are used to fill orders. This provides a very high degree of accuracy when Back Testing.

Historical Bid and Ask prices are available when using historical data provided by the Sierra Chart Historical Data services. In most cases this service is being used for historical data for charts. This historical Bid and Ask price data begins on June 23, 2014.

Follow these steps to use actual Bid and Ask prices during back testing.

1. Set Sierra Chart to store data tick by tick. Refer to Tick by Tick Data Configuration. This is an essential step.
2. Perform an automatic or manual Replay Back Test. It must be a Replay Back Test.

Troubleshooting Automated Trading System Behavior

If an automated trading order action is ignored, the reason for this will be given in the Trade >> Trade Service Log. Refer to this log for the reason why.

When using the Spreadsheet System for Trading study, refer to Ignored Signals With Spreadsheet Systems or Alerts for further information.

It may also be helpful to go through the Spreadsheet System for Trading Test Procedure to help with understanding how a basic Spreadsheet automatedtrading system will work and indirectly provide help to troubleshoot your automated Spreadsheet trading system.

For an ACSIL automated trading system, refer to Debugging/Troubleshooting Automated Trading Systems.

Also, refer to Troubleshooting Automated Trading System Behavior.