Support Board
Support Board
Search Board
Control Panel
View My Posts / Threads
Direct MessagesDate/Time: Thu, 19 Sep 2024 16:42:06 +0000
[User Discussion] - ACSIL study - inconsistent results with replay back test
View Count: 1999
| [2014-02-25 14:37:37] |
| dominikos - Posts: 106 |
|
I'm backtesting a custom trade study with futures data from CTS saved to a file. To eliminate variation, I hard coded the period for the chart. The same test executed produced multiple times produces slightly different results, in order of tens of dollars. My strategy does not use time and only calculates once the bar is completed. Is there any reason why the test results should be different when run on the same data set with the same strategy?
|
| [2014-02-25 16:51:05] |
| |
|
How are you performing the back test? From the command on the Trade menu?
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2014-02-25 17:06:32] |
| dominikos - Posts: 106 |
|
yes, I do the following: Trade->Clear Trade Simulation Orders & Data Trade -> AutoTrade System Replay Back Test jump is 00:01:00, speed: 1.0 my chart uses data range, chart is "number of trades per bar", and new bar is at session start, session is 00:00:00 23:59:59. I hope this helps. |
| [2014-02-25 17:27:52] |
| |
|
Are all of the trades performed once a bar is completed? There can be a slight difference in the number of times a study function is called during a back test due to timing differences related to when control is yielded back to the user interface. If a trading system is programmed in the most stable way and only generates trades at a bar close, there should not be any difference with results. However, we do not know exactly how your study function works. We do plan to make the number of calls completely consistent . However, as a practical matter this does not mean much, because when you do live trading, you will end up with yet another result. So any small variance in back testing, would be very small to differences you would see with live trading. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2014-02-25 17:29:06
|
| [2014-02-25 17:39:51] |
| dominikos - Posts: 106 |
|
I do check for bar closed status if (sc.GetBarHasClosedStatus() == BHCS_BAR_HAS_NOT_CLOSED) { return; } This most likely have to do with how simulation executes trades, I use limit to buy and stop/limit to sell. What concerns me is that sometimes the discrepancies are larger. |
| [2014-02-26 21:25:09] |
| |
|
The trade simulation system is going to execute a trade always identically as long as the bid and ask prices are the same at the time an order is given, and we would expect that would be the case. What this is indicating, is that your trading system is not giving consistent results based upon slightly different number of times it is called even though the current bar formation, bid, ask and last trade prices are the same when it may be called an additional time. You need to look at your own trading system for a cause. When these issues come up, they cannot always be attributed to something on the Sierra Chart side. An interesting test would be to use a persistent variable and reset it when sc.Index = 0 . Increment it on every call into the study function. When the back test is finished you can then check the value of that variable and see if it is different based upon different back test results. You can use this to determine when the replay back testing is done: https://www.sierrachart.com/index.php?l=doc/doc_ACSIL_Members_Variables_And_Arrays.html#scReplayStatus We can ensure that the study is called a consistent number of times and in the next release or two, this will be resolved. However, it does not as a practical matter mean anything because although a user might feel better that they get a consistent result every time they run a replay, assuming that your study is working consistently, and we have no control over that at all, it does not change the fact that with live trading you will get yet another different result. This is why the very end, you must focus in on your own system and find out where the problem is because that is primarily or completely where the problem lies. For example, try one of the Sierra Chart provided trading systems. Run it 100 times. You will get a consistent result every time. We would be interested to find out otherwise. We are going to post some of these comments in our back testing section because periodically we get these kinds of questions coming up and users like to push the responsibility to us, but they are missing the point of where the problem actually lies and do not realize that the inconsistency actually reveals the nature of their trading system. If there is something that we can improve upon, then we will do that. One improvement is that we will be storing the actual bid and ask prices that existed in real time and they can be used for replay for greater accuracy. And we would like to know, if you run our own trading systems if you end up with inconsistent results after running them repeatedly. If not, then you can see where the problem actually lies, and that is in your own code and it reveals that your own code does not function in a consistent way. And in this case, your focus must be on your own system. Not trying to see why Sierra Chart does not make it work consistently because it cannot. Ultimately this is up to you. If you do notice an inconsistency with the results of the Sierra Chart provided trading systems, then we will look into it and see where the problem is. We have just done some additional testing right now using one of the Sierra Chart provided systems and got a completely consistent result out of it as expected after two tests. Nevertheless, we want to know what your results are. The Sierra Chart example trading systems can be found in Analysis >> Studies >> Add Custom Study >>Sierra Chart Custom Studies and Examples>> Trading Example*. Another thing has come to mind, if you are running a trading system, that involves multiple charts, then this is something very different and there is a lot more complexity to this. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2014-02-27 04:52:53
|
| [2014-02-27 03:59:40] |
| dominikos - Posts: 106 |
|
Thank you for the pointers. I will report back once I test it, hopefully will provide some benefits to others.
|
To post a message in this thread, you need to log in with your Sierra Chart account: