Support Board
Date/Time: Tue, 26 Nov 2024 11:46:11 +0000
speed backtesting raffect reliability
View Count: 1521
| [2023-01-06 14:18:40] |
| User719206 - Posts: 93 |
|
i READ on this forum that the backest is reliable , but more i improve the speed more it s missing trade ,if i come back at the same date with a slower backtest speed the trade is filled , it's in standart replay. does it normal or do i something wrong ? |
| [2023-01-06 16:14:35] |
| User719206 - Posts: 93 |
|
it's the case even in calculate every tick mode , how can I have a good reliability and a fast backtest ?
|
| [2023-01-06 18:52:50] |
| User61168 - Posts: 403 |
|
Here's my experience backtesting for past 3 years in Sierra Chart. My experience is limited strictly to simple alerts only so keep that in mind. - Never use standard replay if your entries are tick based. if you have entries at bar open/close then this is probably okay just to eye ball and test your logic is working or now. Regardless, I have stopped using this mode . - "Accurate trading system back test mode" - this is far from accurate. I never use this mode at all. - "Calculate same as real time" - this is also flaky and dependent on speed. for simple study collections, it works well but if you go above 25+ studies, it becomes highly unpredictable. I stopped using this also - "Calculate at every tick" - I had to invest in a latest CPU model laptop last year out of sheer frustration and now this is the only mode I have chosen to use and still I never go above speed 30 taking me days to complete my backtest. 100% of my backtest results from old laptop had to be discarded after buying the new laptop. - Breakdown your backtesting range into 3 months interval. It drastically slows down if the historical data file goes above a gig in size Overall, my experience with backtesting has been highly unpredictable so never ever flip around these modes.... test one setting, validate it works 100% on a small sample and then stick to that setting in order to maintain consistency. Even after all these precautions, take backtesting with a grain of salt imo and expect live results to vary :- |
| [2023-01-10 12:04:40] |
| User719206 - Posts: 93 |
|
thanks a lot for your answer User1168 , it's absoluly not what i would read but thanks to write it , I will follow your advice :)
|
| [2023-01-10 18:07:03] |
| User61168 - Posts: 403 |
|
You are welcome. It's worth the time spent if others can learn from my mistakes. I used to be a replay monkey spending hours and days testing silly strategies. Here are some better options to incorporate in your journey (and to the benefit of other readers) 1) Add Time to chart header via Global Settings > Customize Chart Header. This will reflect the "true" speed of chart replay. I usually set the speed to a level such that chart moves at the speed of 1sec for every minute. 2) There are many documented features on how to improve overall performance. A huge one is to display very few bars on the chart by zooming in. To achieve consistency, use cntrl+shift+mouse scroll wheel to zoom in by watching the gap between bars in pixel in the bottom right corner of the chart. I use a 130 pixel gap to ensure consistency. I usually show only one or two bars on my chart. Also resize the chart to make it small in size instead of window being in maximized state.... less things to draw, less things are visible, lesser is the load on GPU/CPU to process 3) You could also hide all studies during replay via Customize Control Bar and adding Hide and unhide studies button. This will improve chart performance tremendously. Do a Hide all studies right before you start the replay. Also set subgraphs to ignore if you are not referring it in any other studies or formulas. There is no need to hide them if they are not being used 4) You could also use menu item Window > Hide Window to hide the entire chart for even faster performance 5) I like my charts to not freeze during replay. If there is a lag in movement of current price, you are running your replay too fast. 6) remove any timer studies if you are using it only for visuals. they slow down the replay 7) remove current price line study from your collection. Something about this study slows down the chart replay. Also avoid using Chart > Bid Ask Average line or show bid/ask lines. They are only there for the visual effects. You don't need these such features that choke up valuable CPU cycles Now, all the above is for market replay performance but here's the biggest tip to minimize our reliance on market repeat replays during "strategy design" phase of your journey: Once a replay is complete for the desired date range, Do all your analytics on the chart or off the chart by doing the following steps: 1) Use simple alert color bar condition to do all your analytics on trade performance. enable Sum of Alerts subgraphs to run your stats 2) Add Write Bar date to file study but be careful not to use this study during replay. This will export all your study data into a flat file to export in excel or google sheets. 3) Alternative to #2, you can use Edit > Export Bar and Study data to text file which is much faster. Both #1 and #2 can print only visible subgraphs so hide/ignore subgraphs you do not want to analyze. And lastly, Believe me or not, a positive expectancy "edge" lies in only one or two things so break it down into smaller and simpler building blocks. If you have a engineering or development background (like I do), we geeks try to over-think and over-complicate stuff by doing too many things in a study collection. At some point in my journey, I had over 150 studies in my collection... now I have zero indicator studies with whatever mathematical equations I need developed in the alert condition with approx 10 universal studies that apply to all my strategies and just 1 color bar alert condition per strategy. My tick-based market replay is super fast at 120 speed after such simplification. Hope this adds further to your learning curve. Sierra is a fantastic analytical platform with many hidden and undocumented features and powerful capabilities. Possibilities are limitless in creating strategies. It just takes time! Date Time Of Last Edit: 2023-01-11 00:02:56
|
| [2023-02-04 18:18:59] |
| User719206 - Posts: 93 |
|
Thanks a lot ! we should include your answer in the documentation. I followed each of your recommandation and it helped me, but 1 second for every minute ,we are not in the same reality ^^ me it's 1 minute = 1 minute more or less. I suppose you practice the single replay chart and not multichart ? Because me i even not use tick by tick data , I have a new powerfull computer , I followed all the sierra recommandation (time storage unit ...) I Have 9 chart , I have 18 Volume by price study for each of them , and it give me 1 minute = 1 minute , like a real time replay Date Time Of Last Edit: 2023-02-04 18:25:55
|
| [2023-02-04 21:27:41] |
| User61168 - Posts: 403 |
|
You are welcome. Everything I listed is based on a single chart replay. I tried multi-chart replays which is a great feature in SC but I would use it for no more than few days of data. Ultimately, we all need to understand the reality of "bigger the physical size of scid file, slower will be the replay". This limitation will be in any feature-rich software package that runs on a personal computer. If you are not using or not requiring tick by tick data and if you entries and exits are based on OHLC (i.e. entries and exits only on candle close) then there is really no need to do market replays. Just code and test the individual alert conditions would be my preferred choice. And, if you are not aware of this, you can run the alert conditions recursively or refer the output of alert condition into the same alert formula or in different alert conditions. Labeling it as a "simple alert condition" is a misconception imo. It can do just about anything a normal programming language can achieve if you structure the alerts properly. You can also run other studies like moving averages on the output of alert conditions or based output of other studies into the alert by using "Based On" etc... extremely powerful capabilities imo. Respectfully...9 charts with 18 volume by price studies on each chart requires a LOT for computing power for any application to deal with. Also, I would question the accuracy of backtesting when using volume profile studies without "tick by tick" data. Your live/forward testing would be quite different from backtesting results. Complete waste of time imo. Edit: Also, I am not sure if this is still true in the latest versions.... Market Replay utilize only one CPU core. This is unfortunate as we will always be limited by the computing power. I have tried running market replays even on the dedicated server from hosting providers, hired engineers to tweak the server performance, went with 16+CPUs, 100+GB RAM and dedicated SSD drive.... absolutely no performance improvement compared to running it on a latest generation laptop. Date Time Of Last Edit: 2023-02-04 21:35:21
|
| [2023-02-05 00:12:28] |
| User719206 - Posts: 93 |
|
in fact i need to do some replay to find the alert condition , i target a pattern . I'm able to reconise it with my spreadsheet trading system. I know what are the factor wich make it more profitable , each time a trade is trigered , i collect something like 10 data that i send in text tag ,to analyse it in excell with pivot table and logisitc regression. it allow me to find conditions that work well, but it's really seasonal (really profitable 3 weeks , and loosing durind 2 weeks ect , so it's trading nothing new) , I think that the context can allow me to understand better , this is why i want to leaked each trend , and collect the data from each of them for each trade to cross with this data too , and make logistic regression and try machine learning.(and it askig a huge amount of data) this plan is supposed to give me a more or less reliable estimation of the percentage that the trade will be winning ,i compare it with the risk reward to know if the probabilitys are with me. It's just a tools who give me an indication. I use VBP to tag the last 300 bar and know where the different POCs are essentialy , so I dont need tick by tick data ,the younger poc in my chart is 30 bar.The 18 vbp study are not really the probleme I think, when I run backetest without the spreadsheet is verry fast. the probleme is this sprendsheet with his 24 column and 300 rows, and all this unoptimized formula,(sometime the time calculation for it is more than 400 ms) I supose I should code a custom study with acsil, but I dont code. Maybe I have to make code it by someone, accepted it's impossible and forget this context data or be verry patient to collect it and make it with this verry slow replay. Date Time Of Last Edit: 2023-02-05 02:45:57
|
| [2023-02-05 03:14:02] |
| User61168 - Posts: 403 |
|
24 column and 300 rows,
I understand where you are in this journey. I have no experience with spreadsheet trading. Seems you need to reduce the 300 row count to improve the CPU cycles involved in look-back. 400ms is too much to process at every tick.I have gone thru the journey of over-complication to over-simplification so I feel your pain and the associated complexities of trying to improve winrate etc. In my experience, optimizing or adding more filters via excel based analysis has not yielded in too much success unless 1) you are exporting tick data into excel and running pivots at the granular level-it would be too much for excel to handle so I gave up on this route 2) your entries/exits are based on OHLC data. Also, watch for repainting of current bar and associated indicators when using non-time based chart type. Your analysis in excel may not account for such hidden scenarios A better alternative (again, based on my experience) is to focus on simple filters like time of day, day of week, stop trading when profit/loss is met etc and if the results are still not favorable, scrap this strategy or add a new strategy to counteract the losses from the 1st strategy. Example: if strategy one is trend following and losses pile up, add a second mean-reversion strategy to prevent or slow down the drawdown. p.s you might want to move this thread to user discussion and get feedback/help from others. |
To post a message in this thread, you need to log in with your Sierra Chart account: