Support Board
Date/Time: Mon, 25 Nov 2024 18:05:57 +0000
[Programming Help] - Renko Chart Bar Based Backtesting Flattening Issue
View Count: 773
| [2024-02-15 14:26:07] |
| gtaranti - Posts: 67 |
|
I'm using an automated trading system with an ASCIL study in a 3-tick Renko chart for Eurostoxx 50 and all in all it's quite nice and performant, compared to the Spreadsheet approach. It's an intraday strategy, so at the end of the day (22:00 CET) the position (if any) must be flatten. So, I've set a flatten rule using sc.BaseDataEndDateTime[sc.Index] and after convert it to time I'm comparing with the time 21:00, in which I want every position flattened. Also I set a debug message in Message log to see if it works correctly. Overall it's working OK and I see messages like : 2024-02-15 12:57:03.021 | Chart: Bar Backtest: FESXH24-EUREX [CBV][M] Renko 3t #1 | Study: VWAP Pullback | Flattening... Time: 21:22:25
But here is the issue: Many times where the Renko bar closes AFTER 22:00, (due to low volume or anything) the position is kept open beyond the EUREX Exchange close and ofcourse the trade backtest statistics are off. I've tried also to use the sc.CurrentDateTimeForReplay variable to check for the close position time but without success. My questions are : 1. Is this problem due to the Renko charts (or any other non-time-based charts)? 2. Can we do something to remedy it while continue using Bar Based Backtesting? 3. Would it be the same if I did a full Replay Backtest? Date Time Of Last Edit: 2024-02-15 14:27:21
|
| [2024-02-17 04:33:21] |
| User43 - Posts: 101 |
|
in my back testing I close positions before market close (here on the US east coast markets close at 16:00) at 15:55. And for not time based bars I test if the time is equal or greater than the close time. Testing just equal 15:55 on volume charts will not work. On time based strategies the time frame must match up with the closing time like 15:55 will work on a 5min chart but not on a 10min or 2min chart. 15:50 would work for 2 or 10 minute charts, or again just test if the time is greater than the expected. |
| [2024-02-17 07:30:52] |
| gtaranti - Posts: 67 |
|
I'm refering explicitly to non-time based charts. You say And for not time based bars I test if the time is equal or greater than the close time.
How do you test the close time? sc.CurrentDateTimeForReplay ? |
| [2024-02-17 09:37:13] |
| User61168 - Posts: 403 |
|
a simple 'trading based on alert condition' study for buyexit and sellexit with time filter of 16:00:01-17:00 does the magic of exiting all open positions and cancelling all open orders upon session close.
Date Time Of Last Edit: 2024-02-17 09:37:39
|
| [2024-02-17 16:32:06] |
| User43 - Posts: 101 |
|
Sorry, I could have saved the detail about time based checking. I'm aware that bars on a renko charts don't close on specific time. User61168 made a great suggestion, which I'll try eventually. Not 100% sure though if it will manage closing precisely when doing bar based back test, or replay with acceleration. In my scripting I use the following to close my position based on passing a specific time. int BarTime = sc.BaseDateTimeIn[sc.Index].GetTime();
bool TradingAllowed = ( !Input_TradingUseTimes.GetYesNo() || (BarTime >= Input_TradingStartTime1.GetTime() && BarTime < Input_TradingEndTime1.GetTime()) || (BarTime >= Input_TradingStartTime2.GetTime() && BarTime < Input_TradingEndTime2.GetTime()) ); bool CloseAll = (Input_TradingUseTimes.GetYesNo() && BarTime >= Input_CloseOpenPositionTime.GetTime() ); When CloseAll becomes true I'll exit all open positions. In the past when I compared using equal BarTime == Input_CloseOpenPositionTime.GetTime()
my positions would not close as expected when trading on a volume chart or having a misaligned time base and exit time.So I changed it to compare >= (equal or greater) which works fine for me. Since I don't want to hold my positions into the close I use a close time of 15:55 (5 minutes before close). That probably can be pushed closer to 16:00. I think something like 15:59:50 (10 seconds before market close) might still work. If you want to exit precisely at a specific time in your script, you need to take several details into account. ACSIL Interface Members - Variables and Arrays: sc.CurrentDateTimeForReplay only works on replay and recommends to turn on ACSIL Interface Members - Variables and Arrays: sc.UpdateAlways . The description also states that during accelerated replay not each second might be triggered. However, when there is an accelerated replay, a study function will not necessarily be called at the interval specified by the Chart Update Interval relative to the replay times. For example, if the chart is set to update every 1000 milliseconds, during an accelerated replay it will not necessarily be called every theoretical second when using sc.UpdateAlways = 1.
Also internally time values are floating point variables. Therefore they suffer from precision issues when comparing them. see ACSIL Programming Concepts: Floating Point Value Error |
| [2024-02-17 16:43:25] |
| gtaranti - Posts: 67 |
|
Thank you both @User43 and @User61168.. I'll definitely try your suggestions and come back with my results.. |
| [2024-02-17 16:54:17] |
| User431178 - Posts: 541 |
|
int BarTime = sc.BaseDateTimeIn[sc.Index].GetTime(); As you are not using time based bars, you might be better suited using one of the below, as you would then not be reliant on a new bar starting. sc.LatestDateTimeForLastBar.GetTime(); sc.BaseDataEndDateTime[sc.Index].GetTime(); If you go with the end date time option, you also need to set sc.MaintainAdditionalChartDataArrays in the set defaults section. Also internally time values are floating point variables. Therefore they suffer from precision issues when comparing them. This is not true, they used to be, but since more than 3 years have been 64-bit int. |
| [2024-02-18 05:00:30] |
| User43 - Posts: 101 |
|
User431178 thanks for the pointers, I'll look into them. So far I'm happy though with the results. I don't care about a minute, if that would make a difference in my back testing, I think my strategy is over fitted. So far my close times are acceptable close to the intended time I've set. Good hint about the int64 for time values, they should then be precise then. |
| [2024-02-18 05:37:23] |
| User61168 - Posts: 403 |
|
I think my strategy is over fitted - if I may..... this is a real problem in strategy development. How do you backtest and build entry/exit criteria and avoid overfitting or curvefilling or parameter optimization (whatever you want to call it)? Here are some techniques/methodology I follow but before you read further, I want to caution you, backtesting will frustrate the life out of you :-)1) Never EVER manually scroll charts back in the history more than 2-5 days if you are an intraday trader! Starring at charts forms all sorts of biases. 2) Never ever watch charts while backtesting during market replays. Only watch to troubleshoot and/or confirm the innerworking of strategy is working as intended. Just blank out the bars or zoom in to show only one bar on the chart. 3) Use simple alerts to check the trade frequency/volume of entry signals 4) After running backtests and if the results show merit with some losing days, do a date-exclude of all winning days and show only losing days in the chart. Hide all the order fills. This becomes your new BLANK canvas to build a completely new/different entry/exit signal. Wipe your brain from the previous trade signal or results. 5) Design a completely new strategy/entry setup. Test this new setup on the filtered losing days to see if it has any merit. If not, scrap the entry signal and go back to the drawing board. 6) once #5 shows some good results, remove the date-exclude filter and run entry from #5 on all days 7) if #6 shows some good merit in overall results, repeat 5 and 6 steps. if not, run the combined signals from 3 and 5 to see if there is any edge in both entry signals together. 8) rinse and repeat until the overall results show merit. 9) if 8 is a success, now the fun begins. Randomize the start and end dates to run atleast 5-10 backtest results to check how strong is the foundation of your edge. CAUTION: 99% of strategies will fail in this step forcing you to either 1) go back to #3 to start all over 2) just f*ck backtesting and go live to see what happens lol Edit: if this approach is too much painful, you could just pick any random strategy, go to trade activity log, apply day of week or time of day filters, go live and take trades only on the profitable weekdays or hours and see what happens. You might get lucky lol... this way it would take minutes to curvefit any strategy of choice :-) Date Time Of Last Edit: 2024-02-18 05:44:46
|
| [2024-02-18 08:41:10] |
| gtaranti - Posts: 67 |
|
Thank you @User61168 for taking the time to share your methodology to backtesting. It's really valuable. I'me new to backtesting and I wanted to ask why Bar Based Backtest show so different results from the Replay Backtest? It's supposed to give you a rough estimate, but in my findings the results are very different, especially in backtesting larger periods of time. What is your preference on that? |
| [2024-02-18 09:30:07] |
| User61168 - Posts: 403 |
|
rough estimate this applies to each and every type of backtest mechanism out there. No solution within or outside of Sierra Chart is 100% accurate imo. Almost all backtests rely on the quality of dataset (intraday or historical) and with SC,a number of things could mess up easily. The numerous configuration chart settings, chart update interval settings, automatic/continuous rollover settings, start date/time versus actual presence of data in the scid file, actual speed used in running replays etc etc. I hate to say this but backtesting should be used only to validate/confirm the inner workings of a strategy and that's about it imo. After years of spinning wheels with replays, it has been a complete waste of time for me. Better to forward test with a live feed in sim or using micros via "Use as Trade Symbol only" setting on a small live account. Across the 4 selections within market replay, I can only recommend using "calculate at every Tick" mode. I use "Calculate same as realtime" to quickly debug/validate my collection. The other 2 in the dropdown list are unreliable. Never tried bar-based backtest so I am not sure what values it offers. I also could not make sense of using replays with 1) Chart overlays, 2) Additional Symbol 3) multi-chart replays via all charts in chartbook or via linked charts. These are all just toooooo darn slow and time consuming and not very accurate even in tick accuracy mode. especially in backtesting larger periods of time I hate to say this but backtesting "larger" periods on tick data is a pipe dream. It will take few days to backtest using more than 1 year of tick data with CPU running hot big time. disclaimer: this is purely based my own experience with replays. Others might have a different opinion or positive experience. |
| [2024-02-18 10:18:55] |
| User431178 - Posts: 541 |
|
I don't care about a minute, if that would make a difference in my back testing, I think my strategy is over fitted. So far my close times are acceptable close to the intended time I've set. Yes, sorry, should've been clearer, that was aimed at gtaranti in reference to the use of renko bars. Many times where the Renko bar closes AFTER 22:00, (due to low volume or anything) the position is kept open beyond the EUREX Exchange close and ofcourse the trade backtest statistics are off. |
| [2024-02-18 16:22:51] |
| User43 - Posts: 101 |
|
User431178 and User61168 Thanks for sharing your experience, and I could not agree more, ... Back testing is a pseudo science. Easy to get suckered into results which will not hold up when trading live. Over fitting and forward looking are factors which easily sneak their way into strategy development. Personally I like to trade only closed bars, typically anything between 2 minutes and 10 minutes charts or on volume charts 1000V to 5000V on MES. Often I see good looking patterns at first glance, with the help of back testing I then find that these setups mostly fail more than they win. With that setup and since my strategies only trigger on a close of a bar, I use bar based back test with good results and performance. I can back test even a year without taking too much time. Typically I run my back tests initially for about 10 days to debug my trading study, and then for a month using different months, as well for half a year. But I still learn and develop my back testing process. @User61168 really great pointers on your process, which I'll for sure keep in mind and integrate into my approach. I did try out replay back testing, but that was just too slow for me (Impatient me ;-). Per my understanding you would need that if your system acts on tick by tick price and volume information. Like Bid Ask volume spread, Footprint data, trade price action on a open bar, ... Bar based can't do that. For that I would probably debug using a fast running replay and then test overnight several trading days at slow speeds. On promising candidates I'd let it run at real time speed overnight and observe also during trading session. |
| [2024-02-18 17:44:19] |
| User61168 - Posts: 403 |
|
Thanks. Couple of more perspectives…. 1) I would not trade (and test) pure micros as trading cost is just too high and unsustainable to scale up. Better option is to generate trade signals on ES and enter trades on MES allowing you to smoothly transition over to ES as account size grows. 2) I don’t use timecharts but since you mentioned volume charts (which are variable time/variable length bar charts), using or not using “fill gaps” option could affect results. I am still not settled on whether to use fill gaps or not since I only use market orders for all trade executions. 3) Look into variable length Renko and that’s all I can say about Renko. These look good to the eye but actual executions on tick makes a difference. 4) I have developed and tested over 500+ strategies and they all have one thing in common….. the work great for first 2-3 months and then start to decay and results become unpredictable once the symbol changes or change in fundamentals or seasonality or for some unknown reason. I would love to put the blame on continuous rollover stuff in SC but I have no hard evidence to put any blame on SC:-) One thing that consistently happens to me in replay is the “slippage” during exit trade on candle close”. During fast markets, my exits in replay take place after 3-6 bars or many times, won’t exit at all (particularly an issue with Renko imo) Based on this evidence, bar to bar trading strategies on variable length bars are fundamentally flawed. Depending on the chart settings, bar builder during high volatility news days or low liquidity days or on rollover days become all jaggered so keep this all into consideration. Anyways, good chat on sharing our unique experiences. I just wish more experienced SC users could also share their experiences on such “gotchas” and pitfalls to look for and avoid. |
| [2024-02-18 18:26:35] |
| gtaranti - Posts: 67 |
|
3) Look into variable length Renko and that’s all I can say about Renko. These look good to the eye but actual executions on tick makes a difference.
I dont' exactly follow this. Do you mean a custom variable-brick-size Renko chart like https://www.gomicators.com/gomrenko-sierra-chart/ ? |
| [2024-02-18 20:19:53] |
| User61168 - Posts: 403 |
|
Nope. Not this type. Fixed brick
|
| [2024-02-18 22:26:00] |
| Sawtooth - Posts: 4120 |
|
"Also internally time values are floating point variables.
64 bit diminished floating point imprecision, but did not eliminate it.Therefore they suffer from precision issues when comparing them." This is not true, they used to be, but since more than 3 years have been 64-bit int. 1. Is this problem due to the Renko charts (or any other non-time-based charts)?
1. sc.LatestDateTimeForLastBar.GetTime(); is the same as J41 in a spreadsheet study, and it updates intrabar, assuming there are trades.2. Can we do something to remedy it while continue using Bar Based Backtesting? 3. Would it be the same if I did a full Replay Backtest? So you could use this for intrabar times to flatten, in non-timebased bars. 2. The Bar Based Backtesting only works on close of bar signals, and not with multiple charts, and other limitations: Auto Trade System Back Testing: Bar Based Back Testing 3. IMO, a Replay Chart backtest is the only valid method. It offer the most flexibility and accuracy. The trade-off is it's very slow. I have developed and tested over 500+ strategies and they all have one thing in common….. the work great for first 2-3 months and then start to decay and results become unpredictable once the symbol changes or change in fundamentals or seasonality or for some unknown reason.
This is my experience also. Except the 'unknown reason' is the market is way too dynamic for an autotrader to be perpetually profitable.Therefore one must continually tweak it, which at best is only quasi-automatic. Here is matrix in Excel of Sierra Chart's backtesting choices and their features and limitations: https://www.sawtoothtrade.com/blog.html Date Time Of Last Edit: 2024-02-19 17:23:49
|
| [2024-02-19 07:12:09] |
| User61168 - Posts: 403 |
|
Thanks Tom. market is way too dynamic for an autotrader to be perpetually profitable.
Therefore one must continually tweak it, which at best is only quasi-automatic. I am just unable to accept failure in autotrading as futures is completely regulated, centralized, fully transparent market providing all market participants with the single global order book without any darkpools and a single universal data feed coming out of CME servers. Date Time Of Last Edit: 2024-02-19 07:12:36
|
| [2024-02-19 10:50:57] |
| User431178 - Posts: 541 |
|
64 bit diminished floating point imprecision, but did not eliminate it. I guess you are talking spreadsheets? |
| [2024-02-19 15:03:35] |
| Sawtooth - Posts: 4120 |
|
I guess you are talking spreadsheets?
Differentiating spreadsheets is irrelevant. Floating point imprecision can only be managed, not eliminated. https://en.wikipedia.org/wiki/Floating-point_error_mitigation |
| [2024-02-19 15:19:09] |
| User431178 - Posts: 541 |
|
Floating point imprecision can only be managed, not eliminated. OK... So maybe you could explain how floating point imprecision is relevant to date time variables that are stored as 64-bit integers? |
| [2024-02-19 17:09:39] |
| Sawtooth - Posts: 4120 |
|
So maybe you could explain how floating point imprecision is relevant to date time variables that are stored as 64-bit integers?
SerialDateTime values are not integers; they have decimal values.If they are stored as integers, then the cut-off occurs at some number of decimal places. As such, there is a limit to its precision, because at some number of decimal places the precision is lost. For example on a spreadsheet, when comparing the SerialDateTime time of 10:25:00 to its TIME function to the millisecond, precision is lost at the 11th decimal place. And there are only 17 decimal places of precision. See attached pic. This subject is esoteric and above my paygrade, and I don't have much more to add. I only know that dealing with floating point imprecision of SerialDateTime values, and very small tick sizes, is an on-going challenge. Date Time Of Last Edit: 2024-02-19 17:13:20
|
 Floating Point Imprecision.PNG / V - Attached On 2024-02-19 17:09:33 UTC - Size: 5.56 KB - 40 views |
| [2024-02-20 09:23:55] |
| gtaranti - Posts: 67 |
|
After debugging I realised that I was not using sc.BaseDataEndDateTime[sc.Index] correctly to get only the time. After fixing this, the flattening occurs in time (!). Also, I'm thankful for the enlighting comments of all participants, especially in the field of backtesting procedures/techniques. |
To post a message in this thread, you need to log in with your Sierra Chart account: