Support Board
Date/Time: Sun, 24 Nov 2024 06:36:36 +0000
Backtest results are radically different from live trading results
View Count: 3974
| [2022-02-23 23:03:38] |
| opmetal - Posts: 70 |
|
Hi, My ACSIL strategy on 10 tick bars performs radically different in backtests than it does in real time. I have all the recommended settings: Global Settings >> Data/Trade Service Settings >> Number of Stored Time and Sales Records: 5000 Global Settings >> General Settings >> Chart Update Interval: 500 Home >> (Table of Contents) Managing Symbol Data >> Tick by Tick Data Configuration Intraday Data Storage Time Unit: 1 Tick Allow Support for Sierra Chart Data Feeds: Yes Today's live trading result was a disaster, posting a large loss. When I backtest it for today in Accurate Trading System Back Test Mode for a Single Chart I get excellent profitability because of fantasy fills that didn't happen in real time. What can be the reason for this? At this point it seems that backtesting results are completely divorced from reality. Date Time Of Last Edit: 2022-02-23 23:04:19
|
| [2022-02-24 18:31:54] |
| opmetal - Posts: 70 |
|
Today I have my strategy running live and in sim, side by side. The differences are astounding. I'm watching sim LIMIT orders get filled on touch. I'm seeing price improvement on limit orders. My sim profitability is 3-5x greater than live orders. Is there a way to make sim orders reflect reality? |
| [2022-02-24 20:39:41] |
| 1+1=10 - Posts: 270 |
|
Use the Simulated Futures Trading service where your orders actually get sent to a server close to the exchange with realistic fills based on Estimated Position in the Queue: Simulated Futures Trading Service If you don't understand what "Estimated Position in Queue" is talking about you need to research Limit Order Books, for example: https://www.5minutefinance.org/concepts/the-limit-order-book Lastly, you definitely should not set the chart update interval to 500 ms. In general, you want the time between when your algorithm receives data and responds with an order to be as short as possible. The factors involved include: Receive Market Data From Exhange: 1. +0 millis -- Trade / limit order update takes place at exchange 2. +1 millis -- Exchange actually sends aforementioned update to your market data provider's servers. (NOTE: while typically this happens in < 1 millisecond Rithmic support staff said this delay can be as much as 15 milliseconds during very high volume news events.) 3. +0.250 millis - ~10 millis -- Data provider receives market data update. The duration here is simply related to where the data provider's servers are. If they're in the same building as the exchange, like SC's Teton or Rithmic then it is near the low end. If the data provider's servers are outside the building, like CQG's retail web api that SC uses, then you can get near the higher end. 4. +??? millis -- Time to send data from provider to your algo? NOTE: SC has two aspects which contribute to this duration. First, is the "SC Remote Buffer Delay". In short, SC's own data feed, Denali, only sends updates every 10 millis -- link below. Secondly, the global/chart "update interval" dictates how frequently your ACSIL algo runs (provided you have sc.AlwaysUpdate = true) -- the minimum setting is 10 millis. Thus, the combination of these two factors delays data getting to your algo by 20 millis in the worst-case and 10 millis in the average-case. 4's Standard-case :: +30.0 millis - 270.0 millis -- ACSIL trading algo on home computer. Factoring in SC's aforementioned delays, and assuming your home computer is outside of the exchange's metro area between 10 - 250 millis away gives you this range for step 5. 4's Improved-case :: +20.0 millis - 25.0 millis -- your ACSIL trading algo is on a server in the same metro as the exchange. For instance, if trading on the CME search "Chicago trading/vps/dedicated server". 4's Better-case :: +1 millis - 5 millis -- here we switch to only using SC for research/idea generation. For our algo we switch to Rithmic's API ($100 month) with the trading server as close as possible to the exchange. For the 1 millisecond case we'd need to be in the same building. For CME see https://www.theomne.net/ OR https://www.beeksfinancialcloud.com/catalogue/DS-EMERALD_5/. 4's Best-case :: +0.250 millis -- Here our algo is actually on the data provider's servers; typically $400 - $1000. I can provide links if you'd like. ------------------------ Send Order to Exchange 5. ACSIL Algo Processing Time -- this is likely under 100 microseconds. 6. Algo's computer/server -> Data Provider/Order Routing's Computer -- 0.250 - 250 millis: NOTE: this is the same as step 4 above although you have to add 1 milli for the risk check unless on 4's best-case. See Rithmic link below for explanation of risk checks. 7. Data Provider/Order Routing -> Exchange -- 0.250 - 10 millis: NOTE: same as step 3 above. 8. Exchange Matching Time -> 0.5 - 5 millis --------------------------------- To conclude, if you use SC and you get a server in Chicago you can get your worst-case round trip (new data to order) time down to ~30 millis. If you get to some of the more esoteric choices I've mentioned you can get down to ~1-5 millis. Let me know if you have questions. ---------------------------------------- Rithmic support explains why there can be delay between when a trade/limit order update occurs at the exchange and when the exchange actually sends the notification of the update to the data providers: https://community.optimusfutures.com/t/how-fast-are-the-various-rithmic-configuration-options/2342 Prices / Data Falling Behind: 4.12 - Sierra Chart Exchange Data Feeds |
| [2022-02-24 21:34:39] |
| opmetal - Posts: 70 |
|
Thank you 1+1=10 for the detailed response! Lots of great info and I will try the Simulated Futures Trading Service, which I didn't know could be useful here. My (Global) chart update interval was 500ms because that is what SC recommends. What should I set it to on the backtest chart? 20ms? Here are the other changes I made following your last link: Global Settings >> Sierra Chart Server Settings: Remote Buffer Delay Send Time in Milliseconds 100 (was 0) Use Real-time Data Compression: High Compression (was Standard Compression) |
| [2022-02-24 22:29:30] |
| 1+1=10 - Posts: 270 |
|
If you’re trading a short-term automated strategy, the problem with any of SC’s backtesting methods is that the time it takes for the data to get to your algo’s computer and for your order to go back to the exchange are not factored in at all! In that time of up to 500 ms, in the worst-case, the live price could have changed by dozens of ticks. To gain some insight in to this simply open a “Time and Sales” on the contract you’re trading; eventually disconnect and look at the time stamps. The only way to overcome this problem, unfortunately, is right export a chart’s data and build a backtesting system outside of SC. If you do, you can factor in how long it takes your algo to react to new information, which you absolutely must know of any sort of short-term trading. You can write an ACSIL study to submit 1 live limit order far away, that won’t get executed, and then wait for the order acknowledgment —- this will more or less be your algo’s reaction time. A much more accurate method in SC for determining the profitability of an ACSIL algo is the Simulated Futures Trading Service as since the data and orders actually go back and forth from the service’s server which is in Chicago it factors in algo reaction time (including limit order queues). The disadvantage is it is only for “forward-testing”, meaning it only provides the ability to trade on “live” data, weather real-time or delayed. The remote buffer delay and chart update interval are part of determine your algo’s reaction time during live trading and while using the Simulated Futures Trading Service. You want them both at their minimum values, 10 (millis.) High compression is a good idea. SC warns about setting some of the values too low but if you’re serious about algo trading you need a computer/server setup capable of processing the data as quickly as possible (which can handle the minimum settings). |
| [2022-02-24 22:33:43] |
| opmetal - Posts: 70 |
|
I appreciate the help, thank you.
|
| [2022-02-24 23:14:24] |
| 1+1=10 - Posts: 270 |
|
Certainly! Adding one last bit of context, according to the CFTC in 2019 fully 50%-90% of trades were automated — the number is likely even slightly higher now. See page 7’s “Exhibit 2: Share Of Automated Futures & Options Transactions”: https://www.cftc.gov/sites/default/files/2019-03/automatedordersreport032719.pdf This is why it is so important to lower your algo’s reaction time. If everyone else can react to market data every 2 millis and it takes yours 100 millis you can’t compete on short-term strategies. Good luck with your trading! |
| [2022-02-24 23:18:36] |
| opmetal - Posts: 70 |
|
That's a fascinating stat. Thanks again for your help!
|
| [2022-03-16 23:45:34] |
| User379468 - Posts: 508 |
|
SC's Denali feed seems to be limited to 20ms snapshots with the Remote buffer delay setting, which will increase the delay mentioned above.
|
| [2022-03-17 00:03:49] |
| 1+1=10 - Posts: 270 |
|
SC's Denali feed seems to be limited to 20ms snapshots with the Remote buffer delay setting, which will increase the delay mentioned above.
I should have actually linked to the SC docs discussing the Remote Buffer Delay setting: Prices / Data Falling Behind: 4.12 - Sierra Chart Exchange Data Feeds According to that you can set it at 10ms (rather than 20ms). So the worst case is 10ms Remote Buffer + 10 ms Chart Update == 20 ms. Of course, on average the data feed delay is precisely half of the worst case delay. Perhaps that idea is intuitive but I'll explain just in case. The best case would be that a market data update arrives at the Denali server from the CME a microsecond before Denali sends a 10 ms snapshot, and also that the snapshot arrives at the SC platform a microsecond before a chart update interval. In that case the SC portion of the delay for that particular market data update is effectively 0 ms. The worst case is what we're describing above where a market data update arrives at the Denali server a microsecond after the last 10 ms snapshot was sent, so it has to wait till the next one, and the snapshot arrives at the SC platform a microsecond after the last chart update interval, so it has to wait till the next one. Then the combined delay is 20 ms. Thus, on average ticks will have a delay in between 0ms - 20ms which equals 10 ms. |
| [2022-03-17 21:27:12] |
| User379468 - Posts: 508 |
|
I should have actually linked to the SC docs discussing the Remote Buffer Delay setting: 4.12 - Sierra Chart Exchange Data Feeds
According to that you can set it at 10ms (rather than 20m In actual testing, setting it to 10ms does not change the delay below 20ms for me (including after disconnect/reconnect), transmission delay remains at 20ms in the log. |
| [2022-03-17 22:00:57] |
| 1+1=10 - Posts: 270 |
|
Oh wow. I’ll refer to your insight then if this subject ever comes up. Thanks for testing!
|
| [2022-03-19 13:00:56] |
| Luigi - Posts: 18 |
|
Have you guys experimented with other feeds than denali regarding measurable delays? When I am using denali feed (NQ futures contract 0.25 range chart) I can see with my eyes that chart stops for milliseconds. First I thought it is the setup as I am running SC on linux or the settings. But then I tested with every possible settings and on windows as well, and around faster markets I can see the same tiny delays in chart refresh when using denali. Now I did experiments with IQ feed and CQG feed with same setup, settings , etc. and IQ feed and CQG are smooth, no visible delays. So my conclusion is that it is likely on the SC denali server side causing the observable delays. Any idea how to measure this delay what I see on denali feed and not on the others? I cannot run the other feeds parallel. (I'd prefer to use denali but not until this issue is there as If I see this ,then the algo is also getting data delays too) |
| [2022-03-19 19:56:03] |
| 1+1=10 - Posts: 270 |
|
Hi Luigi, Have you guys experimented with other feeds than denali regarding measurable delays?
I personally haven't. Although I was attempting to find a short-term edge a few months ago I actually moved to a long term swing trading approach on futures spreads using seasonality, the law of one price, commitment of traders and new reports. Perhaps the other commentators have? SC is great for quite a few things: 1. idea generation 2. backtesting 3. getting historical data 4. live trading with latencies above 30 ms 5. a wonderful api to help in all the previous tasks But, SC's engineers were not targeting the extremely low latency automated trading use-case. If you want to go as low latency as possible with a round-trip time around 1.5-2 millis!, the most cost-effective solution is the following two steps: 1. Use Rithmic API ($20-$100/m) for data feed and order routing: --- https://edgeclear.com/rithmic/ && https://optimusfutures.com/Rithmic-API.php 2. Run your algorithm from Rithmic's VPS ($99/m) located in the same server rack! as their data feed/order routing server (located in the same building as the CME's exchange servers -- the CME's Aurora, Illinois building: --- https://www.theomne.net/virtual-private-servers/ Regarding speed of Rithmic see: 1. 3rd party trader explaining he gets 1-2 millis with Rithmic -- he was actually asking about going even faster!: 2. Rithmic support explaining their 1.5 milli latency: https://community.optimusfutures.com/t/how-fast-are-the-various-rithmic-configuration-options/2342 Want to go faster? Say less than <0.3ms: 1. Trading Technologies: $400/m + per contract fee w/ $2,000 cap -- https://www.tradingtechnologies.com/resources/pricing/ 2. Rithmic's Diamond API: $1000/m -- https://www.ampclearing.com/rdiamondapi.html 3. CQG has a product which is like $1200/m but I can't find the link. Hopefully this helps. |
| [2022-03-19 20:23:06] |
| 1+1=10 - Posts: 270 |
|
Opps! Here’s the 3rd party Rithmic link: https://futures.io/traders-hideout/43510-fastest-cme-retail-trading-api.html
|
| [2022-03-20 13:03:16] |
| Luigi - Posts: 18 |
|
Thanks for the summary and links! I don't even dream about playing the low latency game. Although I use fast charts, my entries are limit orders and the entry algo is perfectly fine with current latency. On the other hand, I don't want additional delays which are avoidable. Bottlenecks which can be eliminated, like SC denali feed tiny, but visible throttling. I need to measure it somehow so I can ask SC support about it ... |
| [2022-03-20 13:37:30] |
| 1+1=10 - Posts: 270 |
|
Hi Luigi, In the s_TimeAndSales structure which are the elements of the s_TimeAndSalesArray returned by https://www.sierrachart.com/index.php?page=doc/ACSIL_Members_Functions.html#scGetTimeAndSales, there is a DateTime field. The s_TimeAndSales member are below: struct s_TimeAndSales { SCDateTimeMS DateTime;// UTC float Price; uint32_t Volume; float Bid; float Ask; uint32_t BidSize; uint32_t AskSize; //This will always be a value >= 1. It is unlikely to wrap around, but it could. It will never be 0. uint32_t Sequence; int16_t UnbundledTradeIndicator; int16_t Type; uint32_t TotalBidDepth; uint32_t TotalAskDepth; ... // Various methods } Perhaps SC stamps the records with the time their Denali data feed server receives each market data update, rather than the time the market data update arrives at your computer? If it is the former then you could just compare the current time in your study against each update's time stamp to measure the delay. I'd recommend starting a new thread to ask SC Engineering whether the Denali time and sales records are timestamped on the Denali server or when they reach the SC platform. |
| [2022-03-20 13:41:47] |
| 1+1=10 - Posts: 270 |
|
UPDATE: According to this thread, SC Engineering said the timestamp originates with the remote data source, so I think you can do the comparison: CME Group and other Data Timestamp vs Timezone The thread was not specifically asking about Denali but I believe it was implied -- of course, if you want to double-check that makes sense too. Date Time Of Last Edit: 2022-03-20 13:42:09
|
| [2022-03-20 14:38:00] |
| Luigi - Posts: 18 |
|
Thanks again 1+1=10 :-) ! I will test it next week or so and if result is meaningful I will start a new thread. So I assume the source T&S is almost continuous in active periods, and I will look for gaps in the update arrivals, which I already see in the chart updates with denali feed vs CQG/IQfeed. Tx. |
| [2022-06-05 21:30:37] |
| User887126 - Posts: 66 |
|
Hi All, Can you give me some VPS provider info for CME colocated server? My broker is quoting me about $1k/month (seems to be in line what is mentioned here) for colocated CME server. i wanted to test it out to see if it is even worth it. This is literally 10 times more expensive than just using other VPS in the chicago area (still only about 1-2ms latency). I wanted to test out both the automated trading and may be simple profit taking built in function from SC. Thank you for any advise. I am currently using sierra chart tetons. |
| [2022-06-05 21:53:43] |
| Luigi - Posts: 18 |
|
https://www.beeksfinancialcloud.com/catalogue/DS-EMERALD_5/ 321 GBP, dedicated linux server CME aurora If your datafeed also SC Denali, then go for the Chicago area cheap VPS 1-2 ms . You will be limited by SC feed anyway -best ~20ms Date Time Of Last Edit: 2022-06-05 21:55:46
|
| [2022-06-05 22:23:02] |
| User887126 - Posts: 66 |
|
Thank you. I do have denali feed. i will probably just try cheap VPS for now and develop my strategy. Does SC run faster on window vs linux? i am just curious. Saving is about 16$/month compare to window server for beeks VPS. |
| [2022-06-05 23:26:36] |
| Luigi - Posts: 18 |
|
I found them about the same, maybe linux a tiny bit faster for my setup. But i would go with the one you feel more comfortable to manage remotely. (on the side note: I am not too happy with SC data feed as besides the 10ms server buffering and 10ms chart update i see occasional 100+ ms lags which ends up about 30ms average chart updates at CME open first few min. But I dont need low latency.) |
| [2022-11-10 23:12:37] |
| User379468 - Posts: 508 |
|
I am not too happy with SC data feed as besides the 10ms server buffering and 10ms chart update i see occasional 100+ ms lags which ends up about 30ms average chart updates at CME open first few min.
What have you found to be better? Also is it as reliable? |
To post a message in this thread, you need to log in with your Sierra Chart account: