Support Board
Date/Time: Tue, 11 Mar 2025 02:29:05 +0000
Denali data feed for algorithmic trading?
View Count: 1153
| [2022-03-31 00:33:24] |
| User814333 - Posts: 13 |
|
I’m new to Sierra Chart and have a few questions about the Denali datafeed from the CME group. This is in regard to using it for algorithmic trading. 1. How often is live data from the exchange uploaded to the Sierra Chart data servers? Just wondering how long it takes for a new order on the exchange to appear in the SC datafeed. 2. Is the data cleaned in any way by the server (i.e. zero/negative prices removed, duplicate data removed, etc). I don’t know how much of this comes from the exchange vs the data providers, but just thought I’d ask. 3. How is data provided to the clients? For example, does SC use a publish/subscribe mechanism where clients subscribe to the symbols they want to follow? 4. Is there anything that would prevent the clients from storing the data they receive in a local data base? For this I’m assuming a custom ASCIL study would be needed, but I’m just wondering if it’s possible. The idea would be to have low latency access to recent data for algorithmic purposes. I’m just curious if the CME Denali feed would be suitable for low latency (tens of millisec level) algorithmic trading (not the ultralow HFT stuff, as that’s at another level). Thanks for any feedback on this. |
| [2022-03-31 14:01:58] |
| |
|
1. We do not know this precisely, but it should be within 1 millisecond. 2. There is no modification of the data and there is no need for this. 3. Yes: where clients subscribe to the symbols they want to follow?
4. Is there anything that would prevent the clients from storing the data they receive in a local data base? Yes potentially due to CME market data rules.Yes latency within the the range of 10 to 20 ms is reasonable: I’m just curious if the CME Denali feed would be suitable for low latency (tens of millisec level) algorithmic trading
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2022-03-31 14:02:35
|
| [2022-04-01 00:58:16] |
| User814333 - Posts: 13 |
|
...it should be within 1 millisecond.
Excellent! There is no modification of the data and there is no need for this.
So data from the exchange never has errors? Does the exchange check the data, or has technology just rendered this very unlikely? It's my understanding it use to be an issue. Yes potentially [unable to store data] due to CME market data rules.
Thanks for the heads up on that. I'll need to look into this, although I'd be surprised if it weren't allowed. Don't charts in SC that display "live" market activity store the chart data locally? |
| [2022-04-01 01:41:03] |
| 1+1=10 - Posts: 270 |
|
Hi User814333, I'm a trader that dabbles in algorithmic futures trading. Is the data cleaned in any way by the server (i.e. zero/negative prices removed, duplicate data removed, etc). I don’t know how much of this comes from the exchange vs the data providers, but just thought I’d ask.
In futures prices can be negative -- definitely keep that in mind in your algo. An example of this was March 2020 crude oil: https://www.cnn.com/2021/04/20/investing/oil-prices-negative/index.html Regarding errors or duplicate data in a real-time futures feed it strikes me as excessively unlikely. If you're worried the most comprehensive response would be from the CME's data team themselves. You can send your question here: https://www.cmegroup.com/market-data.html#contact 4. Is there anything that would prevent the clients from storing the data they receive in a local data base? For this I’m assuming a custom ASCIL study would be needed, but I’m just wondering if it’s possible. The idea would be to have low latency access to recent data for algorithmic purposes.
Don't charts in SC that display "live" market activity store the chart data locally?
Yes, SC stores the data in local files. Yes, from a technical perspective an ACSIL study, which allows custom C++, can write to a database. However, the CME charges significant amounts for data use & distribution. You can see the fee schedule here: https://www.cmegroup.com/market-data/distributor/files/mdla-cme-schedule-5-apr-2020.pdf For example, real-time data distribution, the first heading, for each CME Group Exchange (CME, CBOT, NYMEX, COMEX) is $24k annually, so $96k in total! If you wanted to play it by the book and know whether your use would be permitted, use the aforementioned CME Data contact form. Also, regarding the best tips for latency with SC I recommend you check out the 3rd post here and perhaps what follows if you're interested: Backtest results are radically different from live trading results |
| [2022-04-01 19:44:31] |
| User814333 - Posts: 13 |
|
Hi 1+1, thanks for the feedback. In futures prices can be negative...
Yes, I recall seeing see that, so eliminating negative values would be problematic in that case. I took your advice and contacted the CME to get their feedback (thanks for the link). I asked them if they had any statistics on how often errors occur in their data. If they respond, I'll let you know what they say. SC stores the data in local files. Yes, from a technical perspective an ACSIL study, which allows custom C++, can write to a database. However, the CME charges significant amounts for data use & distribution.
I was assuming I would use the non-professional Denali data service for this. Needless to say, it would be a non-starter if I had to subscribe directly with the CME at those rates. But in my inquiry to the CME Group, I also asked if they had any policies prohibiting local storage of the data from a provider. It will be interesting to see what they say. Thanks again for your help. I appreciate it. |
| [2022-04-01 20:23:18] |
| 1+1=10 - Posts: 270 |
|
@User814333, I would indeed be interested in the CME’s response to the error question. Oh, one thing I likely should have said is that before you worry too much about writing to a database make sure to check whether the default ACSIL ways of accessing historical market data are or are not already low-latency enough. Anyway, good luck with your trading! |
| [2022-04-25 19:32:27] |
| User814333 - Posts: 13 |
|
I would indeed be interested in the CME’s response to the error question.
I just wanted to report back that I still haven't received a response from the CME about this. Given that it's been several weeks, I'm not too optimistic I will get one. Could be they just don't have a statistic for that. I have one other question concerning the Denali data feed. When streaming live data, how are data interruptions handled by SC (due to, say, equipment failure or a cut line disrupting internet access)? When the connection is re-established, is all the missing data downloaded automatically and filled in on the charts? Thanks! |
| [2022-04-25 21:22:30] |
| John - SC Support - Posts: 38547 |
|
When the connection is re-established, is all the missing data downloaded automatically and filled in on the charts?
Yes, this is exactly what happens. It doesn't matter what the reason is for the missing data (bad Internet connection, system turned off for the night, etc.), as long as you are using the Denali Exchange Data Feed, if you have missing data, it will get filled in first before starting the real-time updates. For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing |
| [2022-04-25 22:00:16] |
| User814333 - Posts: 13 |
|
as long as you are using the Denali Exchange Data Feed, if you have missing data, it will get filled in first before starting the real-time updates
Thanks John. Good to know. I think I recall reading that data is saved in a data file in the SC folder. Is that correct? If so, is that where an ACSIL study would go to access the data for analysis? |
| [2022-04-25 22:14:08] |
| John - SC Support - Posts: 38547 |
|
I think I recall reading that data is saved in a data file in the SC folder. Is that correct?
Yes, this is correct. Intraday data is stored in files with the extension .scid, whereas daily data is stored in files with the extension .dly. These are located in the "Data" folder under your main Sierra Chart installation folder.If so, is that where an ACSIL study would go to access the data for analysis?
No, a custom study will get the data from internal memory of Sierra Chart. Basically Sierra Chart loads the data into an array for a chart and then ACSIL has access to that data array. Refer to the following:ACSIL Interface Members - Variables and Arrays: sc.BaseDataIn[][] / sc.BaseData[][] You could parse the .scid file if you want. Refer to the following: Intraday Data File Format For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing |
| [2022-04-25 22:39:40] |
| 1+1=10 - Posts: 270 |
|
@User814333, I’m sorry they didn’t respond. Thanks for reporting back! |
To post a message in this thread, you need to log in with your Sierra Chart account: