Support Board
Date/Time: Wed, 27 Nov 2024 08:38:30 +0000
[Locked] - Denali feed slow during news
View Count: 1822
| [2023-09-01 17:39:40] |
| User61168 - Posts: 403 |
|
User61168: I've traded with various software, data feeds, and layouts for many years. Latency is super easy to recognize visually when looking at the order book (DOM).
Reading this and other responses, you guys are reporting "subjective" perception on lags which is not very helpful. Engineering needs the heart beat messages to troubleshoot this further so better to wait for the next news release and follow the instructions provided. And just to say it respectfully, anyone trading news release on CME from outside the USA should absolutely invest in a dedicated or bare metals VPS server hosted in Chicago. p.s. At work, my team frequently deals with similar end user issues with browser-based SAAS applications and unless we get detailed http header information from a local browser to our internal servers, troubleshooting becomes very challenging and a complete waste of time. 99% of such issues results in server hops over the internet (or even with WAN-connected corporate networks) BEFORE the request comes to our edge routers in our data center. I can certainly relate as to why SC would chose to lock such threads. Regarding my experience today, it was acceptable with one exception which seems unrelated to denali. I have a Exit Loss on Candle Close study which usually does not work in fast markets. It exits my trade after several candle closes @ 9:35:17am EST. I just write this off as slippage and cost of doing business. This also happens in market replay at high speed. attaching screenshot Edit: 9:30:35am ET candle close exit trade at loss worked beautifully with no slippage. Date Time Of Last Edit: 2023-09-01 17:56:18
|
 Screenshot 2023-09-01 103852.png / V - Attached On 2023-09-01 17:39:58 UTC - Size: 4.84 KB - 113 views |
| [2023-09-01 17:56:30] |
| Rui S - Posts: 190 |
|
Reading this and other responses, you guys are reporting "subjective" perception on lags which is not very helpful. Engineering needs the heart beat messages to troubleshoot this further so better to wait for the next news release and follow the instructions provided.
And just to say it respectfully, anyone trading news release on CME from outside the USA should absolutely invest in a dedicated or bare metals VPS server hosted in Chicago. p.s. At work, my team frequently deals with similar end user issues with browser-based SAAS applications and unless we get detailed http header information from a local browser to our internal servers, troubleshooting becomes very challenging and a complete waste of time. 99% of such issues results in server hops over the internet (or even with WAN-connected corporate networks) BEFORE the request comes to our edge routers in our data center. I can certainly relate as to why SC would chose to lock such threads. Regarding my experience today, it was acceptable with one exception which seems unrelated to denali. I have a Exit Loss on Candle Close study which usually does not work in fast markets. It exits my trade after several candle closes. I just write this off as slippage and cost of doing business. This also happens in market replay at high speed. attaching screenshot Thank you for your input. I have two different data feeds from two different providers, both located in the US. There is one of those data feeds that often fails while the other one rarely has a "hiccup". See the footage have I attached above and give me your conclusion, please. Thank you. |
| [2023-09-01 17:59:19] |
| User753428 - Posts: 163 |
|
for what it's worth, i had chart update interval set to 20ms and i did not experience any freezing today, unlike the other two days. Reading this and other responses, you guys are reporting "subjective" perception on lags which is not very helpful.
if you were following this thread, you would know that one of the two incidences (last Friday) was around a 40s-60s data freeze. that's way too long for it to be a "subjective" perception. Engineering needs the heart beat messages to troubleshoot this further so better to wait for the next news release and follow the instructions provided.
during the 40s-60s freeze, the message log did not print anything. trust me, i was intensely checking the heartbeat messages during that exact moment and it totally skipped over. see my comment on this thread (Glitch in feed at 10am Est). it is post #6. And just to say it respectfully, anyone trading news release on CME from outside the USA should absolutely invest in a dedicated or bare metals VPS server hosted in Chicago.
respectfully, i don't think you understand the problem. ppl aren't complaining about not receiving the data at the "correct time." if there's a news release at 8:30:00am EST, nobody's gonna complain about receiving the tick data at 8:30:50am EST due to latency if they're located in Tokyo. the problem is the manner in which the data is being transmitted and displayed through the front-end sc program. if i'm located in europe or new zealand, there will be a ton of latency so the ticks may come through later than when it actually occurred. but as long as the ticks are displayed in the correctly-timed sequence as it occurred at the exchange-level (albeit with slight latency), ppl would be satisfied. 99% of us aren't running front-running algos that are latency-sensitive. Date Time Of Last Edit: 2023-09-01 18:00:45
|
| [2023-09-01 18:12:06] |
| joshtrader - Posts: 488 |
|
Did a side by side comparison. Today it was better. However, I do not think we had the volume surge I've seen before. The Rithmic feed is much snappier, and updates more frequently. As the end of the video shows, both ladders are set with a 15ms update interval. I always use this because with a 60Hz refresh rate on my displays, anything lower would not be perceivable on my displays (as a 15ms refresh updates theoretically 66 times per second). The NFP in the beginning is pretty well sync'd, but if you look at 33:00, the ISM report which came out 10 seconds later shows a clear latency in the Denali feed (on the right side). Rithmic on left, Denali on right. The video is only 30fps but you can still observe the lag in the Denali feed. Note I recorded this in 4k so be sure to set the YT quality to 2160p for the best view. Link starting at 33:00: https://youtu.be/yC6fVfGmkyw?t=1980 |
| [2023-09-01 18:13:38] |
| user_xyz - Posts: 416 |
|
SC - please don't lock this thread. SC Users - Those experiencing this problem please report back with data, logs, heartbeat messages, etc real data that can be tracked, not just 'me too' comments and pointless replies. They take up SC time and more likely to lock. I have experienced this issue many times and been involved in almost all these threads. I would like to focus efforts on resolution here, not across many more threads. I mentioned earlier I would report back post NFP unfortunately I am traveling and will report back on next major release. Date Time Of Last Edit: 2023-09-01 19:12:18
|
| [2023-09-01 18:15:02] |
| Rui S - Posts: 190 |
|
Are you sure 9 AM Eastern time?
My apologies, I meant to write CST. We need to see the heartbeat messages
Unfortunately I didn't have the Heartbeat logging activated, but I can check which servers I was connected to. Meanwhile, as I disconnected that instance and just reconnected now to retrieve the log, I cannot see the previous log anymore. How can I get it? Thank you. |
| [2023-09-01 18:28:36] |
| Rui S - Posts: 190 |
|
Hello joshtrader, Thank you for the video. Did a side by side comparison.
Could you please do it next time with NQ as well? Thank you. |
| [2023-09-01 18:29:36] |
| User753428 - Posts: 163 |
|
SC - please don't lock this thread.
SC Users - Those experiencing this problem please report back with data, logs, heartbeat messages, etc real data that can be tracked, not just me too comments. I have experienced this issue many times and been involved in almost all these threads. I would like focus efforts on resolution here, not across many more threads. i'd like to echoe this sentiment. this has been going on for so long it is time we all pool together our resources into a single thread to help Sierrachart Engineering find a resolution to this. that being said, many times these data freezes aren't reflected on the heartbeat log so i'm not sure what more data we can provide in these cases besides screen-captured videos. so please, Sierrachart Engineering, please don't delete/censor/lock this thread. as frustrated as we are, we're all doing this because we want to see your trading platform succeed. one tip i'd like to suggest to the engineering team is, next time there is a major news release, have two charts side-by-side: one with denali feed, one with Rithmic/CQG feed. set the remote buffer delay and chart update interval to the lowest setting. you will see for yourself that the Rithmic/CQG feed is snappier. |
| [2023-09-01 18:39:30] |
| Rui S - Posts: 190 |
|
respectfully, i don't think you understand the problem. ppl aren't complaining about not receiving the data at the "correct time." if there's a news release at 8:30:00am EST, nobody's gonna complain about receiving the tick data at 8:30:50am EST due to latency if they're located in Tokyo. the problem is the manner in which the data is being transmitted and displayed through the front-end sc program.
if i'm located in europe or new zealand, there will be a ton of latency so the ticks may come through later than when it actually occurred. but as long as the ticks are displayed in the correctly-timed sequence as it occurred at the exchange-level (albeit with slight latency), ppl would be satisfied. 99% of us aren't running front-running algos that are latency-sensitive Very well explained, thank you. I live in Portugal, Europe, and I have been scalping for years. Do I have a disadvantage because of that? Sure. Can I adjust my trading to overcome that latency (65ms from my house in Lisbon, 130ms from my house in Algarve - ping test)? Of course I can. I just need a reliable and fast data feed! |
Attachment Deleted. Ping Test.jpg / V - Attached On 2023-09-01 18:38:38 UTC - Size: 129.19 KB - 118 views |
| [2023-09-01 18:46:59] |
| Rui S - Posts: 190 |
|
as frustrated as we are, we're all doing this because we want to see your trading platform succeed.
Absolutely!! one tip i'd like to suggest to the engineering team is, next time there is a major news release, have two charts side-by-side: one with denali feed, one with Rithmic/CQG feed. set the remote buffer delay and chart update interval to the lowest setting. you will see for yourself that the Rithmic/CQG feed is snappier.
That would be very good. But snappier or not, just make it rock solid under heavy volume and volatility, please! Date Time Of Last Edit: 2023-09-01 18:50:41
|
| [2023-09-02 10:43:36] |
| Sierra_Chart Engineering - Posts: 17191 |
|
This thread is now locked and kwookoh is now banned from this board. Despite the claims, of users trying to help, you are only creating more burden on us, by posting irrelevant and out-of-date information which has no relevancy whatsoever to the current context. Specifically we are referring to kwookoh. You are creating more burden on us and misleading other users. Ultimately, this is wasting our time and taking away our time, from development and providing effective support to users. It wastes our time because we have to take the time to read through it and explain the situation and explain why it is not relevant. It is actually you who has the misunderstandings. Not us. We will briefly explain why what was posted is not relevant. You posted a thread from three years ago. That particular incident, which lasted a week or two, was the result in network I/O performance improvements, and a different way in which buffering was handled. These network I/O performance improvements did made a major difference in I/O performance for the better but the way in which we handled outgoing data buffering, which was based on an assumption, significantly offset those performance improvements. We then made a further minor change to buffer data, in a structured way, at a single common point before data transmission to the network adapter which then resolved the problem. This change, in combination with a network I/O performance improvements brought a major improvement, to network I/O performance. We also added LZ4 compression which is very fast and reduces bandwidth usage significantly. These changes brought huge fundamental differences with Sierra Chart data feeds for the better. The problems reported before these changes, simply are irrelevant. They have no relevancy to the current context whatsoever. It is not helpful to us to be doing what this kwookoh is doing and ultimately he is being abusive towards Sierra Chart. Not helpful. And that is why we made the decision to ban this user off the board. Despite the illusion, Rui S you are the only one who reported a problem represented by your video, at that time. We have no other reports of the issue but that is not relevant to us since you are reporting, significant issues, repeatedly. We will help you isolate the problem. And we have been contemplating this and we are going to put out a new release, with some diagnostic tools to help us further with this. Allow a few days for that. We are still quite sure, that there is a some other problem, either a local performance issue, or some network communication issue, specific to you, which is leading to the problem. Rui S, clearly, you are having problems which go well beyond, what other users are experiencing. And therefore, when there is an incident, which further investigation, Which may have subsequently be determined to be on our side, then you jump onto that thread and incident and then it associate your much more extensive issues at numerous other times, with that problem. This association is incorrect. We also want to briefly comment on this: Did a side by side comparison. Today it was better. However, I do not think we had the volume surge I've seen before.
The Rithmic feed is much snappier, and updates more frequently. As the end of the video shows, both ladders are set with a 15ms update interval. I always use this because with a 60Hz refresh rate on my displays, anything lower would not be perceivable on my displays (as a 15ms refresh updates theoretically 66 times per second). The NFP in the beginning is pretty well sync'd, but if you look at 33:00, the ISM report which came out 10 seconds later shows a clear latency in the Denali feed (on the right side). What you failed to provide is of how you have this setting set: Sierra Chart Server Settings: Remote Buffer Delay Send Time In Milliseconds (Global Settings >> Sierra Chart Server Settings >> General >> General) We do not know. That makes a huge difference, in the update speed of the Denali Data Feed. One reason we will lock a thread is when there is unnecessary or misleading information being posted, and when this then creates a burden being placed on us, which is no help to anyone. We are being placed under more of a burden by having to read through and respond to all of this and it is helping no one. This week we are going to put out a new version to allow a user to specify both the server address and port to use. This will allow us to have a user connect to a particular server process. We will then increase the heart beat frequency of that process, to every five seconds instead of every 20 seconds. And we will also connect to that same process from a remote location and compare with what we see to what the user sees. We are also going to add two additional fields of information to heartbeat messages. The peak send buffer size, and the time it occurred. We also want to comment on regarding something that kwookoh said about heartbeat messages. When there is a lack of heartbeat messages due to communication being stopped from the server, that information is very helpful to us! And then when these messages return, and then we look at the content of those messages they have important information which is helpful! The Denali Data Feed and other Sierra Chart data feeds are very different today, as compared to the past. We also have a lot more users. Here are some changes which have occurred over the years: -The use of overlapped Network I/O and I/O completion ports for significantly higher I/O performance. - Multiple high-performance Dell R640 servers configured for maximum performance. - 10 Gb networking with 10 Gb Internet connections. These 10 Gb Internet connections do have burst caps and we are increasing these burst caps. We think this was part of, but not completely, the recent issues during extreme market activity but was not a problem, on Friday 2023-09-02. The burst caps were not even are being approached anywhere close. In all cases, network activity was 50 to 70% away from the burst bandwidth cap. These Internet connections can sustain bandwidth, at that burst cap indefinitely. - Data distribution from a single common location, in Chicago CH1. This brings about complete consistency and low latency for Denali Data Feed distribution. -LZ4 compression with compression performed on multiple threads. - Multiple data distribution threads, which have increased throughput of our real-time servers by at least three times and more depending upon the number of users. This was a recent improvement. There may be more things but this is all that comes to mind right now. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2023-09-02 11:10:54
|
To post a message in this thread, you need to log in with your Sierra Chart account: