Support Board
Date/Time: Thu, 28 Nov 2024 05:54:48 +0000
[Locked] - Technical Notice: US Consolidated Tape Packet Loss Issue
View Count: 1026
| [2023-05-07 16:10:10] |
| Sierra_Chart Engineering - Posts: 17199 |
|
This posting is about packet loss. Sierra Chart obtains, the NASDAQ, NYSE and AMEX data feeds, through services called UTP and CTA. These are controlled by NASDAQ and NYSE respectively. These services provide us what are called multicast data feeds. These are UDP data feeds. It is possible to lose packets somewhere in the process of the transmission of these feeds, or the processing of these feeds. This results, in missing data which will result in lower volumes that you will see for the symbols on these data feeds. The new consolidated tape is data feed for NYSE and AMEX was released May 1, 2023. We had no packet loss, for this data feed on that day. And the NASDAQ data was released in the weeks earlier and we only had very minimal packet loss with that feed. Minimal packet loss can be recovered through the use of the retransmission feed which we are currently working on. Since May 2, 2023 we have been experiencing significant packet loss with NYSE and AMEX. This ranges anywhere from .2% to 2%. The particular amount of loss, depends upon the particular symbol, and the particular multicast group that symbol is on. It is hard to isolate the source of a problem like this, but with our experience with packet loss, and having tried numerous things to resolve this, and with careful consideration and consultation with a service provider, giving us access to these feeds, it has become clear that the issue, most likely is microbursts of data transmission, which are exceeding the network connection capacity that we currently have. This was something that we thought would not be a problem because we were not seeing it with the NASDAQ feeds, and we did not see it May 1 and we have had no trouble with the equivalent data feeds being provided by Barchart which are also multicast feeds and providing data from NASDAQ, NYSE and AMEX. The difference though with the feeds from Barchart as compared direct from the exchanges, is that from the exchanges, they are providing us low latency feeds, from network switches, which have a higher capacity than 10 Gb. The connections we use are 10 Gb. We are going to have to upgrade our connections to 40 Gb. That may not happen this week. We have to rely upon a service provider for this. We hope it will be completed this week but if not this coming week, it will be completed the following week ending May 19. We are doing our best to get this done as quickly as possible. All the necessary hardware to support this, we already possess. Until we get this issue resolved we are not going to be offering delayed data for the US Consolidated Tape. --- Another thing to understand, that has confused some Sierra Chart users is the new real time US Consolidated Tape data feed, is different than the NASDAQ TotalView data feed. NASDAQ TotalView is documented here: Denali Exchange Data Feed: Nasdaq TotalView Data Feed Notes If you are using the NASDAQ TotalView symbols like: AAPL-NQTV You will see lower volumes. Subscribing to the US Equities Consolidated Tape data feed does not change this. You need to use, one of these two symbols to see the full volume: AAPL AAPL-USCT Use one symbol or the other but not both. Using both will lead to a conflict where you will only get data for one symbol. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2023-05-08 08:11:29
|
| [2023-05-08 18:39:20] |
| Sierra_Chart Engineering - Posts: 17199 |
|
The 40 Gb connections will be installed next week. They cannot be done this week. After a technical discussion, it has been made clear that this will solve the problem. We are pulling too much data over 10 Gb connections and experiencing microbursts that are exceeding the connection speed.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing |
| [2023-05-10 07:58:00] |
| Sierra_Chart Engineering - Posts: 17199 |
|
We want to give an update regarding the degree of the packet loss. For NYSE and AMEX the packet loss ranges anywhere from .1% to .7%. We are not seeing higher percentages at this time. For NASDAQ it is .01% to 0.02%. This will vary by symbol. In the case of the multicast groups containing the symbol SPY, the loss is down to .1% If there is any significant loss, we have been reloading the historical data on our servers containing the historical data for the new US Equities Consolidated Tape from the data maintained by our Barchart Openfeeds, in order to maintain historical data integrity. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing |
| [2023-05-15 13:52:18] |
| Sierra_Chart Engineering - Posts: 17199 |
|
For Friday ( 2023-05-12) the packet loss ranged on average from .03 to .7% depending upon the symbol. Most of the packet loss occurs with NYSE and AMEX. Even with the packet loss, we are seeing higher volumes, with our new US equities Consolidated Tape feed as compared to the prior data feed we had from Barchart Open feed. Although part of that may have something to do with, filtering of trades based upon their sale condition. But it is not related to the inclusion of odd lot trades on the new feed. Since those are excluded in our comparisons. We are waiting for the 40 Gb upgrade this week. We already did preparation for it over the weekend which went well. Part of that upgrade, included 25 Gb connections to our servers. Each server, will have 2 25 Gb connections. We also had the manufacturer of our network switches run a test to ensure that there will be no packet loss, when receiving 40 Gb of sustained multicast/UDP market data over the main trunk/line, and distributed to multiple other ports on the switch. They did the test and there was 0% loss. We were happy they did that. They used this type of equipment: https://www.keysight.com/us/en/products/network-test/network-test-hardware/xgs12-chassis-platform.html Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2023-05-15 13:54:06
|
| [2023-05-23 13:24:30] |
| Sierra_Chart Engineering - Posts: 17199 |
|
We want to update this thread. The 40 Gb connections have not yet been provisioned. We are just waiting on our service provider for this. Provisioning of a connection of this capacity/speed takes time to provision since these are not commonly utilized ports. Originally they told us they did not have additional ports. But then they were able to locate them. We expect to have these operational in about 1.5 weeks from now. In the meantime we have been able to minimize the packet loss from .02 to .5%. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2023-05-23 13:24:53
|
To post a message in this thread, you need to log in with your Sierra Chart account: