Support Board
Date/Time: Sat, 23 Nov 2024 12:35:33 +0000
Post From: Data Feed Stopping/Lagging Issue (non-Denali specific)
| [2020-09-19 14:25:38] |
| Sierra_Chart Engineering - Posts: 17145 |
|
Regarding compressing real-time data from Sierra Chart data feeds, we think we could potentially have this working and available this coming week assuming there are no technical complications. We will at least begin testing, this week. Probably even as soon as this weekend. We do not foresee any difficulties with compressing real-time streaming data using LZ4 because this is a high-performance algorithm. Compression will be done on dedicated threads and we do not expect this to introduce any perceivable latency. Our testing shows that we can compress about 1000 bytes in a microsecond (Effectively 1 GB per second throughput). So as you can see there is no practical latency introduced with compression. Regarding the issue that some users see with stopping data or lagging data, based on feedback, this issue is resolved for most users. We have been doing monitoring of our data processing and real-time server programs and processes to ensure there are no problems with those. And there are no issues with delays or overload with those processes. They work very efficiently and are stable. We also observe no problem at the operating system level on the servers which could cause lagging data. We also observe no problem with the local network switching equipment. Our servers are very high-performance, and operate nearly all of the time with 90+% excess capacity. We have always maintained that the issue is network connectivity related for affected users and that is specifically what it is. We are not saying we are faulting specific users Internet connections, although in some cases this is exactly the reason for the problem. Instead we are saying if you have a problem this is network connectivity related. The problem can be anywhere along the path of communication between your computer and the server. This path of communication, is two ways. You have to consider the return path for packet acknowledgments. At the same time we also say that there are things that we can do to help. What we can do to help would be increasing network capacity at the server side, increasing receive side buffer sizes and dynamically increasing them as necessary, reducing the number of IP packets (using larger packets), and using compression. We have contacted our infrastructure provider for the Denali data feed about the issue that some users see and they have assured us with their network monitoring, that we are not exceeding any thresholds that they see which could lead to this problem. Nevertheless we have increased the available assured bandwidth that we have to us in the Aurora data center by five times (very costly by the way). And we have increased the number of users on that server to utilize that bandwidth and also use that server for any relaying. Our infrastructure provider through their own monitoring of their network infrastructure says there are no issues with packet loss or buffering of data. But they always say at the same time they make no assurance as to what happens over the public Internet. They refer to the public Internet as the "public toilet". We are also going to be setting up a relay on the West Coast of the US and once we have compression in place we will set up a relay in Germany. We have one in London. With data compression these relays should work quite well. Finally we want to reiterate, that the data compression is going to reduce latency and be a major improvement with Sierra Chart provided data feeds. The data to be transmitted will be reduced by about 70%. This data reduction is going to be a definite solution to lagging or stopping data for anyone affected by this. We are going to try some different modes of compression and we should be able to achieve a higher compression. Maybe 90%. So if we can compress the data by 90%, and have this out in less than a few microseconds, this is a dramatic improvement! This is going to put the Sierra Chart data feeds including the Denali Exchange Data Feed and the Sierra Chart Exchange Data Feed in a class, unmatched by anyone. This will be a definitive resolution to any latency issues. We will then be able to remove the "Low Bandwidth" option we have in Sierra Chart. Since there will no longer be a need for it. And then we should also theoretically be able to transmit market by order data without any practical issue. But we do not expect that to come until later. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2020-09-19 15:40:42
|