Support Board
Date/Time: Tue, 26 Nov 2024 04:46:02 +0000
[Locked] - Important Notice: Denali Data Feed Infrastructure Updates
View Count: 2446
| [2022-10-28 07:40:05] |
| Sierra_Chart Engineering - Posts: 17179 |
|
Over the last month, we have been working on a project to move servers we have with different providers, into a new co-location space in Equinox CH1 in Chicago at 350 Cermak. This utilizes high-performance network routers and switches. Entirely a 10 Gb network. Prior to the transition of the Denali Data Feed for CME data, into this serving space, we had organized a 10 Gb Internet connection, and a 10 Gb cross connect for CME data. These are replacing multiple single gigabit connections. These 10 gig connections, have been delayed. This is resulting, in lags and pauses occasionally when the market is busy for our Denali Data Feed CME users. We do apologize for this issue. Everything will be complete, by the end of this coming week. Thursday evening there were several changes implemented to mitigate the problem because we are able to serve data from multiple locations but we are somewhat limited on available bandwidth. When everything is complete, the Denali Data Feed will be running on a much faster higher performance network than we had previously. Entirely a 10 Gb network. And we will be able to scale up as necessary with no delays. We apologize for this issue and we are disappointed with the problem. The issue is that some of our service providers were not getting things done, for us on time. We do take the problem seriously, and we are implementing additional redundancy so there is not any possibility of having interruptions. Finally, we cannot guarantee, that there will not be some issues when the market is very active, between now and until these 10 Gb connections are are in place. We expect the problem be fully resolved, by November 4. The 10 Gb cross connect, will be established, over this weekend, Saturday October 29, and we think that itself is sufficient to resolve the issue. ---- One final note, we really have operated with Sierra Chart for all of this time, with very minimal use of network bandwidth. As we grow though, we are needing more. Although not that much more. It is remarkable, that we have operated so long, with good performance, with only gigabit connections. Although we have used 10 Gb connections, in the background for exchange connections. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2022-10-28 16:06:41
|
| [2022-11-12 19:31:24] |
| Sierra_Chart Engineering - Posts: 17179 |
|
We have had reports of additional issues, with the Denali Data Feed, lagging or pausing. We have confirmed this issue from one of our backup servers on Thursday 2022-11-10. That server was taken off-line. That server was used in order to distribute bandwidth usage among the available bandwidth we have. Since we took it off-line, we have not had any problems with bandwidth limitations among our other servers at least as far as we can tell. Certainly not from our own colocation space in CH1. And we also had a performance issue, related to the operating system, on one system which was causing the data to pause for up to 30 seconds and then a big burst of data would go out. This occurred on the date of November 2, 2022. That date only. After a careful investigation of the issue, we determined this was an operating system level problem. On that system we restarted the operating system, and we disabled Services that were not essential. What the exact cause was we do not know but it was clear it was definitely the fault of the operating system. This was an unusual problem that we have not seen before. And we took appropriate steps to solve it on that particular system. We do not like to talk about the background infrastructure too much, because inevitably users will look at what we say, and associate problem they have with this and think that that was the cause. It could be but in many instances it is not. We do acknowledge, there have been some issues lately with the Denali Data Feed when the market has been very active. This is one reason, we are moving away from our existing infrastructure provider to our own set up with with very high switching capacity, far far in excess of what is needed, a full 10 Gb architecture, and our own unified redundant Internet connectivity. Perhaps the problems you have been seeing lately are due our other infrastructure provider. Depending upon what server you are connected to you could be using the systems connected to that infrastructure. It just depends upon what server you are connected to. Since we do not have access to that networking equipment, we cannot confirm or deny whether the issue is there or not. We can tell you our infrastructure provider will strongly deny it. And in general we would agree with them. There still is one remaining item we have to complete with our hardware upgrades, which is an Internet upgrade in our new co-location space in CH1. This is literally several weeks behind. We are told it is going to happen this week. We have been very very disappointed at the delay and we do apologize for this. We have been very aggressive to get this completed as soon as possible. We just have been at the mercy of our service provider, related to this. Once this is completed, we will be serving all real-time data from one location and that is CH1. This is through the Equinix Connect product which uses two different carriers. We are told these are NTT and Lumen. It is a scalable product, we can purchase as much bandwidth as we need. And we have fully the ability to monitor exactly what is happening in real time and if we have a lot of user reports of a problem we can instantly see if there was any constraint on our side at that time. We are also working on performance updates with our real time server process. These performance updates are important, but we do not think, without these improvements, and the with the current architecture, there is a problem. The evidence of that is clear but when the market is extremely active, perhaps the data flow may not be as smooth. These software performance updates will be ready sometime in late November 2022 or early December 2022 we expect. So not too far off. It is also important for users to have high-performance routers which can handle a large amount of packets per second. You may be surprised to find out in some cases your router is not fast enough. You should check the specifications of it. ---- Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2022-11-12 23:07:35
|
| [2022-12-06 17:44:05] |
| Sierra_Chart Engineering - Posts: 17179 |
|
We had several reports today (2022-12-06) from users, of the Denali Data Feed pausing or lagging. This would relate to Internet connectivity only. Not with the core/background Denali Data Feed. We do not like to go into a lot of background details because inevitably sometimes people will associate problems they have had at other times to what we are describing, and think that was the issue, when the Internet connectivity issue was elsewhere. We also do not want to start blaming a service provider but at this point the blame is very well deserved. In this thread we talked about upgrading our Internet connections to 10 Gb connections. That was partially complete but not fully about three weeks ago. Although it was partially completed we had sufficient Internet bandwidth available. And the issue with limited bandwidth was effectively resolved. We also described a Internal cross connect that needed to be installed and that was completed also about three weeks ago without any issue. That was another essential thing that we required and that solved the problem associated with the lack of that Cross Connect. The problem with the 10 Gb internet connections that were provided to us, is that they were not configured as we required. Up until today we were operating with 1 10 Gb connection and 3 1 Gb connections. The properly configured 10 Gb connections were scheduled to be completed today or tomorrow but they were actually completed overnight. Our service provider told us they were going to be using the same IP block for these new 10 Gb connections, that the other 10 Gb connection which was misconfigured was using. So we took that 10 Gb connection off-line so that users would not experience an interruption when they reassigned the IP block. Since yesterday evening we have been operating with 3 1 Gb connections. One of these 1 Gb connections has reached capacity today at least one time. It may have reached capacity more times than we could tell because of microbursts which are not captured in the logging. This whole story of the Internet connection upgrade, is extremely frustrating for us and has had, numerous mistakes and massive incompetence on the part of the data center provider Equinix. Due to no fault of ours whatsoever, they have made, mistake after mistake after mistake and delay after delay after delay. They could not possibly have done any worse job than they have had for us. Complete total massive incompetence. Are we have also been warned about that, that Equinix or in particular Equinix Chicago is a disaster. Now that is an exaggeration of course, but others have had poor experience with them as well. However, CH1 is where we need to be because that is where all of the exchange connections exist. These connections should have been completed 2.5 months ago without issue, but it has been delays and misconfigurations all of this time. And what they delivered to us, overnight once again had a new misconfiguration if you can believe it. Disregard, to specifications, which were clearly in the order. Our extreme displeasure with this, has been communicated to them very forcefully. We expect they will correct the misconfiguration sometime today and these 10 Gb connections will become operational, tomorrow evening. The Internet connectivity provided by Equinix, is good quality. Once it is properly configured, it works just fine. These are fiber connections, connected into very high-performance routers/switches with capacity 10+ times beyond what is required. ---- Update: Above we described a misconfigured 10 Gb connection. That did not prevent us from using it. It consisted of a primary and a secondary connection and the misconfiguration prevented us from using the secondary connection. We are going to be able to bring one of the 10 Gb connections we have, back online, this evening. That will solve the problem. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2022-12-06 18:32:19
|
| [2022-12-06 18:46:25] |
| Sierra_Chart Engineering - Posts: 17179 |
|
And we also want to point out, we do have full backup Internet connections outside of CH1 to handle a full interruption with the Internet service but that is not expected since everything is fully redundant. 2 physical connections within the data center, and the data center uses 2 ISPs. We have two core routers with redundant power supplies fed by true redundant power in the data center. It is a fully redundant set up. Nevertheless we still maintain off-site backup connections. We have a total of 4 of those. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing |
| [2022-12-07 15:36:48] |
| Sierra_Chart Engineering - Posts: 17179 |
|
We had another incident this morning that affected some but not all users, where the Denali data froze. This indirectly related to the Internet connectivity issues we have described, in post 3 above. These Internet connectivity issues will be solved this evening when we finally get properly configured 10 Gb connections to replace the other connections and then we can fully utilize the 10 Gb connections. The problem from this morning, was that there were too many users on 2 out of the approximately 10 different server processes. The reason this relates to Internet connectivity is that due to configuration changes we made last night with Internet connectivity, there was uneven distribution of users. We also described in this thread performance improvements, with our real-time server program. We are changing the architecture of that to utilize more threads (the program is already multithreaded but one area there is a limitation), for the distribution of data to users. That we expect will be done at the end of December. If these architecture improvements were in place this morning, the problem would have been avoided. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2022-12-07 15:39:04
|
| [2022-12-08 01:53:54] |
| Sierra_Chart Engineering - Posts: 17179 |
|
We want to keep you up-to-date with what we did. We were in the data center today. We patched the new 10 Gb connections to our routers and they did assign the new IP blocks that we required for them. However, there was no light signal on either of them. We started a trouble ticket, and they said that these connections have not yet been turned on. At this point we do not know when they will be turned on but we hope it will be tomorrow. Or this week. Hopefully it is not going to be delayed until next week. We will push them to get this done as quickly as possible. It should not be difficult for them to get those operational, tomorrow, December 8. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing |
| [2022-12-08 14:35:58] |
| Sierra_Chart Engineering - Posts: 17179 |
|
Our analysis, about the issue yesterday affecting some users being a problem with too many users on a server process, we are fairly sure at this point is incorrect. We think it was an operating system level performance issue. Our real-time server program is very efficient and can handle a large number of users. We have not had an issue like this before. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2022-12-08 16:58:19
|
| [2022-12-08 16:59:50] |
| Sierra_Chart Engineering - Posts: 17179 |
|
We are quite sure, it was a system performance issue which caused the issue yesterday. We do run Windows and that particular system which had the issue did not have the best performance options for the processor, and the paging file and the allocation of processor resources to programs. We have corrected all of this now. The paging file will be disabled after the close today. We do not use paging files. We just keep large amounts of RAM on the system and disable paging files. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing |
| [2022-12-09 14:49:07] |
| Sierra_Chart Engineering - Posts: 17179 |
|
We want to keep you up-to-date with what we did. We were in the data center today. We patched the new 10 Gb connections to our routers and they did assign the new IP blocks that we required for them. However, there was no light signal on either of them.
We started a trouble ticket, and they said that these connections have not yet been turned on. At this point we do not know when they will be turned on but we hope it will be tomorrow. Or this week. Hopefully it is not going to be delayed until next week. Regarding this we were advised, by the NOC to swap the send and receive sides of the fiber cables connecting to the patch panel. We did that Thursday evening on December 8 and we have a light signal! It was too late for us to switch over to those Internet connections. However, we will do that at 4:30 PM US Eastern time this afternoon. This will not cause any interruption at the time. Well actually it can cause an interruption but users will just simply be reconnected to the next available server. It will not cause any interruption with order routing since that runs on different servers. When we first noticed the problem, we started a trouble ticket with Equinix and a technician came out to the cabinet in about 10 minutes or so, and met with us, and said that the Internet connections had not yet been turned on at the NOC. We were charged $130 (half hour labor rate of 260 USD per hour) for being told one sentence. What he told us was not true! They had been on, but we needed to reverse the transmit/receive sides. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2022-12-09 16:30:54
|
| [2022-12-10 19:57:20] |
| Sierra_Chart Engineering - Posts: 17179 |
|
We have successfully changed over to the new 10 Gb connections, Friday evening. The trading evaluator connections were not working Friday evening and this morning but they work now. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing |
| [2022-12-11 22:55:23] |
| Sierra_Chart Engineering - Posts: 17179 |
|
We now have a total of 4 Internet connections in the data center. Each of them going into a separate router.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2022-12-11 23:03:34
|
To post a message in this thread, you need to log in with your Sierra Chart account: