Support Board
Date/Time: Fri, 18 Apr 2025 06:47:26 +0000
[Locked] - US Equities/CFE Data Early Morning Weekday Carrier Maintenance Notification
View Count: 4732
| [2025-01-13 23:52:55] |
| Sierra_Chart Engineering - Posts: 19259 |
|
Update: Our partner carrier has advised a technician is on site. Additional testing is underway to confirm the location of a suspected fiber cut. Additional updates will be provided at 8:00 PM ET or upon the availability of new information.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2025-01-14 00:59:20
|
| [2025-01-14 01:05:51] |
| Sierra_Chart Engineering - Posts: 19259 |
|
Update: Our partner carrier has confirmed a repair crew is on site and have identified the damage to their cable. Crews are currently working to shift available slack to replace the damaged section of the cable. Additional updates will be provided at 9:30 PM ET or upon the availability of new information. We also pushed our other carrier, to provide us two additional circuits. We are not just getting one but two additional circuits. All of this has been in active discussion with them and we just got the final details we needed today after insisting, that we cannot continue waiting. One of the locations we wanted a backup circuit to go to they could not do in the short term so we are adding another one over to Chicago CH1 and another one over to NY4 for a total of 35 Gb which we can upgrade later as needed. 35 Gb for the time being should be more than enough, being we still have this existing 100 Gb circuit but obviously it's not reliable but still even if we do not use the 100 Gb, 35 Gb should be more than sufficient. If not, we will upgrade it to 50 Gb (Split between two different, data centers, geographically diverse). We just do not want to change an existing order that we already have in place and has been just pending an upgrade which we are going to hold off on because we don't want to cause any further delays. At this point we are not making any excuses. We are very sorry for what has happened. There still is more work ahead to get the circuits in place since they involve two different service providers and coordination between them. And one of them, is known to be slower than they they will say but we will push this along. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2025-01-14 02:00:54
|
| [2025-01-14 02:01:29] |
| Sierra_Chart Engineering - Posts: 19259 |
|
The service has now been restored. We are in the process of patching the missing historical data.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing |
| [2025-02-11 01:13:00] |
| Sierra_Chart Engineering - Posts: 19259 |
|
We want to put out this notice (quoted below), of carrier maintenance, which will be affecting the CFE data feed this Monday evening / early Tuesday morning (February 10, 2025). The other data feeds like US Equities, and the CBOE Indices will not be affected because during this time range they are not active. CBOE indices is active during this time range but not for US markets. Our 10 Gb backup connection to NY4, has been provisioned by our new carrier (Cogent). But we are still waiting for port provisioning, at the NY4 data center by our service provider who provides the data feeds. We hoped it would have been done by now and it should have been done by now but it is not. But it should just be days away and once we have that connection which is a 10 Gb connection, then there will no longer be any further interruptions. And we apologize for this entire matter. We have learned a lot about these carrier services with this experience. And we have also learned really to not trust anyone. Since this delay with getting this additional circuit was because we had put trust, in an alternative solution recommended by our service provider which ended up becoming complex and costly, which we ended up learning months later after it was originally recommended. We have also, developed automatic failover, between our primary and backup circuit. The failover will occur within 50 seconds. But even this is too much. Our objective is to provide 100% uptime and we will not accept anything less than that. But during this transition time, to our improved circuits, there is the potential still for a 50 second interruption. There is a solution to this, but does require a significant amount of more hardware infrastructure. Which we are gradually working towards. We are also adding a new 100 Gb Wavelength connection, between our data center location in Michigan, and the Digital Realty Franklin Park Illinois data center.: https://go2.digitalrealty.com/rs/087-YZJ-646/images/DS_Digital_Realty_2310_ORD12_ORD13_ORD14_factsheet.pdf This notice is being sent to notify you of the following planned maintenance event on the Crown Castle Fiber network.
Ticket Number: CM20250114041 Location of Work: Chicago, IL Scheduled Start Date & Time Scheduled End Date & Time Time Zone 2/11/2025 12:00 AM 2/11/2025 4:00 AM Eastern 2/10/2025 11:00 PM 2/11/2025 3:00 AM Central 2/10/2025 10:00 PM 2/11/2025 2:00 AM Mountain 2/10/2025 9:00 PM 2/11/2025 1:00 AM Pacific 2/11/2025 5:00 AM 2/11/2025 9:00 AM GMT Expected Customer Impact: Service Affecting Expected Impact Duration: Up to 2 hours Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2025-02-11 08:25:28
|
| [2025-02-11 08:23:03] |
| Sierra_Chart Engineering - Posts: 19259 |
|
The above maintenance was completed in about 35 minutes. The CFE feed was interrupted during that time.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing |
| [2025-03-12 21:51:24] |
| Sierra_Chart Engineering - Posts: 19259 |
|
We have sad news to report regarding this: Our 10 Gb backup connection to NY4, has been provisioned by our new carrier (Cogent). But we are still waiting for port provisioning, at the NY4 data center by our service provider who provides the data feeds.
We hoped it would have been done by now and it should have been done by now but it is not. But it should just be days away and once we have that connection which is a 10 Gb connection, then there will no longer be any further interruptions. And we apologize for this entire matter. We have learned a lot about these carrier services with this experience. And we have also learned really to not trust anyone. Since this delay with getting this additional circuit was because we had put trust, in an alternative solution recommended by our service provider which ended up becoming complex and costly, which we ended up learning months later after it was originally recommended. We have also, developed automatic failover, between our primary and backup circuit. The failover will occur within 50 seconds. But even this is too much. Our objective is to provide 100% uptime and we will not accept anything less than that. But during this transition time, to our improved circuits, there is the potential still for a 50 second interruption. There is a solution to this, but does require a significant amount of more hardware infrastructure. Which we are gradually working towards. After so much effort, and also expense, this has been a significant failure. The circuit we have from this fiber carrier has severe packet loss on it. We cannot 100% say the issue is with this particular new circuit itself, but it seems very likely. It does not make sense the issue would be anywhere else. This is a different type of circuit than a wavelength circuit. It is a switched circuit. Nevertheless it should not have packet loss but it does. It is effectively not usable by us. If there is a complete outage we will still be able to deliver data but with very significant packet loss probably around 5 to 10%, perhaps worse. All of the failover functionality has been developed. And released. Now the good news is that this particular 10 Gb circuit is only temporary. We have two new 100 Gb wavelength circuits on order, which are going to be delivered in June and July 2025 respectively. Today there were 2 periods of degradation with our existing 100 Gb wavelength circuit. Not sure anyone would have noticed but with our logging of packet loss we can tell that occurred. And also this interrupted the NASDAQ TotalView data feed for about 1.5 minutes these two different times. There is no missing TotalView data but there is repeating data in the charts for -NQTV symbols and we will take care of that. We will post here when that is done. The reason for repeating data has to do with the behavior of the NASDAQ TotalView TCP data feed. The NASDAQ TotalView connection uses a TCP connection. We have the ability to use UDP multicast and we are going to change to that this evening. This type of connection will not have this significant of an interruption. If the interruption in the circuit is 5 seconds you would only see a 5 second interruption with UDP multicast. Not going on a minute or two. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2025-03-13 04:49:20
|
| [2025-03-13 04:50:20] |
| Sierra_Chart Engineering - Posts: 19259 |
|
There is no missing TotalView data but there is repeating data in the charts for -NQTV symbols and we will take care of that. We will post here when that is done. Follow the instructions here to re-download the data where you see repeating data in the chart: Historical Intraday Data (Missing Data / Download Errors / Viewing More Data): Retrying the Download When There is Data in the Chart We have now switched to the UDP multicast feed for NASDAQ TotalView. And we will be working with the fiber carrier, and the service provider in NY4, regarding the packet loss issue. But it is very discouraging and will take time to solve. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2025-03-13 04:51:08
|
| [2025-03-20 15:17:19] |
| Sierra_Chart Engineering - Posts: 19259 |
|
Our new 10 Gb connection NY4 is working well now. The packet loss we were seeing, is not happening. At least for now. NASDAQ TotalView is running over this new connection. So we are no longer going to be affected by these early-morning carrier maintenance activities. And we do have much better circuits on order coming soon. We mentioned that we would be using the NASDAQ TotalView UDP connection but we switched back to the NASDAQ TotalView TCP connection for the primary feed to avoid any lost packets. This is unrelated, but we are also doing server and connectivity upgrades in the Aurora data center for CME data. This will take about 60 days to complete. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2025-03-20 15:18:55
|
| [2025-03-27 04:23:25] |
| Sierra_Chart Engineering - Posts: 19259 |
|
The packet loss issue with our 10 Gb connection to NY4 is definitely resolved. It is a circuit from Cogent, and Cogent reported this evening to us: After further investigation, we have determined that the packet loss you experienced coincided with another customer undergoing an attack. During this time, it appears that the device you are connected to was being overwhelmed. The Cogent backbone team intervened and successfully implemented a blackhole route to mitigate the attack, which helped restore normal traffic levels. Please let us know if you have any further questions or need additional information. We are not seeing any further packet loss. Our new replacement 100 Gb wavelength, over to CH2, is going to be delivered sometime in June. Our 100 Gb existing wavelength is going to be affected again by maintenance during March 27 and the morning of March 28. The carrier says the interruption will be for two hours. We have no idea when that will be, but with our 10 Gb connection to NY4 there is not going to be an interruption that would last more than 50 seconds. And we have moved or we are going to move all of NASDAQ (Just for March 27), NASDAQ TotalView, CBOE futures, and CBOE global indexes to NY4 as the primary circuit. With failover to Chicago. Contemplating what Cogent said, it actually confirms that the use of wavelength circuits is much safer. Since this uses a dedicated optical wavelength frequency for our own use at 100 Gb and it can never be interfered with. The problem with the wavelength is that it is not automatically re-routable. In the case of a physical fiber cut, the circuit goes down. This is why then back it up with a 10 Gb point-to-point layer 2 connection which we have to NY4. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2025-03-27 08:33:59
|
To post a message in this thread, you need to log in with your Sierra Chart account: