Support Board

Hello,

Yesterday at 4 PM US Eastern time we had an interruption with our Teton order routing server due to a hardware memory error. This is an extremely rare condition which has not previously occurred. This condition was recovered from 15 minutes later.

The state data for order routing is saved every two minutes to files. Any order fills that occurred within two minutes before this incident would have led to those fills getting reprocessed upon reconnection to the iLink. This would lead to incorrect position quantities.

Please check all of the current positions for users of the Teton order routing because we do not receive an end of day position file from you. You can find these in "Trade >> Trade Positions Window" within your installation of Sierra Chart used for risk management.

If you need to make an adjustment you can perform a manual fill following these instructions:
Trade Account and Risk Management: Add Manual/Correcting Order Fill

We have made a change to save the critical state information every five seconds.

Great care has been taken to ensure that order fills do not get reprocessed. There is also a backup safety check to a not allow fills to alter the position quantity if they are older than 30 minutes.

Due to the fact that we recovered on the same server and the state information was being saved every two minutes previously, is the reason that this potentially could have occurred. As we said we have reduced this down to 5 seconds. The decision to use two minutes was in order to maintain the highest performance and lowest latency at all times for order routing, and the fact that an incident like this, has never occurred with any of our dedicated servers in the last 15 years.

If we were not able to recover, on the same server, we did have a backup available with a real-time copy of all order and other trading information state information and could have failed over to that but it was not necessary. That backup server is refreshed every 3 to 5 seconds.

Yesterday morning we also had installed a second server in the Aurora data center colocated with the CME and we are going to be using that as a primary server and moving half of the users over to that server. This was already planned, and in progress before this incident. So in a rare event like this, fewer users would be affected.

We also last week, performed a failover to our backup server using a CME test iLink to verify failover procedures. This transition was successful. We do have in place all backup procedures. And we went through additional testing overnight.

We are going to be replacing one or more of the memory modules in the server that was affected, over the weekend. We do not expect any further incidents this week until the module is replaced.