Support Board
Date/Time: Tue, 26 Nov 2024 11:34:06 +0000
DTC client implementation in C#-some pointers needed
View Count: 4918
| [2014-02-12 19:38:32] |
| User57650 - Posts: 50 |
|
Hello, I am trying to implement a DTC client in .Net that would receive market data from your Relay Server. It should be fairly easy to establish a socket connection to your Relay Server and start sending/receiving messages. In pseudo code, I would just use Socket.BeginReceive( byte[] buffer, int offset, int size, SocketFlags socketFlags, AsyncCallback callback, Object state ) and process received buffer data in a callback method. One thing I am not certain about is how to properly read in the received bytes. Will that byte array (buffer) be filled with bytes based on respective message types defined in DTC protocol? How will I know I am at the beginning of a message? Based on my understanding, each message structure has Size and Type at its beginning, so I could technically use that information to read off remaining bytes from the array. This would assume that whatever message I receive is GUARANTEED to have size/type at the start of it and that I will receive a complete message. I would then also have to make a conversion of the received bytes for each particular C++ variable type in the structure to a corresponding type in C#. Similarly, using the above logic I could send messages by populating a byte buffer with appropriate bytes that would comprise a given message structure in DTC protocol and then send it over the socket. Is this roughly how I should try to go about this? Thank you for assistance. |
| [2014-02-12 19:56:12] |
| |
|
Will that byte array (buffer) be filled with bytes based on respective message types defined in DTC protocol?
YesHow will I know I am at the beginning of a message? Based on my understanding, each message structure has Size and Type at its beginning, so I could technically use that information to read off remaining bytes from the array. This would assume that whatever message I receive is GUARANTEED to have size/type at the start of it and that I will receive a complete message. I would then also have to make a conversion of the received bytes for each particular C++ variable type in the structure to a corresponding type in C#. Similarly, using the above logic I could send messages by populating a byte buffer with appropriate bytes that would comprise a given message structure in DTC protocol and then send it over the socket. This is a correct understanding. We would recommend creating the equivalent C# data structures used by the relay server which are defined in the DTCHeader.h file.Is this roughly how I should try to go about this? Thank you for assistance. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2014-02-12 20:04:28] |
| User57650 - Posts: 50 |
|
Thank you for quick response. I will attempt that. Few more quick questions that would be great to know answer to are here: Now, let's assume I create equivalent C# data structures, what it they later change on your side? (i.e., if DTCHeader.h file changes some of their existing structures) Will I have to keep up with that all the time? Also, when creating equivalent data structures in C#, I am not sure how the following line affects that process: '// The byte ordering is little endian.' Thank you once again. |
| [2014-02-12 20:36:59] |
| |
|
DTC is relatively new, and there have been some changes. But it is reaching a point where it is becoming stable. The only significant change coming is going to be changing the quantity types to floats rather than integers. We are going to try to work on that next week. Other than that, the data structures should not be changing. There might possibly be new members added to a data structure, but not a change. It is not part of the protocol for changes to occur. The reason for this particular change is we want to be conformant with FIX which uses float types for quantities. '// The byte ordering is little endian.'
Almost all computers use little endian byte ordering so this is not going to be a concern. You can disregard it.One thing about reading data over a network socket is that you cannot always be assured of receiving a complete DTC message at once or even receiving the first two bytes of the message which indicates its size, at once. You may receive a partial structure or size and have to wait for the remaining data to arrive on the network socket. The wait time is not significant. For a local connection, this would be within the very low microsecond range. The point is, when you first are reading for the message size, you need to make sure you have at least two bytes of data in the incoming socket buffer before checking for the size of a message. And when you want to read a complete message/structure out, you need to make sure that you have enough bytes in the buffer before proceeding to process the message/structure. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2014-02-12 20:39:59
|
| [2014-02-12 20:40:16] |
| |
|
There were some small edits just now to the previous posting.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2014-02-12 20:52:56] |
| User57650 - Posts: 50 |
|
Thank you for the response. I am familiar with the possibility of the message not arriving all at once, but I think it's a very useful remark to everyone trying to implement this. I think now I have a good starting point to implement this. Will let you know if I have additional questions if they come up during development. Thank you for great support.
|
| [2014-02-12 21:05:12] |
| |
|
And for the record, this thread is about this feature: https://www.sierrachart.com/index.php?l=doc/doc_RelayServer.php Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2014-02-17 04:05:56] |
| User57650 - Posts: 50 |
|
I have progressed in writing a C# version of a number of DTC protocol messages (structs). Currently I am able to receive and process data from the Relay Server, both level1 (quotes+trade updates) and level2 (market depth). What interests me is the following: Currently, it seems that to receive level1 data (quotes+trade updates), one has to have a chart open for a particular instrument to receive data for that instrument via Relay Server. Similarly, one only receives market depth updates when instrument's DOM is open. Will there be a programmatic way to request subscriptions for a list of instruments without having to open charts/DOM for each instrument of interest? Seems like original DTC documentation supports it (via s_MarketDataRequest/s_MarketDepthRequest) but it's not (yet?) implemented for the Relay Server. I have implemented C# versions of MarketDataRequest/MarketDepthRequest and other related structures and tried sending them to the Relay Server but it doesn't seem to have any effect (I understand based on current documentation this is likely not available). Another question of interest is whether more than one application will be able to receive data from Relay Server. Based on my current understanding (allowing only one socket connection to one selected port), this is not (yet?) possible. Thank you for your responses. |
| [2014-02-17 04:32:48] |
| |
|
Currently, it seems that to receive level1 data (quotes+trade updates), one has to have a chart open for a particular instrument to receive data for that instrument via Relay Server. Similarly, one only receives market depth updates when instrument's DOM is open. And also we could support some kind of suffix code to the symbols to flag to maintain market depth data without using any of the market depth features. We will have to think about that. Will there be a programmatic way to request subscriptions for a list of instruments without having to open No.charts/DOM for each instrument of interest? Another question of interest is whether more than one application will be able to receive data from Relay Server. No.And one thing we also need to make clear about this feature, is we are cautious about it because it likely would make data providers we work with uncomfortable because it can lead to abuse of exchange data. So it is not likely we would do anything to expand upon the functionality. Originally it was designed for our own internal use. So in general, work with what you have and do not look for any further expansion of the functionality beyond what we said in this post. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2014-02-17 05:59:25
|
| [2014-02-17 06:08:13] |
| |
|
Do not be concerned about the last paragraph in the prior post in regards to the feature being withdrawn. It will not be. We just have to be careful with its implementation. This is all.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2014-02-20 14:11:17] |
| User57650 - Posts: 50 |
|
Thank you, Intraday File Update List is very useful for what I was asking. It would be really great if you could implement some sort of flag to maintain market depth data, then any application using Relay Server could get all the data needed without the user having to do much (other than specifying symbols inside Update List). I am surprised about your remarks regarding more than one application accessing data as all of this is restricted to local machine (just like IQFeed allows) but I have to admit I am not familiar with how this is perceived by exchanges. |
| [2014-02-25 14:34:42] |
| User57650 - Posts: 50 |
|
Hello, as I am testing Relay Server functionality using IQFeed and it gives all trades as they happen. I was able to determine that it also reports Non-last-qualified trades and unfortunately, there is now way to distinguish them from last-qualified trades. I understand that DTC protocol wants to remain as tight and as efficient as possible but I do think it would be great if trade updates had a field (maybe a single char/byte) that would flag what type of trade it is, or at least if it is a last-qualified trade or not. This is quite important for anyone who does serious volume analysis as these non-last qualified trades typically 'cloud' volume analysis as they originate from 'implied' trades (spread trades etc.) and don't necessarily mean anything in terms of existing bid/ask volume pressure. Please let me know what you think about that. Thank you. PS: just thought about this, if the above is not what you would like to do, maybe another simpler solution would simply be to not relay non-last qualified trades at all (not ideal but still better than having them and not knowing they are non-last qualified). Date Time Of Last Edit: 2014-02-25 14:44:02
|
| [2014-02-25 16:56:54] |
| |
|
DTC can certainly support a field indicating the type of trade. However, the particular DTC structures being used are compact and simple ones. This feature was meant for Sierra Chart's own internal use and we only have a need for the most compact simple DTC structures. What we will do is not relay the trades from IQ Feed which are not identified as normal trades. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2014-02-26 00:56:23
|
| [2014-02-25 17:37:07] |
| ejtrader - Posts: 688 |
|
Not trying to hijack this thread : SC Team - Is there any prototype source code for any working "DTC client" ( in c++ or c# ) available to explore further? Thanks |
| [2014-02-26 00:47:37] |
| User57650 - Posts: 50 |
|
Thank you for the answer regarding not relaying non-last-qualified trades. I assume you will not relay those trades via DTC for any underlying data feed, not just IQFeed. As far as ejtrader post. I plan on providing Sierra with the code I wrote for DTC client in C#-fully functional client in C# that can connect/reconnect to the relay server, receive and process market data (level1 and market depth). This should enable anyone doing development in C# to quickly use it/extend to to their needs as I have defined a C# equivalent of a number of C++ message structs, which can serve as template for further development for trading functionality etc. |
| [2014-02-26 00:49:50] |
| ejtrader - Posts: 688 |
|
User57650 - REALLY appreciate it !!
|
| [2014-02-26 00:57:09] |
| |
|
I assume you will not relay those trades via DTC for any underlying data feed, not just IQFeed. This only applies to IQ Feed.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2014-02-26 02:04:32] |
| User57650 - Posts: 50 |
|
Do other data feeds your platform supports also report non-last-qualified trades?
|
| [2014-02-26 02:37:53] |
| |
|
Not really sure. Possibly mytrack, TD Ameritrade and Barchart might have a trade indicator which indicates this. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2014-02-26 04:23:21] |
| User57650 - Posts: 50 |
|
I am now looking at market depth updates sent by the relay server and here is a sequence I recorded, with fields being: message type (118),MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT, message size, MarketDataSymbolID, Price, Side, Volume, UpdateType This sequence is for crude oil. Please look at it as there seem to be strange things happening: 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.87,AT_ASK,5,DEPTH_INSERT_UPDATE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.84,AT_ASK,3,DEPTH_INSERT_UPDATE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.85,AT_ASK,5,DEPTH_INSERT_UPDATE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.86,AT_ASK,9,DEPTH_INSERT_UPDATE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.86,AT_ASK,11,DEPTH_INSERT_UPDATE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.85,AT_ASK,4,DEPTH_INSERT_UPDATE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.83,AT_BID,1,DEPTH_DELETE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.82,AT_BID,5,DEPTH_INSERT_UPDATE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.82,AT_BID,5,DEPTH_DELETE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.81,AT_BID,59,DEPTH_INSERT_UPDATE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.81,AT_BID,59,DEPTH_DELETE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.8,AT_BID,75,DEPTH_INSERT_UPDATE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.8,AT_BID,75,DEPTH_DELETE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.79,AT_BID,11,DEPTH_INSERT_UPDATE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.79,AT_BID,11,DEPTH_DELETE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.78,AT_BID,6,DEPTH_INSERT_UPDATE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.78,AT_BID,6,DEPTH_DELETE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.77,AT_BID,9,DEPTH_INSERT_UPDATE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.77,AT_BID,9,DEPTH_DELETE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.76,AT_BID,8,DEPTH_INSERT_UPDATE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.76,AT_BID,8,DEPTH_DELETE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.75,AT_BID,10,DEPTH_INSERT_UPDATE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.75,AT_BID,10,DEPTH_DELETE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.74,AT_BID,9,DEPTH_INSERT_UPDATE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.74,AT_BID,9,DEPTH_DELETE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.73,AT_BID,5,DEPTH_INSERT_UPDATE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.79,AT_BID,10,DEPTH_INSERT_UPDATE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.78,AT_BID,5,DEPTH_INSERT_UPDATE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.84,AT_ASK,4,DEPTH_INSERT_UPDATE There is a set of updates above such as 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.82,AT_BID,5,DEPTH_INSERT_UPDATE 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.82,AT_BID,5,DEPTH_DELETE where an insert/update at a price is immediately followed by a delete of the same quantity at the same price. This seems to happen quite a bit-I recorded a few hours of this data on a number of instruments-and it's everywhere. When looking at order book display of the data this seems to create holes that get filled in from time to time and tend to reoccur. Clearly I must be missing something here or misinterpreting that a DEPTH_DELETE really means removing that price level from DOM. Please advise. |
| [2014-02-26 04:41:18] |
| |
|
The volume for these must always be 0: 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.74,AT_BID,9,DEPTH_DELETE This will be corrected. While it is not really considered a bug because it is harmless, it should be 0 because the size is meaningless on a delete. DEPTH_DELETE means actually to delete the price level. However, it does look like clearly a mistake that there is a DEPTH_INSERT_UPDATE immediately before some of them for the same price and size. That does not make sense. This will be resolved. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2014-02-26 04:54:28] |
| |
|
It could also be the problem is on the IQ Feed side as well. Do you notice this with other data feeds? For example try the Bitcoin Data (All) service and try one of the BTCUSD symbols. Although not the one from MTGox. They are out of business. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2014-02-26 04:55:46
|
| [2014-02-26 05:01:49] |
| User57650 - Posts: 50 |
|
I can't say how it is with other feeds as I don't have access to them through Sierra (at least I think I don't). I know however, having programmed DOM using IQFeed API directly, that it is not an issue of IQFeed. Then in fact sierra's DOM on IQFeed data would have to be incorrect, which is not the case as I compared your DOM (being connected to IQFeed from Sierra) to the one I have using IQFeed API and they match. |
| [2014-02-26 05:41:15] |
| |
|
You do have access to Bitcoin Data (All) in the list of Data/Trade Services in Global Settings >>Data/Trade Service Settings. Are you noticing a problem when you do handle DEPTH_DELETE as an actual delete of that price level from the DOM and then shift down any other levels above it. We are going to update the DTC documentation to explain this more clearly. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2014-02-26 05:41:40
|
| [2014-02-26 14:02:29] |
| User57650 - Posts: 50 |
|
"Are you noticing a problem when you do handle DEPTH_DELETE as an actual delete of that price level from the DOM and then shift down any other levels above it." Here is how I handle the following message: 118,MARKET_DEPTH_INCREMENTAL_UPDATE_COMPACT,20,1,101.83,AT_BID,1,DEPTH_DELETE In the above case, on receiving that message, I would remove 101.83 from DOM bids, all bids ABOVE it will have depth UNCHANGED and all bids BELOW 101.83 will have their depth shifted down by one. I assume DEPTH = 0 corresponds to the best (highest) bid and the higher the depth, the lower price in bids. please let me know if that's correct. Assuming it is, the messages received don't make sense as they in many cases add a level and immediately after that remove it, creating holes. |
To post a message in this thread, you need to log in with your Sierra Chart account: