Support Board
Support Board
Search Board
Control Panel
View My Posts / Threads
Direct MessagesDate/Time: Thu, 25 Apr 2024 06:46:03 +0000
[DTC] Subscribing to market data correctly
View Count: 5742
| [2019-03-19 10:58:53] |
| taffer - Posts: 3 |
|
My problem is that the steps in the specification (https://dtcprotocol.org/index.php?page=doc/DTCMessageDocumentation.php#MarketData) simply do not work as described. Though I managed to find out some workaround, I am still unable to control the SymbolID sent by the server in various messages. Issue 1: MARKET_DATA_REQUEST handling • By default the server does not respond to MARKET_DATA_REQUEST at all! Not even a reject. I have to open an intraday chart in the Sierra GUI first for the selected symbol. • If I opened an intraday chart for the symbol at least once, the server responds to my subscription request; however, the SymbolID in the response is an arbitrarily chosen value and does not equal to the one I sent in my request! Even worse, this ID can vary for a symbol if I shut down and restart Sierra. After a while I get a SECURITY_DEFINITION_RESPONSE from which I can sync the IDs as its RequestID seems to be the same as the SymbolID in the market data messages. This is not described so in the specs, though. Issue 2: Security Definition handling • If I already opened intraday charts for some symbols when I connect to Sierra, I will get the SECURITY_DEFINITION_RESPONSE messages automatically (regardless whether I checked Auto Send Security Definition For New Symbols in the GUI). Now I use this for initializing symbols in advance. • However, if I want to subscribe to a new symbol I bump into the first issue. I tried to overcome this by sending a SECURITY_DEFINITION_FOR_SYMBOL_REQUEST first but the server simply ignores this request! Again, if I open an intraday chart and then I send a MARKET_DATA_REQUEST, then I will start to get market data with a random SymbolID, and after a while I will get also the SECURITY_DEFINITION_RESPONSE with an arbitrary RequestID, which is different from the ones I used in the SECURITY_DEFINITION_FOR_SYMBOL_REQUEST and MARKET_DATA_REQUEST messages but after all I can sync the ID. The market data messages recieved between my MARKET_DATA_REQUEST and the lately received SECURITY_DEFINITION_RESPONSE will be either lost (if I don't recognize the ID) or misinterpreted (if I associated the ID with another symbol). Questions: 1. Can I get market data purely by DTC, without the need of clicking in the Sierra GUI? 2. Is it possible to get the same SymbolID I sent in the request? I would prefer a reject message in case of conflict rather than no response at all. 3. If the answer for question 2. is no, then would it be possible to negotiate the SymbolID in advance? Eg. by getting a response for a security definition request as it is expected based on the specs? Again, without clicking in the GUI I will not get any SECURITY_DEFINITION_RESPONSE even if I request it; I get it automatically later containing an arbitrary ID. And please, could you update the documentation accordingly? Date Time Of Last Edit: 2019-03-19 11:04:02
|
| [2019-03-19 11:07:17] |
| |
|
You must be doing something wrong on your side. The documentation is 100% correct. Let's just take this one item at a time. It does not help to describe all of this. Clearly you are doing something wrong with a very basic item. It sounds like you have been following this documentation here: Relay Server Do not use the relay server mode. First, did you log on, and get a logon response? Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2019-03-19 11:08:22
|
| [2019-03-19 11:17:47] |
| taffer - Posts: 3 |
|
OK it seems I use relay server mode. I enabled DTC in Sierra by the Enable DTC Protocol Server and I connect to localhost:11099. Is there maybe a direct address I can connect to? And here are the relevant lines from the logs of the login from my client: 2019-03-19T12:12:31.8512911+01:00 [DEBUG]: DtcSocket connected: localhost:11099 2019-03-19T12:12:31.8542832+01:00 [DEBUG]: Outgoing DTC Message: Type=EncodingRequest|ProtocolVersion=8|Encoding=BinaryEncoding|ProtocolType=DCT 2019-03-19T12:12:31.8722355+01:00 [DEBUG]: Incoming DTC Message: Type=EncodingResponse|ProtocolVersion=8|Encoding=BinaryEncoding|ProtocolType=DTC 2019-03-19T12:12:31.8742305+01:00 [DEBUG]: Outgoing DTC Message: Type=LogonRequest|ProtocolVersion=8|Username=***|Password=***|GeneralTextData=<null>|Integer_1=<null>|Integer_2=<null>|HeartbeatIntervalInSeconds=30|TradeMode=Simulated|TradeAccount=<null>|HardwareIdentifier=<null>|ClientName=<null> 2019-03-19T12:12:31.8871948+01:00 [DEBUG]: Incoming DTC Message: Type=LogonResponse|ProtocolVersion=8|Result=Success|ResultText=Connected to SC DTC Protocol server. Service=cqgfix|SymbolSettings=cqg|ReconnectAddress=<null>|Integer_1=0|ServerName=SC DTC Server|MarketDepthUpdatesBestBidAndAsk=False|TradingIsSupported=False|OCOOrdersSupported=False|SymbolExchangeDelimiter=<null>|SecurityDefinitionsSupported=False|HistoricalPriceDataSupported=True|ResubscribeWhenMarketDataFeedAvailable=False|MarketDepthIsSupported=False|OneHistoricalPriceDataRequestPerConnection=True|BracketOrdersSupported=False|UseIntegerPriceOrderMessages=False|UsesMultiplePositionsPerSymbolAndTradeAccount=False|MarketDataSupported=False 2019-03-19T12:12:31.8891895+01:00 [INFO]: Logon completed. Connected to SC DTC Protocol server. Service=cqgfix|SymbolSettings=cqg 2019-03-19T12:12:31.8951737+01:00 [DEBUG]: Incoming DTC Message: Type=MarketDataFeedStatus|Status=Available Though MarketDataSupported is False in the logon response, immediately after I get a MarketDataFeedStatus with Status=Available. And market data is working after all but only with the workarounds I described above. If you need some further logs of specific tests please let me know. Date Time Of Last Edit: 2019-03-19 11:29:02
|
| [2019-03-19 12:07:39] |
| taffer - Posts: 3 |
|
I actually followed this description: Using DTC Server for Data and Trading in Another Sierra Chart Instance but this mentions only DTC connection by enabling DTC Protocol Server. Now I asked my customer for whom I develop the DTC client whether he got a host:port address for his account but he didn't. Sorry for asking but is there a page with the available direct DTC servers, then? I was unable to find them. |
| [2019-03-19 13:26:03] |
| |
|
There must be something wrong with your setting of the structure members you are sending and the interpretation of the structure members for received messages. You have to examine your code closely and see what the issue is. If you see a message in the Sierra Chart message log that says this upon sending your logon message, then relay server mode is being used: Using relay server mode.
If you are using the Sierra Chart DTC server on the same computer as the DTC client, then you just would use 127.0.0.1 for the address. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2019-03-19 13:27:31
|
| [2019-03-19 14:17:00] |
| User406106 - Posts: 6 |
|
Yes, "Using relay server mode" is in the log. How can I connect directly, then? Or in non-relay mode? > There must be something wrong with your setting of the structure members you are sending and the interpretation of the structure members for received messages. Since I use binary encoding I notice if I get garbage due to wrong data sizes/misaligned fields. But that would not explain that everything works if I open and intraday chart before subscribing to a symbol. Or is there a way to talk someone eg. via Skype? I believe we could solve the issue in 2 minutes instead of this long discussion. |
| [2019-03-19 16:20:20] |
| User406106 - Posts: 6 |
|
I managed to solve the issue. I think, it is worth to share the solution and the lessons learned. The hint was actually in the relay-setup steps you sent: > Send a LOGON_REQUEST message with the LOGON_REQUEST::Integer_1 field set to 2. The issue is again that the specification is incorrect. According to the documentation (Authentication and Connection Monitoring Messages: LOGON_REQUEST [s_LogonRequest structure] Client >> Server) the LOGON_REQUEST is an optional field And here is the handling of unset fields with binary serialization: DTC Messages and Procedures: Indication of Unset Message Fields This section tells how a binary serializer should treat unset fields: > In the case of an integer, this will be INT_MAX. But if I set this value to 0 (or I don't treat it as optional), then I can connect in non-relay mode. ________________________________________ > The documentation is 100% correct. And sorry but I have to mention this is not correct at all. Or at least I can point out some room for improvement. Just some examples I found so far: • SymbolID in almost every MARKET_DATA* message is still described as [unsigned int16] types in the documentation. Actually they are int32 or uint32 almost everywhere. • I found several messages, which contain additional fields compared to the documentation. I could peek them from the C++ headers, though. • It would be really helpful if the documentation contained the data size of the enum fields. The C++ headers may help though. • What I really miss is an accurate binary structure layout of the fields. I struggled a lot of time with implementing a correct binary serializer even with the help of the C++ headers. It seems that the binary encoding actually reflects the in-memory layout of a specific C++ compiler, an architecture-dependent field layout, which may cause "gaps" between the fields. The following example helps to highlight the issue (note the AlignmentFields among the actual fields): // This is how I describe MarketDepthUpdateLevelCompact message. // Please note the gaps I applied to reproduce the in-memory representation of the C++ version. // These information cannot be retrieved from the documentation private static readonly DtcField[] fields = { new PrimitiveField<uint>(nameof(SymbolID)), new PrimitiveField<AtBidOrAsk>(nameof(Side)), // 2 bytes enum new AlignmentField(2), // 2 bytes gap! new DecimalField<float>(nameof(Price)), new DecimalField<float>(nameof(Quantity)), new PrimitiveField<MarketDepthUpdateType>(nameof(UpdateType)), // 1 byte enum new AlignmentField(3), // 3 bytes gap (or MarketDepthUpdateType is int32 enum but C++ headers indicate as byte enum) new DateTimeField(nameof(DateTime), DtcDateTime.DateTimeWithMilliseconds, true), // 8 byte double new PrimitiveField<uint?>(nameof(NumOrders)), }; As you can see I use a general metadata array for the fields so I do not need to reimplement the messages for supporting a new serializer as it was done in the C++ version. But this is quite a non-trivial task with these materials. Anyway, thank you for the link, it helped much highlighting the issue in my case after all. Date Time Of Last Edit: 2019-03-19 16:23:42
|
| [2019-03-20 22:08:36] |
| |
|
And sorry but I have to mention this is not correct at all. There is not any documentation inaccuracy other than the symbol ID should be indicated as a 32 bit integer instead of 16-bit. We have corrected this.However, you must always refer to the header file for field sizes, and all of the available message fields and use the data structures there. See below. Certainly you cannot leave out a field in a binary data structure. Clearly that is a mistake. Optional in this case just simply means that you do not have to use it. But certainly you need to keep it at its default value of 0 which the constructor will set for the message. This section tells how a binary serializer should treat unset fields: > In the case of an integer, this will be INT_MAX. • What I really miss is an accurate binary structure layout of the fields. This is available. Go to:https://dtcprotocol.org/ At the menu at the top look at the DTC Files. They are all there. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2019-03-21 08:28:01
|
| [2019-03-21 08:17:11] |
| User406106 - Posts: 6 |
|
Thank you for your latest response. I did not expect further answers so I really appreciate it. > Certainly you cannot leave out a field in a binary data structure. Actually what I learned from the server's responses is that I can leave them out (Size can be less than the actual structure size) if there are only unset optional fields after a field. For the logon fields I created a workaround so I can define a custom unset value for a field so it still can be optional (so a JSON serializer still can omit this field, for example). > At the menu at the top look at the DTC Files. They are all there. Thank you, of course I have them and they helped a lot with the missing fields/wrong sizes/enum sizes. However, they do not indicate the memory alignment gaps between the fields, which is a C++ artifact (see the example in my previous answer). But by this time I know I have to deal with this issue and how to calculate them by the field position. And thank you for correcting the documentation. I know this is a huge material to maintain. Date Time Of Last Edit: 2019-03-21 08:29:31
|
| [2019-03-21 08:26:33] |
| |
|
We will update the structures as soon as we can to fill in the gaps with reserve members. But you can determine where those are at by looking at the size of the members and by this line in the header file: #pragma pack(8) Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
To post a message in this thread, you need to log in with your Sierra Chart account: