Login Page - Create Account

Support Board


Date/Time: Tue, 26 Nov 2024 19:37:18 +0000



[Programming Help] - SerraChart as a DTC Protocol Client

View Count: 742

[2023-08-09 22:14:27]
d9e5c763 - Posts: 108
I have referred to the link https://www.sierrachart.com/index.php?page=doc/DTC_TestClient.php, and I'm have a few unresolved issues. Currently, I have only implemented the download and viewing of historical data, and I'm seeking assistance with the following:

1. Unable to display real-time prices including trading dom, time and sales window, qouteboard and chartbooks. (I have solved it myself)

2. Whether I turn the "Enable JSON Message Logging" option on or off in the DTC Service, I still see the output of "JSON data:" in the Message Log. I must save and then restart the program for this option to take effect.

3. Does the transmission of Sierra Chart data files (*.scid) through the DTC protocol require strict adherence to chronological order? If missing data is found earlier in the sequence, is the only option to delete and redownload it in Sierra Chart?

4. a question regarding HISTORICAL_PRICE_DATA_REQUEST:
When I use an external script to write historical trading data into the SCID file, what I can ascertain is that the last trade time in the SCID file is UNIX 1693583988553, which corresponds to UTC 2023-09-01 15:59:48.553. Then, when I open Sierra Chart and connect to my DTC server, the software requests data starting from UNIX 1693583988000, which is UTC 2023-09-01 15:59:48. If I simply return data based on this timestamp, I encounter the issue of writing duplicate trading data into the SCID file. Or is it that Sierra Chart merely requests extra data for the software to compare? Even so, I often face situations where multiple trades with the same buy/sell direction are generated at the same millisecond. How should this be resolved, or how does Sierra Chart handle such situations?
Date Time Of Last Edit: 2023-09-01 20:52:14
Attachment Deleted.
Attachment Deleted.
Attachment Deleted.
Attachment Deleted.
Attachment Deleted.
[2023-08-10 23:04:02]
Sierra_Chart Engineering - Posts: 17182
1. You will need to determine the reason for this. We do not know what the issue is in your case.

3. Yes to the second question.

Is it necessary to use a format other than JSON to display real-time quotes?
No.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, use the Teton service:
Sierra Chart Teton Futures Order Routing
[2023-09-01 20:56:03]
d9e5c763 - Posts: 108
I have an additional question about HISTORICAL_PRICE_DATA_REQUEST. Everything seems to be working well, except for a few minor details that still need to be addressed. Also, concerning the DTC protocol, is this within the scope of technical support offered here?
[2023-09-05 18:44:05]
d9e5c763 - Posts: 108
Upon further observation, I've discovered another reason why the data doesn't align correctly during multiple dis/connect dtc data feed. Although I am sending JSON data into Sierra Chart like this:
JSON data: {"Type": 107, "SymbolID": 1, "AtBidOrAsk": 2, "Price": 25731.2, "Volume": "7", "DateTime": 1693937669.857} | 2023-09-05 18:14:29.899
which includes millisecond-level information of type "t_DateTimeWithMilliseconds," I am certain that in the Time and Sales Window in Sierra Chart, I can only see data with second-level precision (even though the options can display numbers like .000001, it seems to be just an issue of floating-point precision).

Additionally, when reading the scid binary files, I find that the stored information also has second-level precision as the smallest unit. Although HISTORICAL_PRICE_DATA_REQUEST only supports "t_DateTime" precision, I can indeed determine the specific information to be passed by reading the tail of the scid file when I receive an HISTORICAL_PRICE_DATA_REQUEST. However, because there is millisecond-level precision loss in this process, I am unable to do so.

output_file_name = f"datas/{product}_OKX.scid"
record_struct = struct.Struct('Q f f f f I I I I')
record_length = struct.calcsize('Q f f f f I I I I')

with open(output_file_name, 'rb') as scid_file:
scid_file.seek(-record_length, os.SEEK_END)
record_data = scid_file.read(record_length)
timestamp, _, _, _, _, _, _, _, _ = struct.unpack('Q f f f f I I I I', record_data)
scid_last_timestamp = int((timestamp / 1000) + int((dt(1899, 12, 30) - dt(1970, 1, 1)).total_seconds() * 1000))
print(f"SCID LAST TIMESTAMP: {scid_last_timestamp}")

I recall that Sierra Chart had a major update in the past, involving changes to the SCID file format, requiring all files to be re-downloaded because the timestamp format had been altered. Could it be that JSON data doesn't support this level of time precision? Because it seems that there isn't such a problem with CME data feed
Date Time Of Last Edit: 2023-09-05 19:01:41
[2023-09-05 21:28:29]
d9e5c763 - Posts: 108
Based on the steps above, I further proceeded with the following:
I wrote all the trading products available on the OKX exchange into SCID files, of course with millisecond-precision data. I found that the trade volumes for all the products are accurate. For instance, the timestamp at the tail end of the SCID file is 1693929407830 (UTC+8 2023-09-05 23:56:47.830), and Sierra Chart requests data starting from 1693929407000 (2023/09/05 23:56:47). In the end, the data in the SCID file is complete and correct. I verified it like this:
I fetched 1-hour candlestick data from the exchange API and calculated all the trades within that hour in the SCID file. The data for over a hundred products was correct.

The only incorrect part appears after multiple disconnections/reconnections with my DTC program. And these data entries, as mentioned before, do have millisecond information in Sierra Chart's log, but they are lost in the software.

I feel I should lay out all the information I could gather. I indeed want to know if there's an option to control this time precision when writing the data. If not, would switching to another encoding format resolve the issue? If the answer is no, will it be addressed in the future? It seems challenging to find a complete DTC server implementation online. Is this functionality not being actively maintained?

Since haven't answered any of my questions above, I don't know if this falls within the scope of technical support. I've indeed searched for relevant parts in the DTC documentation but couldn't find any related descriptions. If my posts are inappropriate, please point it out. Thank you very much.
Date Time Of Last Edit: 2023-09-05 21:36:05
[2023-09-08 13:40:42]
d9e5c763 - Posts: 108
I continue to troubleshoot. First, I decided to switch to binary encoding. Since Python has the struct module, this looks very straightforward. For example, I can send the SECURITY_DEFINITION_RESPONSE based on the documentation and the definitions in DTCProtocol.h. Here's the specific code:

async def send_security_definitions(writer):
with open(QUOTE_BOARD_FILE_LOCATION, 'w') as file:
for product in PRICES_DICT:
file.write(f"{product}_OKX\n")

total_products = len(PRICES_DICT)

for index, product in enumerate(PRICES_DICT):
instrument_data = INSTRUMENTS_DICT.get(product, {})
multiplier = PRODUCT_MULTIPLIERS.get(product, 1)
tick_size = str((Decimal(instrument_data.get('tick_size', 0)) * multiplier).normalize())
tick_value = str((Decimal(instrument_data.get('tick_size', 0)) * (Decimal(instrument_data.get('ct_value', 0)))).normalize())
decimal_places = len(tick_size.split('.')[1]) if '.' in tick_size else 0
price_display_format = decimal_places if decimal_places <= 9 else -1

product_prefix = product.split('-')[0]
description = await get_product_descriptions(product_prefix)

format_string = '2Hi64s16si64sfifB2f32sBfBIi3fIfBf64si2f8sfIB'
security_definition_data = struct.pack(
format_string,
struct.calcsize(format_string), # Size (uint16)
dtc["SECURITY_DEFINITION_RESPONSE"], # Type (uint16)
0, # RequestID (int32)
product.encode('ascii'), # Symbol (char, 64 bytes)
b"OKX", # Exchange (char, 16 bytes)
0, # SecurityType (int32)
description.encode('ascii'), # Description (char, 64 bytes)
float(tick_size), # MinPriceIncrement (float)
price_display_format, # PriceDisplayFormat (int32)
float(tick_value), # CurrencyValuePerIncrement (float)
1 if index == total_products - 1 else 0, # IsFinalMessage (uint8)
1.0, # FloatToIntPriceMultiplier (float)
1.0, # IntegerToFloatPriceDivisor (float)
b"", # UnderlyingSymbol (char, 32 bytes)
0, # UpdatesBidAskOnly (uint8)
0.0, # StrikePrice (float)
0, # PutOrCall (uint8)
0, # ShortInterest (uint32)
0, # SecurityExpirationDate (int32)
0.0, # BuyRolloverInterest (float)
0.0, # SellRolloverInterest (float)
0.0, # EarningsPerShare (float)
0, # SharesOutstanding (uint32)
0.0, # IntToFloatQuantityDivisor (float)
0, # HasMarketDepthData (uint8)
1.0, # DisplayPriceMultiplier (float)
b"", # ExchangeSymbol (char, 64 bytes)
0, # RolloverDate (int32)
0.0, # InitialMarginRequirement (float)
0.0, # MaintenanceMarginRequirement (float)
b"USDT", # Currency (char, 8 bytes)
0.0, # ContractSize (float)
0, # OpenInterest (uint32)
0 # IsDelayed (uint8)
)

writer.write(security_definition_data)
await writer.drain()

However, I also noticed that MARKET_DATA_UPDATE_BID_ASK and MARKET_DATA_UPDATE_TRADE have issues similar to those mentioned above with JSON encoding:
For MARKET_DATA_UPDATE_BID_ASK, I can see on the Quote Board that the ask price and ask size are displayed as very large numbers, while the bid price and bid size are not. I have double-checked the documentation and DTCProtocol.h file, and everything seems to be in accordance with the specifications. However, MARKET_DATA_UPDATE_BID_ASK_COMPACT doesn't have these issues.
Similarly, for MARKET_DATA_UPDATE_TRADE, in the Time and Sales Window, I see that the Volume is always 1 and the Price is always 0, but the Time is displayed correctly in milliseconds. MARKET_DATA_UPDATE_TRADE_COMPACT doesn't have these issues, and it appears that MARKET_DATA_UPDATE_TRADE_COMPACT is consistent with MARKET_DATA_UPDATE_TRADE in JSON encoding. (MARKET_DATA_UPDATE_TRADE_COMPACT is not listed in the documentation but is clearly defined in the DTCProtocol.h file).
I haven't tested historical data yet as it's currently not operational. I might have gotten the order of the data wrong; I'm not sure yet.
[2023-09-08 16:00:42]
d9e5c763 - Posts: 108
It looks like my issue has been resolved. The key was the compiler directives in the header file.

cat dtc.h | grep pragma
#pragma once
// not beneficial. So this could be changed to #pragma pack(1). It is
#pragma pack(push, 8)
#pragma pack(push, 1)
#pragma pack(pop)
#pragma pack(push, 1)
#pragma pack(pop)
#pragma pack(push, 1)
#pragma pack(pop)
#pragma pack(push, 1)
#pragma pack(pop)
#pragma pack(push, 1)
#pragma pack(pop)
#pragma pack(pop)

The message types that didn't have issues are all tightly packed.

As for the JSON information, I also found some related discussions, such as this one on Sierra Chart's support board(DTC Server only sending MARKET_DATA_UPDATE_TRADE_COMPACT). Perhaps my earlier issues are related to this.
[2023-09-09 03:02:12]
d9e5c763 - Posts: 108
After attempting to write several DTC messages, I realized that if I wanted to implement trading functionality as well, the error rate would be too high. Using CppHeaderParser could also introduce other invisible issues. So, I decided to use GPB (Google Protocol Buffers). After my tests, I found that even if I sent a MARKET_DATA_UPDATE_TRADE message, Sierra Chart internally still treats it as MARKET_DATA_UPDATE_TRADE_COMPACT:


Side = trade['side']
Side = dtc_pb2.AtBidOrAskEnum.AT_BID if Side.upper() == "SELL" else dtc_pb2.AtBidOrAskEnum.AT_ASK
Price = float(trade['px']) * multiplier
Volume = int(trade['sz']) / 1.0
DateTime = int(trade['ts']) / 1000.0
LastTradeId = int(trade['tradeId'])

MarketDataUpdateTrade = dtc_pb2.MarketDataUpdateTrade()
MarketDataUpdateTrade.SymbolID = SymbolID
MarketDataUpdateTrade.AtBidOrAsk = Side
MarketDataUpdateTrade.Price = Price
MarketDataUpdateTrade.Volume = Volume
MarketDataUpdateTrade.DateTime = DateTime

SerializedMarketDataUpdateTrade = MarketDataUpdateTrade.SerializeToString()

DeserializedMarketDataUpdateTrade = dtc_pb2.MarketDataUpdateTrade()
DeserializedMarketDataUpdateTrade.ParseFromString(SerializedMarketDataUpdateTrade)
print("Deserialized DateTime: ", DeserializedMarketDataUpdateTrade.DateTime)

SerializedMarketDataUpdateTradeSize = len(SerializedMarketDataUpdateTrade) + 4
Header = struct.pack('<HH', SerializedMarketDataUpdateTradeSize, dtc_pb2.DTCMessageType.MARKET_DATA_UPDATE_TRADE)
FullSerializedMarketDataUpdateTrade = Header + SerializedMarketDataUpdateTrade

writer.write(FullSerializedMarketDataUpdateTrade)
await writer.drain()

I could see that the data I eventually sent in did indeed have millisecond-precision floating-point numbers, but this data disappeared within the software. MarketDataUpdateBidAskFloatWithMicroseconds did not have this issue. Another point is that writing to SCID files through HistoricalPriceDataTickRecordResponse now also has millisecond precision. So, one of the issues (which would occur when using JSON) has been resolved. I hope this issue can be fully addressed in the future.
[2023-09-10 23:12:36]
d9e5c763 - Posts: 108
I've added a picture that shows the timestamp information under GPB (Google Protocol Buffers). Additionally, I've also implemented the sending of MarketDataUpdateTrade using binary encoding. It seems that after sending the data packed according to the alignment specified in the header file, I can't see millisecond-level information in the timestamps within Sierra Chart. However, when I use incorrect data types like int64_t or float, I can see millisecond precision, but of course, this data is incorrect. So, it looks like this is related to how Sierra Chart internally processes the data. This is because messages from HistoricalPriceDataTickRecordResponse that also use the t_DateTimeWithMilliseconds data type do indeed leave millisecond-level information in the SCID files when sent via GPB. I'm 100% sure of this.

In scenarios where MarketDataUpdateTrade is sent at a very high frequency, CPU usage in Sierra Chart is significantly higher when using GPB compared to binary encoding. This doesn't seem like a good choice. It appears that using GPB for data other than high-frequency data and using binary encoding for the rest is the best approach.

Feel free to elaborate or ask further questions if needed.
imagescreenshot-20230911-065817.jpg / V - Attached On 2023-09-10 23:12:33 UTC - Size: 201.21 KB - 100 views
[2023-11-03 16:46:25]
d9e5c763 - Posts: 108
After the last post, the problems described above have been largely resolved. For the record, I'm writing down the key points I remember below:
Do not use MARKET_DATA_UPDATE_TRADE to update trade data. This message type will ignore the millisecond data. Although I have not found this mentioned in the documentation or elsewhere, it is indeed the case.
The solution is to use MARKET_DATA_UPDATE_TRADE_WITH_UNBUNDLED_INDICATOR_2. Although it also ignores the microseconds part of the data, this is clearly not a problem. Both message types will use the microseconds part to ensure that each trade data timestamp is unique. This is mentioned in the documentation section "Intraday Data File Format."

The key to ensuring the continuity of real-time and historical data written to the SCID file is to ensure that historical data must include the latest trade data when sent after receiving a request, especially when EndDateTime is 0. Previously, I unnecessarily wrote a piece of code to ensure that MARKET_DATA_UPDATE_TRADE and HISTORICAL_PRICE_DATA_TICK_RECORD_RESPONSE data are connected and do not overlap, which seems unnecessary and could even disrupt Sierra Chart's own processing mechanism.

Use HISTORICAL_PRICE_DATA_RECORD_RESPONSE instead of HISTORICAL_PRICE_DATA_TICK_RECORD_RESPONSE to send historical data, even for tick-level data. There's a detail not mentioned in the documentation and the proto file that initially led me to choose HISTORICAL_PRICE_DATA_TICK_RECORD_RESPONSE. It is known from the C++ header files that HISTORICAL_PRICE_DATA_RECORD_RESPONSE's t_DateTime can actually be t_DateTimeWithMicrosecondsInt. The software handles this automatically, so only by using HISTORICAL_PRICE_DATA_RECORD_RESPONSE can you send tick data with millisecond precision.

Now, only on some very thinly traded products, such as those that only have a trade every few minutes, there may be inconsistencies between the data written to the SCID file and the actual data. However, these are easily detectable. Once detected, there's no problem as long as Sierra Chart re-downloads the data. The method of checking remains the same: extract the SCID data, generate the specified candlestick data, such as for 5 minutes, continuously produce a certain number, such as 5 bars of 5-minute candlestick data, and compare it with the exchange data. If there are inconsistencies, they always involve the physical trading volume being more than the server's. That is, there may be overlaps, but no cases of missing data.

To post a message in this thread, you need to log in with your Sierra Chart account:

Login

Login Page - Create Account