DTC Protocol Discussion Forum
- DTC Protocol Discussion Forum |
- Search Board |
- Control Panel |
- View My Posts / Threads |
- Direct Messages
Date/Time: Fri, 29 Nov 2024 14:33:05 +0000
Historical data zip format?
View Count: 4815
| [2016-10-24 17:44:35] |
| DabbaDo - Posts: 148 |
|
From Sierra Chart server, I receive historical price data without problems when it is NOT zipped. When zipped (i.e. the header shows UsingZLibCompression == 1), the next bytes coming across the wire are in what format? The first two bytes seem NOT to be compressed. Are they a batch size? And the next two bytes seem not to be a DTCMessageType, unless perhaps they are compressed. In other words, is there some documentation/guidance about the bytes sent from the server following a HistoricalPriceDataResponseHeader message where UsingZLibCompression == 1? Cheers, Dale P.S. I figured this out. I must skip past the 2-byte ZLib header (ref. https://tools.ietf.org/html/rfc1950) and then the DeflateStream class decompresses the records with no problems -- ref. https://msdn.microsoft.com/en-us/library/system.io.compression.deflatestream(v=vs.110).aspx [From version 4.5, .Net started using ZLib under the covers.] [But it still might be worth a documentation one-liner, IMHO.] Date Time Of Last Edit: 2016-10-25 09:16:09
|
| [2016-10-25 04:26:03] |
| |
|
We are not familiar with this because we use this library which handles all of this: http://www.zlib.net/ |
| [2018-01-06 21:18:37] |
| AndrewAMD1 - Posts: 31 |
|
I have a question about zlib with DTC. (Long term, I might implement this if I run into bandwidth problems.) So with uncompressed messages, I can always read the type and size up front and size my buffers accordingly. (This is using C++17 & Boost Asio.) But I think it is not this simple when using zlib. Okay, so I request historical data and get an uncompressed header. But then the rest of the messages are in compressed zlib? I need a feasible way to allocate buffers and decode without ruining the decompression process. 1) In general, how should I approach receiving zlib data from the historical server? 2) How am I supposed to know when I received all messages for a particular request if the "last message" flag is already encoded? |
| [2018-01-07 03:22:45] |
| |
|
Okay, so I request historical data and get an uncompressed header. But then the rest of the messages are in compressed zlib? Yes.1. You should do this incrementally. When you get a batch of data on the socket, give it to zlib. It will respond with some uncompressed data and you put that into a buffer and then process as many complete messages out of it that you can. Continue with this process. 2. You will be looking at the decompressed messages and you know you are done, when there is the flag indicating the final record has been sent. s_HistoricalPriceDataRecordResponse::IsFinalRecord |
To post a message in this thread, you need to log in with your Sierra Chart account: