Support Board
Date/Time: Wed, 05 Mar 2025 04:15:20 +0000
[Programming Help] - Issues with DTC Historical Data Requests
View Count: 788
| [2022-01-07 19:16:11] |
| User41727 - Posts: 124 |
|
When using the historical data server via DTC with Protobuf encoding in Python, requests can terminate unexpectedly and erroneously. This applies to version 2339 as well as earlier ones. Steps to reproduce: 1) Go through the DTC connection procedure. 2) Request (daily) historical data for any symbol, e.g. SPY for the last 10 years (HISTORICAL_PRICE_DATA_REQUEST) 3) Bars are being returned via HISTORICAL_PRICE_DATA_RECORD_RESPONSE until a point is reached where the DTC server sends an incomplete payload, i.e. the header type is correct, but when reading the data from the socket, fewer bytes have been sent than indicated by the header, resulting in an inability to parse the packet. The aforementioned issue is reproducible every time, but only seems to occur after more than 1000 bars have already been sent. In most cases it took between 1300 and 1400 bars independent of the starting point. The issue can be avoided by not requesting more than 1000 bars per request. However, there appears to be a second issue where the "EndDateTime" setting is being ignored (MaxDaysToReturn = 0), so the server is always reaching the error point if StartDateTime lies too far in the past. The only solution for this is to disconnect the client before that happens and then start new requests for any remaining data in chunks of 1000 bars. |
| [2022-01-10 04:04:34] |
| Sierra_Chart Engineering - Posts: 18656 |
|
You need to look for the issue on your side.. You probably are missing some data. There definitely no issue with the Sierra Chart historical data server provided within Sierra Chart. Here is an example: HD Request # 1 | Downloading Intraday chart data for EURUSD to the file EURUSD.scid. Service: fxcm | 2022-01-09 23:01:51.434
HD Request # 1 | Download start date-time: 2021-12-10 23:01:51.000000. File last date-time: 00:00:00.000000 | 2022-01-09 23:01:51.434 HD Request # 1 | Using server: ds12.sierracharts.com port 10149 | 2022-01-09 23:01:51.435 Socket (2) | Creating socket. | 2022-01-09 23:01:51.443 Socket (2) | Send buffer size: 0 | 2022-01-09 23:01:51.444 Socket (2) | New receive buffer size: 5242880 | 2022-01-09 23:01:51.444 Socket (2) | Receive buffer size: 5242880 | 2022-01-09 23:01:51.444 Socket (2) | Connecting to IP: 204.15.197.14. | 2022-01-09 23:01:51.444 HD Request # 1 | Setting DTC encoding to Protocol Buffers | 2022-01-09 23:01:51.472 HD Request # 1 | Sending historical data logon request message. | 2022-01-09 23:01:51.472 HD Request # 1 | Requesting Intraday data. Start date-time: 2021-12-10 23:01:51. Record interval: 0. Symbol: EURUSD | 2022-01-09 23:01:52.038 HD Request # 1 | Receiving Intraday data for EURUSD starting at 2021-12-12 17:06:05.513000 | 2022-01-09 23:01:52.084 HD Request # 1 | Timestamp of first Intraday data file record written: 2021-12-12 17:06:05.513000 | 2022-01-09 23:01:52.189 Socket (2) | New receive buffer size: 10485760 | 2022-01-09 23:01:52.278 Socket (2) | Receive buffer size: 10485760 | 2022-01-09 23:01:52.278 Socket (2) | New receive buffer size: 16777216 | 2022-01-09 23:01:52.365 Socket (2) | Receive buffer size: 16777216 | 2022-01-09 23:01:52.365 Socket (2) | CloseSocket call. | 2022-01-09 23:02:10.249 Socket (2) | 0 bytes remaining in receive buffer when socket closed. | 2022-01-09 23:02:10.249 Socket (2) | Maximum receive buffer size during socket life: 10550272 | 2022-01-09 23:02:10.249 Socket (2) | Signaling to core to close socket | 2022-01-09 23:02:10.250 Socket (2) | Shutdown started. Waiting for graceful close. | 2022-01-09 23:02:10.250 Historical data download thread signaled to stop. | 2022-01-09 23:02:10.250 HD Request # 1 | Received 2855783 Intraday data records from 2021-12-12 17:06:05.513000 to 2022-01-09 23:01:59.840000 (28.2 days) and wrote 2855783 records for EURUSD | 2022-01-09 23:02:10.251 Socket (2) | Socket gracefully closed by remote side. | 2022-01-09 23:02:10.251 Socket (2) | Closed. | 2022-01-09 23:02:10.251 HD Request # 1 | Add time&sales data | Last record in file date-time: 2022-01-10 04:01:59.840000 | 2022-01-09 23:02:10.252 HD Request # 1 | Add time&sales data | Remembered last file record date-time: 2022-01-10 04:01:59.840000 | 2022-01-09 23:02:10.255 HD Request # 1 | Completion time: 19s | 2022-01-09 23:02:10.256 HD Request # 1 | Intraday data download complete for EURUSD. Unique request ID: 1 | 2022-01-09 23:02:10.256 You can see the very large data transfer. Very fast, and without any issue. Additional information: Data and Trading Communications (DTC) Protocol Server: Historical Price Data Server Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2022-01-10 04:06:22
|
| [2022-01-10 04:29:41] |
| Sierra_Chart Engineering - Posts: 18656 |
|
Another example: HD Request # 1 | Downloading Historical Daily chart data for XAUUSD. Starting date: 1972-01-10. Service: fxcm | 2022-01-09 23:27:33.424
HD Request # 1 | Using server: ds12.sierracharts.com port 10150 | 2022-01-09 23:27:33.424 Socket (2) | Creating socket. | 2022-01-09 23:27:33.426 Socket (2) | Send buffer size: 0 | 2022-01-09 23:27:33.426 Socket (2) | New receive buffer size: 5242880 | 2022-01-09 23:27:33.426 Socket (2) | Receive buffer size: 5242880 | 2022-01-09 23:27:33.426 Socket (2) | Connecting to IP: 204.15.197.14. | 2022-01-09 23:27:33.427 SC DTC Data with Market Depth | Network connection to server complete. | 2022-01-09 23:27:38.165 SC DTC Data with Market Depth | Starting socket receive thread. | 2022-01-09 23:27:38.165 SC DTC Data with Market Depth | Sending encoding request to server: Binary VLS. Compression: standard | 2022-01-09 23:27:38.167 HD Request # 1 | Setting DTC encoding to Protocol Buffers | 2022-01-09 23:27:38.168 HD Request # 1 | Sending historical data logon request message. | 2022-01-09 23:27:38.168 SC DTC Data with Market Depth | Setting DTC encoding to Binary VLS | 2022-01-09 23:27:38.190 SC DTC Data with Market Depth | Sending logon request message. | 2022-01-09 23:27:38.190 HD Request # 1 | Requesting historical Daily data starting at 1972-01-10. | 2022-01-09 23:27:38.722 HD Request # 1 | Receiving historical Daily data for XAUUSD starting at 1972-01-11 | 2022-01-09 23:27:38.760 HD Request # 1 | Writing historical Daily data to the file XAUUSD.dly | 2022-01-09 23:27:38.760 SC DTC Data with Market Depth | Received logon response. | 2022-01-09 23:27:38.845 SC DTC Data with Market Depth | Server Name: SC Realtime Server. | 2022-01-09 23:27:38.845 SC DTC Data with Market Depth | Server protocol version: 8. Client protocol version: 8 | 2022-01-09 23:27:38.846 SC DTC Data with Market Depth | Successfully connected. | 2022-01-09 23:27:38.846 SC DTC Data with Market Depth | Trading is not supported. | 2022-01-09 23:27:38.846 SC DTC Data with Market Depth | Restarting real-time data updates for symbol: XAUUSD | 2022-01-09 23:27:38.846 SC DTC Data with Market Depth | Received security definition for symbol XAUUSD | 2022-01-09 23:27:38.970 Socket (2) | CloseSocket call. | 2022-01-09 23:27:39.043 Socket (2) | 0 bytes remaining in receive buffer when socket closed. | 2022-01-09 23:27:39.043 Socket (2) | Maximum receive buffer size during socket life: 417792 | 2022-01-09 23:27:39.043 Socket (2) | Signaling to core to close socket | 2022-01-09 23:27:39.043 Socket (2) | Shutdown started. Waiting for graceful close. | 2022-01-09 23:27:39.043 Historical data download thread signaled to stop. | 2022-01-09 23:27:39.044 HD Request # 1 | Received 12805 Daily data records from 1972-01-11 00:00:00 to 2022-01-10 00:00:00 (50.0years) and wrote 12805 records for XAUUSD | 2022-01-09 23:27:39.045 HD Request # 1 | Completion time: 6s | 2022-01-09 23:27:39.045 HD Request # 1 | Daily data download complete for XAUUSD. Unique request ID: 1 | 2022-01-09 23:27:39.045 Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing |
| [2022-01-10 13:59:23] |
| User41727 - Posts: 124 |
|
I have only verified the issue with daily bars. There are no data missing, and the issue does not occur based on a specific date, but based on the number of bars received. It is impossible that the issue is on my side as I am able to load the data in smaller chunks and the very same code used to work without problems in the past. The only thing that has changed is the SC version. Either way, none of this explains why the "EndDateTime" setting is being ignored, which is definitely an issue on your side as "StartDateTime" works just fine. |
| [2022-01-10 14:42:15] |
| |
|
The issue could not be on the Sierra Chart side. It absolutely is not. The code/functionality, for sending historical data is identical, no matter what method of encoding is used. And encoding data using protocol buffers is very straightforward and is done consistently no matter what type of data is being sent (Historical price data, streaming market data, trading related data) and protocol buffers is extensively tested in Sierra Chart and used by other users. The EndDateTime field is not supported when requesting historical data with a time period of 1 day and greater. And we ran a test for historical daily bars, in the most recent log we posted. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2022-01-10 14:43:20
|
| [2022-01-10 16:48:21] |
| User41727 - Posts: 124 |
|
The issue cannot possibly be with my code as it is literally only reading data from the socket, which works reliably up to a certain number of bars, but consistently fails when a certain threshold is being exceeded. This means this is either on you or the OS. The EndDateTime field is not supported when requesting historical data with a time period of 1 day and greater.
Why would that not be the case, and, even worse, why would that fact not be documented? |
| [2022-01-10 17:27:46] |
| |
|
The issue cannot be on the Sierra Chart side and we did update the documentation regarding EndDateTime.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
To post a message in this thread, you need to log in with your Sierra Chart account: