Support Board
Date/Time: Fri, 29 Nov 2024 18:44:37 +0000
Post From: Your Market Depth Studies have a MAJOR flaw!
| [2023-01-18 00:34:06] |
| voided - Posts: 45 |
|
I must be making a mistake, then. I have been testing my program with CLG23_FUT_CME on the .depth file for January 11th: CLG23_FUT_CME.2023-01-11.depth I read and parse the depth records. After reading your post, I checked the timestamps, printing each records timestamp, in the sequence read from disk, along with the duration (in seconds) since the previous record. Several had negative durations: ----- 3882564794899000 -0.393 3882573483952000 -1.700 3882579410123000 -0.026 3882580289013000 -0.644 3882595486811000 -0.027 3882596068603000 -0.129 3882597650570000 -0.877 3882599583655000 -0.137 3882601154027000 -0.063 3882602609175000 -0.165 3882604751565000 -0.019 3882605352708000 -0.081 3882606215738000 -0.026 3882607331439000 -0.030 3882608099468000 -0.008 3882609234477000 -0.022 3882612036599000 -0.094 3882612583541000 -0.023 3882613812842000 -0.003 3882617143484000 -0.006 3882617437276000 -0.008 3882619084757000 -0.137 3882621050867000 -0.555 3882621598414000 -0.237 3882622719117000 -0.007 3882623830650000 -0.024 3882625978142000 -0.014 3882627429297000 -0.198 3882628440203000 -0.069 3882628942626000 -0.617 3882629886540000 -0.415 3882631069988000 -0.427 3882631356149000 -0.110 3882633649364000 -0.099 3882634415297000 -3.419 3882634950445000 -2.342 3882636000080000 -305.156 ----- I also noticed some extremely large gaps, during which the intraday .scid files recorded many trades. A full list of depth records with more than 10 seconds since the prior record follows: ----- 3882614052330951 164.757 3882615577679513 1379.228 3882616627764529 974.451 3882617143489678 428.746 3882617437284224 206.422 3882619084893547 1565.991 3882620012736179 850.890 3882621051421557 963.978 3882621598650844 459.398 3882622719124114 1022.769 3882623830674489 1042.644 3882624178089245 262.180 3882625978155526 1717.477 3882627429495371 1366.029 3882628053722469 476.641 3882628440272371 310.200 3882628943243471 403.628 3882629886955481 856.914 3882631070415447 1100.475 3882631356258940 136.383 3882633649463388 2218.236 3882634418715692 668.591 3882634952786561 400.066 3882635862949969 832.561 3882636305235979 321.985 3882638700426000 2700.346 3882638772532000 48.775 3882638790394000 17.862 3882638830344000 39.950 3882638869399000 39.055 3882638893349000 21.514 3882638938351000 45.002 3882638963351000 25.000 3882638975707000 12.356 3882639000351000 18.000 3882639019352000 19.001 3882639037352000 18.000 3882639071353000 31.000 3882639104019000 26.665 3882639121203000 15.088 3882639166356000 17.001 3882639183356000 13.000 3882639195200000 11.844 3882639247357000 47.733 3882639262358000 15.001 3882639277358000 11.000 3882639298358000 16.000 3882639319021000 15.000 3882639332101000 13.080 3882639367022000 15.000 3882639582009000 19.769 3882640234350000 10.550 3882640949746000 12.867 3882641494432000 10.265 3882641965455000 12.540 3882642662214000 12.811 3882642742091000 10.002 3882642946160000 16.128 3882642960183000 14.023 3882642998780000 16.471 ----- As you can see, there are gaps of up to 2700 seconds (45 minutes) between depth records. So my questions are: 1. Are the records not guaranteed to be in sequence? 2a. Why are there such long intervals between some of the records? 2b. If these intervals indicate missing data, is there any way to obtain it? Regarding 2b, I tried right-clicking on an intraday chart for CLG23_FUT_CME from a point prior to Jan 11th but within the 30 day window (i.e., January 2nd 2023, 15:00) and selecting "Download Depth Data from Date-time," but nothing changed in my depth files. I had deleted files prior to January 6th, for example, and they were not recreated. None of the other existing depth files past that point changed, either. Here are the entries in the message log subsequent to selecting "Download Depth Data from Date-time": ----- Triggering next historical data download in queue. | 2023-01-17 17:06:57.787 No download requests in the queue to start downloads for. | 2023-01-17 17:06:57.787 CLG23_FUT_CME[M] 30 Min #18 | Reloading chart. | 2023-01-17 17:06:57.827 Socket (5) | Socket gracefully closed by remote side. | 2023-01-17 17:06:57.830 Socket (5) | Closed. | 2023-01-17 17:06:57.831 CLG23_FUT_CME[M] #2 | Reloading chart. | 2023-01-17 17:06:57.835 CLG23_FUT_CME[M] #2 | StartDateTimeForLoadingOrderFills: 00:00:00 | 2023-01-17 17:06:57.913 Opened cached Depth file: C:\SierraChart\Data\MarketDepthData\CLG23_FUT_CME.2023-01-18.depth | 2023-01-17 17:06:58.976 DTC Client socket (2) | Timed out waiting for shutdown. Performing an immediate socket close. | 2023-01-17 17:07:10.254 DTC Client socket (2) | Closed. | 2023-01-17 17:07:10.254 Opened cached Intraday file: C:\SierraChart\Data\NGH23-NGJ23.FUT_SPREAD.CME.scid. Thread ID: 1584 | 2023-01-17 17:07:24.332 DTC Client socket (2) | Deleting network socket now after delay. | 2023-01-17 17:07:40.273 Added historical Market depth data request for CLG23_FUT_CME to the queue. | 2023-01-17 17:08:21.672 Market depth data recording state for symbol CLG23_FUT_CME is set to download 'Pending'. | 2023-01-17 17:08:21.672 Triggering next historical data download in queue. | 2023-01-17 17:08:21.672 HD Request # 168 | Downloading Market depth data for CLG23_FUT_CME. Service: nymex.marketdepth. | 2023-01-17 17:08:21.673 HD Request # 168 | Download start date-time: 2023-01-12 17:08:21.000000. | 2023-01-17 17:08:21.682 HD Request # 168 | Using server: ds28.sierracharts.com port 10149 | 2023-01-17 17:08:21.683 Socket (2) | Creating socket. | 2023-01-17 17:08:21.683 Socket (2) | New receive buffer size: 5242880 | 2023-01-17 17:08:21.683 Socket (2) | Connecting to IP: 142.215.18.147. | 2023-01-17 17:08:21.683 HD Request # 168 | Setting DTC encoding to Binary VLS | 2023-01-17 17:08:21.786 HD Request # 168 | Sending historical data logon request message. | 2023-01-17 17:08:21.786 HD Request # 168 | Requesting market depth data. Start date-time: 2023-01-12 17:08:21. Symbol: CLG23 | 2023-01-17 17:08:22.365 HD Request # 168 | Decompressing data. | 2023-01-17 17:08:22.445 HD Request # 168 | Receiving market depth data for CLG23_FUT_CME starting at 2023-01-12 17:08:21.000000 | 2023-01-17 17:08:22.456 HD Request # 168 | Truncating file to starting date-time of 2023-01-12 17:08:21.000000. | 2023-01-17 17:08:22.456 HD Request # 168 | Timestamp of first market depth data file record written: 2023-01-12 17:08:21.000000. | 2023-01-17 17:08:22.667 Socket (2) | CloseSocket call. | 2023-01-17 17:08:54.955 Historical data download thread signaled to stop. | 2023-01-17 17:08:54.955 Socket (2) | Shutdown started. Waiting for graceful close. | 2023-01-17 17:08:54.955 HD Request # 168 | Received 16337566 Market depth data records from 2023-01-12 17:08:21.000000 to 2023-01-17 17:08:51.521000 (5.0 days) and wrote 16337566 records for CLG23_FUT_CME | 2023-01-17 17:08:54.955 HD Request # 168 | Completion time: 33s. | 2023-01-17 17:08:54.955 HD Request # 168 | Market depth data download complete for CLG23_FUT_CME. Unique request ID: 168 | 2023-01-17 17:08:54.955 Removing historical data download ID 168. | 2023-01-17 17:08:54.955 Real-time market depth file updates started for CLG23_FUT_CME | 2023-01-17 17:08:54.955 Opened cached Depth file: C:\SierraChart\Data\MarketDepthData\CLG23_FUT_CME.2023-01-18.depth | 2023-01-17 17:08:54.961 CLG23_FUT_CME[M] #2 | Flagging to reload Intraday data on market depth download completion. | 2023-01-17 17:08:54.961 CLG23_FUT_CME[M] 30 Min #18 | Flagging to reload Intraday data on market depth download completion. | 2023-01-17 17:08:54.961 Triggering next historical data download in queue. | 2023-01-17 17:08:54.961 No download requests in the queue to start downloads for. | 2023-01-17 17:08:54.961 Socket (2) | Socket gracefully closed by remote side. | 2023-01-17 17:08:55.005 Socket (2) | Closed. | 2023-01-17 17:08:55.005 CLG23_FUT_CME[M] #2 | Reloading chart. | 2023-01-17 17:08:55.009 CLG23_FUT_CME[M] 30 Min #18 | Reloading chart. | 2023-01-17 17:08:55.017 CLG23_FUT_CME[M] #2 | StartDateTimeForLoadingOrderFills: 00:00:00 | 2023-01-17 17:08:55.088 Opened cached Depth file: C:\SierraChart\Data\MarketDepthData\CLG23_FUT_CME.2023-01-18.depth | 2023-01-17 17:08:55.474 Opened cached Intraday file: C:\SierraChart\Data\CLQ23-CLU23.FUT_SPREAD.CME.scid. Thread ID: 1584 | 2023-01-17 17:10:20.304 ----- It appears to have only touched the depth files for 1-18 and 1-17, despite files from 1-6 through 1-18 existing. Thanks for your help. edit: And one related question. In order to maintain the correct order book state, is it enough to parse the depth records alone? Or do trades need to be subtracted from the existing quantities at each level? I was under the assumption that every change to the quantity at each level would come from the add/mod/delete depth record commands, but if I also need to subtract the trades, that would be nice to know. Date Time Of Last Edit: 2023-01-18 01:12:50
|