Support Board
Date/Time: Sat, 23 Nov 2024 09:07:53 +0000
SC loads THOUSANDS OF YEARS of ES futures contracts after a crash
View Count: 385
| [2024-06-24 13:42:10] |
| User512353 - Posts: 42 |
|
SC 2646 Trade Service: IBKR Symbol: ES OS: Linux with wine After a computer crash (not caused by SC), while SC was running with an open Chartbook that included ES futures contracts: After restart of the computer, restart of SC and then loading the same chartbook that was active before the crash, SC is now trying to load data for ES contracts for thousands of years into the future. This is observed by SC creating the files ES-yyyymm-CME.dly and ES-yyyymm-CME.scid in the SC data folder for lots of years into the future. The SC application becomes unresponsive. I killed it when it arrived at year 5000 or so. The same can be observed for other chartbooks that include the ES symbol, which were not open during the crash. So, it is probably not a problem that is related to the chartbook itself (i.e. corrupt chartbook config data). Chartbooks not involving the ES symbol load normally. The observed behavior is repeatable. Please advise how to fix this problem. Thanks! |
| [2024-06-24 13:55:00] |
| User512353 - Posts: 42 |
|
The created files are always 56 bytes for the .scid file and 58 bytes for the .dly file Even for existing contracts, like ES-202412-CME these file sizes are true. Which means that SC is seemingly not really trying to download any valid data. It just creates the files. |
| [2024-06-24 14:20:59] |
| Sierra_Chart Engineering - Posts: 17145 |
|
What is the exact symbol of the chart? And what is the computers date set to? And what is this set to: Continuous Futures Contract Charts: Continuous Futures Contract Rollover Options Not sure how this can happen but it must relate to the year of the symbol of the chart and possibly the computers date but not sure how the computers date would have a relevancy. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing |
| [2024-06-24 16:28:18] |
| User512353 - Posts: 42 |
|
Exact symbol is ES-202409-CME; volume-based rollover, back-adjusted OS time zone is CEST (UTC+2) (Central Europe) SC time zone is UTC my ES charts use time zone CDT (UTC-5) (Chicago) I could resolve the problem as follows: start SC new chartbook new historical chart for ES-202409-CME new intraday chart for ES-202409-CME -> default is that these charts do NOT have any rollover enabled then, for the intraday chart, enable volume-base rollover, back-adjusted => this worked. older ES contracts loaded and no craziness with year 5000+. Closed this new chartbook. Loaded an existing chartbook with ES, and then it also worked. It also works with the chartbook that was loaded during the crash. Seems that the problem has disappeared. (Except that I need to clean up about 2x 4x 3000 = 24000 files.) But questions remain: 1. If SC is started, is it aware of the termination of a previous execution? 2. If yes, does it then execute any recovery/repair action for potentially corrupt data? 3. Does SC generally do any consistency checks on configuration/trade/price data? I cannot be sure that today's price/volume data is correct. I will reload today's data after today's RTH session. But question 3 is valid for a much wider scale of "incidents". |
| [2024-06-24 22:16:06] |
| User512353 - Posts: 42 |
|
The SC message logs for the incidents where it went into the spin loop when loading the chartbook end like this: 2024-06-24 13:18:51.199 | ES-202409-CME [CBV] #2 | Reloading chart. 2024-06-24 13:18:51.303 | + Loading additional symbol data. 2024-06-24 13:18:52.519 | Added historical Intraday data request for ES-202412-CME to the queue. 2024-06-24 13:18:52.521 | ES-202409-CME [CBV] #2 | Performing continuous chart historical Daily data download for symbol ES-202412-CME starting at last date in file. 2024-06-24 13:18:52.521 | Added historical Daily data request for ES-202412-CME to the queue. 2024-06-24 13:18:52.522 | Added historical Intraday data request for ES-202503-CME to the queue. 2024-06-24 13:18:52.523 | ES-202409-CME [CBV] #2 | Performing continuous chart historical Daily data download for symbol ES-202503-CME starting at last date in file. 2024-06-24 13:18:52.523 | Added historical Daily data request for ES-202503-CME to the queue. 2024-06-24 13:18:52.524 | Added historical Intraday data request for ES-202506-CME to the queue. 2024-06-24 13:18:52.526 | ES-202409-CME [CBV] #2 | Performing continuous chart historical Daily data download for symbol ES-202506-CME starting at last date in file. 2024-06-24 13:18:52.526 | Added historical Daily data request for ES-202506-CME to the queue. 2 [... and so on in infinity...] |
| [2024-06-24 22:36:49] |
| Sierra_Chart Engineering - Posts: 17145 |
|
1. No. 2. When saved Sierra Chart files of any type are being read, and there is any improper data or structure being detected, it handles that particular condition gracefully and appropriately. This is not referring to invalid prices. We cannot say what is happening unless we see the Message Log. The only thing we can think of is this depends upon the symbol. The symbol must have been thousands of years into the future. Possibly the computers date could cause a problem, but we would have to look into that. 3. We do not understand what you mean by this. If you are using a symbol like ES-401003-CME, Sierra Chart is not going to determine if there is anything wrong with that symbol. There is absolutely no checking at all of historical data. If you suspect there is a problem with the price data in a chart, re-download the historical data. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2024-06-24 22:39:58
|
| [2024-06-24 22:39:11] |
| Sierra_Chart Engineering - Posts: 17145 |
|
We now see the Message Log at post #5. We do not have an explanation for that. We have never seen that before. We can also see that the system Date is correct.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing |
| [2024-06-24 22:45:42] |
| Sierra_Chart Engineering - Posts: 17145 |
|
Actually we see why this happens. This must have been a setting that you set. If you are loading data in the chart based upon a date range and the last Date to load, was into the future and you may very well have set that thousands of years into the future to always cause the current data to load, then this would be the issue. Refer to this setting: Chart Settings: Load Data Limiting Method: Date Range >> Date Range From / To (Chart >> Chart Settings >> Data Limiting >> Load Data Limiting Method: Date Range menu) This is a setting that you would have deliberately set. It is not a Sierra Chart fault in any way. That really is impossible. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2024-06-24 22:45:55
|
| [2024-06-25 00:19:51] |
| allons_trading - Posts: 30 |
|
After moving my two installations of SC to a new NVME drive, my laptop had crashed twice. In one of the SC installations, I noticed the number of data files tripled from 300+ to 900+ with a number of empty ES and NQ data files. ES is enumerated from 09 to 24, while NQ is enumerated from 00 to 99 for each contract. I've checked all my charts in this installation, and the charts that are using a date range to limit data loading all have the Date Range To field empty. My other SC installation also has a few charts using date range to limit data with Date Range To field empty, but no additional empty data files were created under that installation. |
| [2024-06-25 13:30:19] |
| User512353 - Posts: 42 |
|
I use date range for some charts, but I always leave the 'Date Range To' field empty. No settings of the chartbook or any chart therein had been changed. Maybe worth noting: The chart#2 in the above message log excerpt, for which the endless loop apparently occurs, is a TPO study chart (intraday; period: 1 day; 0.50x30min) |
| [2024-06-25 13:51:23] |
| Sierra_Chart Engineering - Posts: 17145 |
|
We did do a code review and there is a safety check used to prevent loading data into the future when the ending date is in the future. There must have been some invalid setting among these group of settings: Chart Settings: Load Data Limiting Method (Chart >> Chart Settings >> Data Limiting >> Date Range and Limiting Method menu) This is the only explanation. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing |
| [2024-06-25 15:12:08] |
| allons_trading - Posts: 30 |
|
To add some more detail, I know my laptop has crashed after the 900+ files were created, and no new files were created. The version of SC when I copied it over to the new drive was 2629. Currently running 2646.
|
| [2024-06-25 17:27:49] |
| User512353 - Posts: 42 |
|
I can only repeat what I wrote above in post #10: No chartbook/chart settings have been changed between the time BEFORE the crash (where no such problems have ever been encountered) and the time AFTER the same chartbook successfully loaded again. It would also have been impossible to change any settings, because the program would almost instantly freeze the GUI when loading the chartbook, while the problem was present after the crash. The only explanations I can think of are: a) SC uses some cached data, which was left in a corrupted state in the crash and which somehow got reset during the procedure I described in post #4 above. b) The crash left some invalid data in the ES-202409-CME.scid or .dly file that SC started acting maliciously upon, until it also somehow was reset. Date Time Of Last Edit: 2024-06-25 17:28:32
|
| [2024-07-29 14:27:33] |
| allons_trading - Posts: 30 |
|
Still trying to track down the random crash on my laptop from time to time, but after restarting, noticed that the number of data files in my Sierra Chart data grew once more, now enumerating all the ES data files from 00-99 to match the behavior I had seen with the NQ data files in this instance.
|
| [2024-07-29 14:44:08] |
| Sierra_Chart Engineering - Posts: 17145 |
|
No, definitely these are not correct at all. There is no cached data. This is nonexistent. And no matter what is contained in the chart data files, that is not going to affect settings. It simply could not. a) SC uses some cached data, which was left in a corrupted state in the crash and which somehow got reset during the procedure I described in post #4 above. b) The crash left some invalid data in the ES-202409-CME.scid or .dly file that SC started acting maliciously upon, until it also somehow was reset. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing |
| [2024-09-04 16:07:31] |
| allons_trading - Posts: 30 |
|
I can confirm after this latest laptop crash, that if one (or more of the data files) become corrupted, SC will hang trying to read this corrupted file and in the process, triple the number of data files in the data directory with empty files by creating data files for each year 00-99 that are not currently in the directory.
|
| [2024-09-04 16:53:24] |
| Sierra_Chart Engineering - Posts: 17145 |
|
This is not making sense to us. What files are getting corrupted. What specific files and how is this then causing the problem. It is not making any sense to us. This is not something that should ever be happening, unless, the number of days to load in the chart is being changed. Or perhaps there is a problem with the month and year in a symbol, possibly. We do not agree with your conclusions. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2024-09-04 16:54:23
|
| [2024-09-04 17:04:22] |
| allons_trading - Posts: 30 |
|
In this last instance, the NQU24_FUT_CME.scid file could not be opened by SC. SC would just hang trying to load the chartbook. When I opened the data directory, the number of files went from 333 to 999. For all NQ contract data files, there were now NQH00-99, NQM00-99, NQU00-99, NQZ00-99 files now in the directory, with empty .dly and .scid for the years that didn't already exist. This instance of SC connects to Stage5/Advantage for Denali/Teton services with the above future symbol name format for both ES and NQ. I've seen this happen with both ES and NQ symbol data files whenever one or both had a data file that could not be fully read by SC.
|
To post a message in this thread, you need to log in with your Sierra Chart account: