Support Board
Date/Time: Sun, 24 Nov 2024 16:33:18 +0000
Post From: Allow Disabling of Compression or Do It in Another Thread
| [2013-11-18 04:36:46] |
| bfalk - Posts: 33 |
|
I guess I was wrong about it being compression, I disabled compression globally on my system and got no performance gain. That then brings me to the question of what is so slow in the Sierra Chart historical data fetch process. The only issue I saw was that SC writes each record one by one (40 bytes each) which has a huge overhead of constantly calling into WriteFile(). SC should at least buffer 1MB worth of data internally and write it to the disk in chunks. The other problem I've noticed is that the amount of entries written to disk are twice that of the amount of entries actually downloaded, I'm assuming this takes place in the *.tmp.scid to *.scid copy. Here's my fetch: HD Request # 2 - Downloading Intraday chart data for @CD# to the file @CD#.temp.scid. Service: dtn | 2013-11-17 23:16:38
Compressing file: M:\SierraChart\Data\@CD#.temp.scid. This may take a while... | 2013-11-17 23:16:38 HD Request # 2 - Requesting Minute data beginning at 1986-07-04 00:16:38 US Eastern Time. | 2013-11-17 23:16:38 IQ Feed: Received Server Connected message from Level 2 port. | 2013-11-17 23:16:38 IQ Feed: Level 2 port indicates market is opened. | 2013-11-17 23:16:38 HD Request # 2 - Receiving Intraday Minute data for @CD# | 2013-11-17 23:16:40 HD Request # 2 - Receiving Intraday data for @CD# starting at 2006-04-21 03:35:00 | 2013-11-17 23:16:40 HD Request # 2 - Timestamp of first Intraday data file record written: 2006-04-21 03:35:00 | 2013-11-17 23:16:40 HD Request # 2 - Received 2111150 records from 2006-04-21 03:35:00 to 2013-11-17 23:16:00 (7.6 years) and wrote 2111150 records for @CD# | 2013-11-17 23:17:22 HD Request # 2 - Intraday download COMPLETE for @CD# | 2013-11-17 23:17:22 HD Request # 3 - Downloading Intraday chart data for @CD# to the file @CD#.temp.scid. Service: dtn | 2013-11-17 23:17:22 HD Request # 3 - Requesting Tick data beginning at 1986-07-04 00:17:22 US Eastern Time. | 2013-11-17 23:17:22 HD Request # 3 - Receiving historical Intraday Tick data for @CD# | 2013-11-17 23:17:23 HD Request # 3 - Receiving Intraday data for @CD# starting at 2013-05-22 00:03:02 | 2013-11-17 23:17:23 HD Request # 3 - Timestamp of first Intraday data file record written: 2013-05-22 00:03:02 | 2013-11-17 23:17:23 HD Request # 3 - Received 4103446 records from 2013-05-22 00:03:02 to 2013-11-17 23:16:49 (180.0 days) and wrote 4103446 records for @CD# | 2013-11-17 23:18:53 HD Request # 3 - Copying downloaded Intraday chart data from @CD#.temp.scid to @CD#.scid | 2013-11-17 23:18:53 Compressing file: M:\SierraChart\Data\@CD#.scid. This may take a while... | 2013-11-17 23:18:53 HD Request # 3 - Timestamp of first Intraday data file record written: 2006-04-21 03:35:00 | 2013-11-17 23:18:53 HD Request # 3 - Intraday download COMPLETE for @CD# | 2013-11-17 23:21:02 The fact that it took 4.5 minutes just to download @CD# is a little bit shocking, especially as I have about 15 different futures I follow and @CD# is probably one of the least active ones. My initial mention of putting it in a different thread still stands, as there is no reason that the file downloading, parsing, and saving should be running on the same thread as the main message loop. I hate when the GUI's of programs become unresponsive because they have a task that should be in the background in the foreground. -B |