Support Board
Date/Time: Fri, 22 Nov 2024 14:26:02 +0000
Downloading intraday data
View Count: 5294
| [2013-05-21 12:19:45] |
| User95208 - Posts: 26 |
|
Please look at this log for CHFJPY.ABBA....It received the missing data from 26-2-13 to 21-5-13 for 84.2 days and then tick data for 6.7 days from 26-2-13 to 4-3-13 for 6.7 days...but does not plot the data for these period on the chart....this log is produced when the SC was started but the chart was not opened...the chart was opened after the log was captured... HD Request # 19 - Downloading Intraday chart data for CHFJPY.ABBA to the file CHFJPY.ABBA.temp.scid. Service: dtn | 2013-05-21 12:02:34 Compressing file: C:\SierraChart\Data\CHFJPY.ABBA.temp.scid. This may take a while... | 2013-05-21 12:02:34 HD Request # 19 - Starting date and time: 2013-02-26 06:59:59 | 2013-05-21 12:02:34 HD Request # 19 - Requesting Minute data beginning at 2013-02-26 01:59:59 US Eastern Time. | 2013-05-21 12:02:34 HD Request # 19 - Receiving Intraday Minute data for CHFJPY.ABBA | 2013-05-21 12:02:35 HD Request # 19 - Receiving Intraday data for CHFJPY.ABBA starting at 2013-02-26 07:00:00 | 2013-05-21 12:02:35 HD Request # 19 - Timestamp of first Intraday data file record written: 2013-02-26 07:00:00 | 2013-05-21 12:02:35 HD Request # 19 - Received 86752 records from 2013-02-26 07:00:00 to 2013-05-21 12:02:00 (84.2 days) and wrote 86752 records for CHFJPY.ABBA | 2013-05-21 12:02:37 HD Request # 20 - Downloading Intraday chart data for CHFJPY.ABBA to the file CHFJPY.ABBA.temp.scid. Service: dtn | 2013-05-21 12:02:37 HD Request # 20 - Starting date and time: 2013-02-26 06:59:59 | 2013-05-21 12:02:37 HD Request # 20 - Requesting Tick data beginning at 2013-02-26 01:59:59 US Eastern Time. | 2013-05-21 12:02:37 HD Request # 20 - Receiving historical Intraday Tick data for CHFJPY.ABBA | 2013-05-21 12:02:38 HD Request # 20 - Receiving Intraday data for CHFJPY.ABBA starting at 2013-02-26 06:59:59 | 2013-05-21 12:02:38 HD Request # 20 - Timestamp of first Intraday data file record written: 2013-02-26 06:59:59 | 2013-05-21 12:02:38 HD Request # 20 - Error downloading historical Intraday data for CHFJPY.ABBA. The download timed out. | 2013-05-21 12:06:55 HD Request # 20 - Received 2793228 records from 2013-02-26 06:59:59 to 2013-03-04 22:46:55 (6.7 days) and wrote 2793228 records for CHFJPY.ABBA | 2013-05-21 12:06:55 HD Request # 20 - Copying downloaded Intraday chart data from CHFJPY.ABBA.temp.scid to CHFJPY.ABBA.scid | 2013-05-21 12:06:55 HD Request # 20 - Timestamp of first Intraday data file record written: 2013-02-26 06:59:59 | 2013-05-21 12:06:55 HD Request # 20 - Intraday download COMPLETE for CHFJPY.ABBA | 2013-05-21 12:08:11 Starting updates for: CHFJPY.ABBA | 2013-05-21 12:08:11 Real-time intraday chart data file updates started for CHFJPY.ABBA | 2013-05-21 12:08:11 http://www.sierrachart.com/image.php?l=1369138774541.png (I had full data...now blank in 2013) http://www.sierrachart.com/image.php?l=1369139005247.png http://www.sierrachart.com/image.php?l=1369139234462.png (I had full data after a few downloads..the chart shows missing data in 2013)... Most of the instruments, where I had downloaded the data with a few retries per instrument shows a missing data for 2013 intraday.... Date Time Of Last Edit: 2013-05-21 12:29:22
|
| [2013-05-21 13:52:19] |
| User95208 - Posts: 26 |
|
Another one...the data on this image was there this morning...I am seeing missing data.... http://www.sierrachart.com/image.php?l=1369144289658.png |
| [2013-05-21 21:18:10] |
| |
|
1. All that we need was one Message Log. Not all of these images and multiple logs. This is not helpful to us and only creates more unnecessary work for us. 2. What we are clearly observing from the logs is 2 problems which have unclear causes. HD Request # 53 - Timestamp of first Intraday data file record written: 2013-01-23 08:59:59 | 2013-05-21 07:47:34 HD Request # 53 - Error downloading historical Intraday data for AUDCHF.ABBA. The download timed out. | 2013-05-21 07:52:08 HD Request # 53 - Received 3444737 records from 2013-01-23 08:59:59 to 2013-02-07 11:45:22 (15.1 days) and wrote 3444737 records for AUDCHF.ABBA | 2013-05-21 07:52:08 HD Request # 53 - Copying downloaded Intraday chart data from AUDCHF.ABBA.temp.scid to AUDCHF.ABBA.scid | 2013-05-21 07:52:08 HD Request # 53 - Timestamp of first Intraday data file record written: 2013-01-23 08:59:59 | 2013-05-21 07:52:08 HD Request # 53 - Intraday download COMPLETE for AUDCHF.ABBA | 2013-05-21 07:53:42 Real-time intraday chart data file updates started for AUDCHF.ABBA | 2013-05-21 07:53:42 one thing is apparently clear, and that is on your system the writing of data to a data file is extremely slow. This can be confirmed if you tell us that while historical data is being downloaded, if you notice that the interface of Sierra Chart is in a frozen state or intermittently becomes frozen. Usually this is the result of antivirus or anti-malware type software. In any case your file system is behaving very very slow. That is clear based upon what we are seeing in the log. We also notice a timeout while downloading historical tick data from IQ feed over a network socket. This could be due to the slow file processing. or a problem with the IQ Feed software. We recommend restarting your computer for this potential issue. The idea behind this is just a general reset. We have no doubt if you try another computer without any kind of anti-malware software on it, you would be unlikely to have a problem like this. You may also want to look at using SSD drives. 3. BTW, I have provided the log files directly from the system a few times (through send for analysis)..... Unless we were told this, we would never be aware of it because we do not look at those log files unless we know someone is sending them that way.Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2013-05-21 21:20:58
|
| [2013-05-22 09:22:33] |
| User95208 - Posts: 26 |
|
Thanks. I understand your observation regarding the writing speed. It is a SATA II HDD on a Xeon x5450 with 20GB ram. When the data is downloading, it does not let us use the chart, obviously, until the data is downloaded and written. If that's is a frozen state, yes it in that state when the download of a large file is taking place. Otherwise, I do not see an intermittent frozen state, although the SC has about 2-3 dozen intraday list. There are two solutions: I run a RAM disk, which can be used to temporarily write the data but we have a capacity problem. My RAM size is 6GB, which is not enough. The other choice is investing in an SSD (like Samsung 840 Pro), which I am seriously considering. It will definitely make a massive difference but with the SC continuously downloading and writing the data, 24 x 5.5, on a few dozen instruments, I will very quickly reach the write cycle limit. If the instrument lists go to about 100s, the SSD won't last long. Would it? |
| [2013-05-22 10:55:03] |
| |
|
You do not have any hardware issues. This is clearly a software problem where there is something on your System slowing down the data writing process. We have never ever known an issue like this, which is uncommon, to be the result of Hardware itself. It is due to anti-malware /Antivirus software generally. When the data is downloading, the user interface should be responsive. In the case of when downloading tick by tick data from IQ Feed, you may get an occasional hesitation of the user interface. But not much. That will be resolved when we use a background thread to write the data for IQ Feed. We do use a separate thread with our own services. We use Intel SSD's on all of our servers and development systems and we have never had any issues with reaching a write limit with them. Even with intense writing. If you can try another computer that is completely clean of antivirus type software. That's what we recommend trying. What you are experiencing is not normal. Have a look at this log and the timestamps at the end to see the speed at which the data was downloaded on our system: HD Request # 5 - Downloading Intraday chart data for EURUSD.ABBA to the file EURUSD.ABBA.temp.scid. Service: dtn | 2013-05-22 10:50:47
HD Request # 5 - Requesting Minute data beginning at 2012-04-17 06:50:47 US Eastern Time. | 2013-05-22 10:50:47 Symbol Error: EURUSD.ABBA is unknown, unavailable, or improperly formatted. | 2013-05-22 10:50:47 HD Request # 5 - Receiving Intraday Minute data for EURUSD.ABBA | 2013-05-22 10:50:48 HD Request # 5 - Receiving Intraday data for EURUSD.ABBA starting at 2012-04-17 10:51:00 | 2013-05-22 10:50:48 HD Request # 5 - Timestamp of first Intraday data file record written: 2012-04-17 10:51:00 | 2013-05-22 10:50:48 HD Request # 5 - Received 410689 records from 2012-04-17 10:51:00 to 2013-05-22 10:50:00 (400.0 days) and wrote 410689 records for EURUSD.ABBA | 2013-05-22 10:50:58 HD Request # 6 - Downloading Intraday chart data for EURUSD.ABBA to the file EURUSD.ABBA.temp.scid. Service: dtn | 2013-05-22 10:50:58 HD Request # 6 - Requesting Tick data beginning at 2012-04-17 06:50:58 US Eastern Time. | 2013-05-22 10:50:58 HD Request # 6 - Receiving historical Intraday Tick data for EURUSD.ABBA | 2013-05-22 10:50:59 HD Request # 6 - Receiving Intraday data for EURUSD.ABBA starting at 2013-01-06 22:00:01 | 2013-05-22 10:50:59 HD Request # 6 - Timestamp of first Intraday data file record written: 2013-01-06 22:00:01 | 2013-05-22 10:50:59 HD Request # 6 - Received 20298766 records from 2013-01-06 22:00:01 to 2013-05-22 10:54:09 (135.5 days) and wrote 20298766 records for EURUSD.ABBA | 2013-05-22 10:54:22 HD Request # 6 - Copying downloaded Intraday chart data from EURUSD.ABBA.temp.scid to EURUSD.ABBA.scid | 2013-05-22 10:54:22 HD Request # 6 - Intraday download COMPLETE for EURUSD.ABBA | 2013-05-22 10:54:22 Real-time intraday chart data file updates started for EURUSD.ABBA | 2013-05-22 10:54:22 Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2013-05-22 10:56:21
|
| [2013-05-22 11:03:41] |
| |
|
Have a look at this log at the end from a test we did on our system: HD Request # 8 - Received 3220833 records from 2013-05-09 10:59:59 to 2013-05-22 10:59:16 (13.0 days) and wrote 3220833 records for EURUSD.ABBA | 2013-05-22 10:59:31
HD Request # 8 - Copying downloaded Intraday chart data from EURUSD.ABBA.temp.scid to EURUSD.ABBA.scid | 2013-05-22 10:59:31 HD Request # 8 - Timestamp of first Intraday data file record written: 2013-05-09 10:59:59 | 2013-05-22 10:59:31 HD Request # 8 - Intraday download COMPLETE for EURUSD.ABBA | 2013-05-22 10:59:32 Real-time intraday chart data file updates started for EURUSD.ABBA | 2013-05-22 10:59:32 It takes exactly 1 second to write 3.2 million records of data to the final output file. In your log, this takes nearly 2 minutes: HD Request # 63 - Received 3662160 records from 2013-01-15 17:59:59 to 2013-01-24 09:40:25 (8.7 days) and wrote 3662160 records for CHFJPY.ABBA | 2013-05-21 10:00:07
HD Request # 63 - Copying downloaded Intraday chart data from CHFJPY.ABBA.temp.scid to CHFJPY.ABBA.scid | 2013-05-21 10:00:07 HD Request # 63 - Timestamp of first Intraday data file record written: 2013-01-15 17:59:59 | 2013-05-21 10:00:07 HD Request # 63 - Intraday download COMPLETE for CHFJPY.ABBA | 2013-05-21 10:01:44 Real-time intraday chart data file updates started for CHFJPY.ABBA | 2013-05-21 10:01:44 You have something on your system causing this problem. That's the issue here. It's not the hardware itself. Your hardware is plenty fast enough. we have plenty of experience with this. There is something running on your system interfering with the process. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2013-05-22 11:30:04
|
| [2013-05-22 12:44:20] |
| User95208 - Posts: 26 |
|
Thanks once again for your observations. I never experienced such issues with other charting applications; How does this look now, which is still not a great improvement. I have tweaked a lot of things on the AV/Malware. Do you need a dedicated IQFeed running just SC or it is fine concurrently with amibroker and/or ninja trader? HD Request # 48 - Receiving Intraday Minute data for XAU.X | 2013-05-22 13:33:07 HD Request # 48 - Receiving Intraday data for XAU.X starting at 2007-04-30 17:52:00 | 2013-05-22 13:33:07 HD Request # 48 - Timestamp of first Intraday data file record written: 2007-04-30 17:52:00 | 2013-05-22 13:33:07 HD Request # 48 - Received 601406 records from 2007-04-30 17:52:00 to 2013-05-21 21:03:00 (6.1 years) and wrote 601406 records for XAU.X | 2013-05-22 13:33:24 HD Request # 49 - Downloading Intraday chart data for XAU.X to the file XAU.X.temp.scid. Service: dtn | 2013-05-22 13:33:24 HD Request # 49 - Requesting Tick data beginning at 1999-09-13 08:33:24 US Eastern Time. | 2013-05-22 13:33:24 HD Request # 49 - Receiving historical Intraday Tick data for XAU.X | 2013-05-22 13:33:25 HD Request # 49 - Receiving Intraday data for XAU.X starting at 2013-01-07 14:30:15 | 2013-05-22 13:33:25 HD Request # 49 - Timestamp of first Intraday data file record written: 2013-01-07 14:30:15 | 2013-05-22 13:33:25 HD Request # 49 - Received 2227913 records from 2013-01-07 14:30:15 to 2013-05-21 21:03:14 (134.2 days) and wrote 2227913 records for XAU.X | 2013-05-22 13:34:28 HD Request # 49 - Copying downloaded Intraday chart data from XAU.X.temp.scid to XAU.X.scid | 2013-05-22 13:34:28 Compressing file: C:\SierraChart\Data\XAU.X.scid. This may take a while... | 2013-05-22 13:34:28 HD Request # 49 - Intraday download COMPLETE for XAU.X | 2013-05-22 13:34:38 Starting updates for: XAU.X | 2013-05-22 13:34:38 Real-time intraday chart data file updates started for XAU.X | 2013-05-22 13:34:38 Date Time Of Last Edit: 2013-05-22 12:45:04
|
| [2013-05-22 20:28:51] |
| |
|
IQ Feed can be used by other programs at the same time. This is no problem. This log certainly looks better. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2013-05-22 20:29:04
|
| [2013-05-23 11:40:10] |
| User95208 - Posts: 26 |
|
I am not downloading more data at this juncture. I have paused it to look at your MP and Footprint charts. My first impression is good. I want to send something (image) that cannot be posted on the forum. Please advise your email address. Re: SSD; I assume an MLC SSD is enough. The SLC SSDs are very expensive. Thanks |
| [2013-05-23 16:42:54] |
| |
|
Yes, MLC is fine. You can use support _AT_ Sierra Chart.com Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
To post a message in this thread, you need to log in with your Sierra Chart account: