Support Board
Date/Time: Mon, 25 Nov 2024 20:37:19 +0000
Strange behaviour when feeding SCID using Intraday Storage Time Unit greater than 1 tick
View Count: 482
| [2024-02-01 19:24:15] |
| weav - Posts: 30 |
|
Hi I am feeding scid files via a python program. The Intraday storage time unit is 1 minute on my instance of SC. I write the first record after the minute barrier has passed and then I update that same record with subsequent ticks until the next minute barrier when I write a new record again. On the updates I change only the high (if required) the low (if required), the last, the number of trades, and the volume. I don't touch the open or timestamp, the remain they same. When I export the underlying data it is normal for 1 minute storage time unit, e.g. 2024/2/1, 08:38:00, 14.13, 14.13, 14.12, 14.12, 6, 6, 0, 0 2024/2/1, 08:39:00, 14.11, 14.11, 14.11, 14.11, 6, 6, 0, 0 2024/2/1, 08:40:00, 14.11, 14.11, 14.10, 14.10, 7, 7, 0, 0 2024/2/1, 08:41:00, 14.10, 14.12, 14.10, 14.12, 7, 7, 0, 0 2024/2/1, 08:42:00, 14.12, 14.12, 14.11, 14.11, 8, 8, 0, 0 2024/2/1, 08:43:00, 14.11, 14.11, 14.11, 14.11, 5, 5, 0, 0 2024/2/1, 08:44:00, 14.11, 14.11, 14.10, 14.10, 5, 5, 0, 0 2024/2/1, 08:45:00, 14.10, 14.10, 14.09, 14.10, 8, 8, 0, 0 2024/2/1, 08:46:00, 14.10, 14.10, 14.09, 14.10, 4, 4, 0, 0 However if export BAR data to a text file I get this, note the extra 1 minute bar being added with prior timestamp after each new bar: 2024/2/1, 08:40:00, 14.11, 14.11, 14.10, 14.10, 7, 7, 0, 0 2024/1/31, 21:59:00, 14.35, 14.35, 14.35, 14.35, 0, 1, 0, 0 2024/2/1, 08:41:00, 14.10, 14.12, 14.10, 14.12, 7, 7, 0, 0 2024/1/31, 21:59:00, 14.35, 14.35, 14.35, 14.35, 0, 1, 0, 0 2024/2/1, 08:42:00, 14.12, 14.12, 14.11, 14.11, 8, 8, 0, 0 2024/1/31, 21:59:00, 14.35, 14.35, 14.35, 14.35, 0, 1, 0, 0 2024/2/1, 08:43:00, 14.11, 14.11, 14.11, 14.11, 5, 5, 0, 0 2024/1/31, 21:59:00, 14.35, 14.35, 14.35, 14.35, 0, 1, 0, 0 2024/2/1, 08:44:00, 14.11, 14.11, 14.10, 14.10, 5, 5, 0, 0 2024/1/31, 21:59:00, 14.35, 14.35, 14.35, 14.35, 0, 1, 0, 0 2024/2/1, 08:45:00, 14.10, 14.10, 14.09, 14.10, 8, 8, 0, 0 2024/1/31, 21:59:00, 14.35, 14.35, 14.35, 14.35, 0, 1, 0, 0 And this is how my bars print on the chart. One good 1 minute bar, which is immediately followed by another "out of order timestamp" 21:59:00 bar from the day before. But that is not how my intraday data is being stored in scid. My python program definitely isn't adding the 21:59:00 bar as you can see from the first intraday data text export. If I disconnect/reconnect the data feed then the spurious extra 1 minute bar with prior timestamp that gets added after every new bar has the timestamp and price of the last bar before reconnection. i.e. if I disconnected/reconnected at 08:30, instead of the extra 1 minute bar having a timestamp of 21:59:00 it will now be 08:30:00 and the last price will be the price at 08:30:00. Then this 08:30:00 bar will be added after every new bar that is added by my program. Its like SC is storing the last known price prior to connection and then not allowing that to be overwritten by my program. Important to note: If I take the 1 minute scid file, and my 1 minute bar writer python program, and move it to another install of SC which uses Intraday storage time unit of 1 Tick, then there is no problem, no extra bars with prior timestamp. Chart is normal and 1 minute bars are correctly displayed. (as per my intraday text file export) So, it's something to do with Intraday Storage Time Unit greater than 1 tick that is causing an extra bar (with timestamp / price of last bar on chart at time of data feed connection) to be added after every new bar written by my program. Somehow (I think) SC is storing that last 1 minute bar at time of connection, and then adding it on the chart after every new 1 min bar is added to scid by my python program. Also something else to note: If i reload the chart all the extra bars disappear. They only occur when SC is updating the bar chart in real time. Does this ring any bells with any of the SC engineering team? I would love to know what causes it, and to be able to use 1 minute storage time unit setting in SC with my externally written scid files. Maybe for intraday time storage unit of greater than 1 tick this will only work if I feed SC via DTC? I've attached a chart to show you how this looks on a chart |
 2024-02-01_19-14-16.png / V - Attached On 2024-02-01 19:19:19 UTC - Size: 47.74 KB - 63 views |
| [2024-02-01 21:35:59] |
| John - SC Support - Posts: 36278 |
|
Are you connected to a data source for which you would be getting data for VIX? If so, then it sounds like you are fighting with the data that is coming into the system from the data source with your code. You would need to be disconnected from the data source to ensure that you are not getting data into the same symbol while you are trying to write to the file.
For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing |
| [2024-02-02 09:01:25] |
| weav - Posts: 30 |
|
hmm, I did wonder that. I'm feeding VIX from an external source (and also VIX9D, VIN, VIF, VIX3M - all of them have the same issue). The SC instance I'm running (where I'm charting the VIX data) is a sub-instance of a master instance which is connected to Trading Evaluator Data/Trade Service. I thought maybe my sub instance is somehow picking up VIX snapshot data from somewhere but I don't explicitly subscribe to CBOE data through Sierra Chart. I subscribe to Denali only. The master instance has intraday storage time unit of 1 tick, and the sub-instance has intraday storage time unit of 1 minute. The master instance (1 tick) charts my externally-fed VIX data just fine (I feed the VIX data into both Data directories, master and sub, as VIX.scid. master being 1 tick granularity and sub being 1 minute granularity), it's only the sub-instance using storage time 1 minute that has this strange issue. If I change the sub-instance to 1 tick storage data as well then there's no problem, it all works as expected. so while it feels like I could be fighting with snapshot VIX data from the connected data feed, I can't think where this would be coming from as it should only be Denali data from the master instance. And because I only experience the issue in a 1 minute storage unit instance of SC, this leads me to believe it could be some kind of anomaly with the non-tick storage unit code in SC and externally fed intraday data Anyway you've given me food for thought. I am going to try a completely standalone clean install of SC (not a sub-instance), chart only my externally-fed VIX data, and play around with intraday storage time unit and the data service. |
| [2024-02-02 16:20:46] |
| John - SC Support - Posts: 36278 |
|
We do not have data for the CBOE VIX data, which is what confused us initially, but we were not sure what you were connected to, so that was our first thought. But, from what you describe, it sounds like you are experiencing the same issue, but only due to your particular setup. Keep in mind that Sub-Instances get their data from the Main Instance. So when you "feed" the main instance, the sub-instance will then pick up that data. In the end, it sounds like you just need to not send data to the sub-instance, but only to the main instance, and then the internal mechanisms of Sierra Chart will take care of getting that data to the sub-instance. For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing |
| [2024-02-07 10:23:27] |
| weav - Posts: 30 |
|
I've simplified the test environment. New standalone install of SC (not a sub instance). Not connected to any data feed. Intraday storage time unit = 1 minute. My external program feeding 1 minute records into VIX.scid. In the chart I still get the additional bar added by SC (I believe) after my program writes a new real time 1 minute record to scid. VIX.scid is correct with sequential data according to "Export Intraday Data to Txt File". But not correct when I export "Bar Data to Txt File" (SC adding the extra 09:18:00 bar?). I've marked up a picture with all details in the attachment. If I change Intraday storage time unit = 1 tick then chart displays real time data in VIX.scid correctly and sequentially - and if I "Export Bar Data to Txt File" similarly all rows are sequential and correct (i.e. no 09:18:00 row after each new real time bar). I know how busy you guys are I'm not expecting an investigation when your time could be spent better elsewhere. I realise it's a pretty esoteric use case of SC, its just frustrating me as I like to store scid data in my sub-instances to 1 minute to save disk space. Which works fine with Denali data but doesn't work when I feed my own data into scid directly. If it triggers any light bulbs great, if not no worries.
|
| 2024-02-07_09-30-38.png / V - Attached On 2024-02-07 09:51:40 UTC - Size: 189.66 KB - 73 views |
To post a message in this thread, you need to log in with your Sierra Chart account: