Support Board
Date/Time: Sun, 22 Dec 2024 13:26:59 +0000
[Locked] - Rollover problem
View Count: 2098
| [2015-04-21 20:23:28] |
| User35326 - Posts: 57 |
|
SC: When I did a rollover from K5 to M5 contract for the CL, I have a gap in the chart from 04/02/15 to 04/16/15, inclusive. I had this same thing happen the last rollover. What box do I need to check or uncheck to correct this? I have the same gap no matter what time frame chart I look at that is intraday. I think I just figured out the answer. I changed the rollover rule to date based instead of volume based and it worked. Thanks. Date Time Of Last Edit: 2015-04-21 20:34:23
|
| [2015-04-21 23:20:51] |
| |
|
Could you follow the instructions here to re-download the data for the May contract: http://www.sierrachart.com/index.php?page=doc/doc_ContinuousFuturesContractCharts.html#ReDownloading Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2015-04-21 23:21:14
|
| [2015-04-22 15:31:15] |
| bekitz3 - Posts: 83 |
|
I noticed the same issue on my end too. I followed the instructions (which is how I handled data reloads anyways) but the problem persists. As User 35326 mentioned, switching to date-based back-adjust [CB] instead of volume-based back-adjust [CBV] does resolve the issue, but when you look back 2-3 months and beyond, the swing hi/lo points are not the same between the two continuous chart types. A couple of examples using Daily charts: Daily [CB]: 6/25/2014 swing HI = 106.39 1/29/2015 swing LO = 47.88 Daily [CBV]: 6/25/2014 swing HI = 107.96 1/29/2015 swing LO = 47.97 Not sure which ones are correct? Obviously the farther back you go, the more discrepancy in price points. This should be fixed one way or the other. I also double-checked this with a strictly intra-day (8 hr.) chart going back the same time period and the data points are as mentioned above. Also, when highlighting the rollover points as suggested in the instructions (Chart >> Show Rollover Dates), there are a few months (in a 12 mo period) not being displayed with a vertical line in the [CBV] chart. All lines are displayed in the [CB] chart. |
| [2015-04-22 17:21:05] |
| User35326 - Posts: 57 |
|
Good Points, Betitz3. SC - can you explain this and perhaps provide a fix for it on your software? Thanks. |
| [2015-04-22 17:55:38] |
| |
|
The basic problem with most everything being said in the thread is that we cannot duplicate the problem and it does not make sense that this problem would exist when the data is re-downloaded because these kinds of issues are due to missing data and once the data is re-downloaded, there should no longer be missing data so the problem should no longer exist. Here is a continuous contract chart using back adjustments and using volume rollover for CL: http://www.sierrachart.com/image.php?l=142972518153.png Once again, follow through the instructions here: https://www.sierrachart.com/index.php?page=doc/doc_ContinuousFuturesContractCharts.html#ReDownloading It should not matter, but update to the latest prerelease as well. If there still is a problem, let us know and we will will give you instructions on what to do to give us to Message Log. But please do not post the message log until you follow our specific instructions so we are not overwhelmed with information. Also, when highlighting the rollover points as suggested in the instructions (Chart >> Show Rollover Dates), there are a few months (in a 12 mo period) not being displayed with a vertical line in the [CBV] chart. All lines are displayed in the [CB] chart. This has been corrected in the latest prerelease.And of course you will have differences in the charts when you switch between date based rollover and volume-based rollover. That is to be expected. There is nothing unusual there. And for a commodity contract like CL, you will need to use the volume-based rollover for accuracy. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2015-04-22 18:00:14
|
| [2015-04-24 15:40:38] |
| bekitz3 - Posts: 83 |
|
you will need to use the volume-based rollover for accuracy.
I've been using [CBV] all the time because I suspected it was the most accurate, but the recent data holes with that continuous chart type and the comments from the other user reporting that [CB] did not have the same missing data peaked my interest in that chart type.It should not matter, but update to the latest prerelease as well.
Actually, updated to 1249 and the data holes are fixed.But unfortunately the following new problem exists: [CBV] http://www.sierrachart.com/image.php?l=1429888763948.png [CB] http://www.sierrachart.com/image.php?l=1429888865814.png Clearly those are not an accurate reflection of CL prices in the last year. I have already done Edit >> Delete All Data And Download for this symbol (per the instructions) AFTER upgrading to 1249. All of the .scid files for the individual components that constitute this 365 day continuous chart are timestamped with today's date. |
| [2015-04-24 16:14:45] |
| bekitz3 - Posts: 83 |
|
I actually had issues with some Daily charts not looking right either. So I deleted all the .dly files and re-loaded those and the intraday problem looks to be resolved: http://www.sierrachart.com/image.php?l=1429891672881.png But now my daily chart is missing bars: http://www.sierrachart.com/image.php?l=1429891776131.png The last bar you see on the chart is for today (4/24). The penultimate bar is 4/17. Suggestions?? |
| [2015-04-24 16:31:34] |
| bekitz3 - Posts: 83 |
|
Nevermind, I did a Edit >> Delete All Data And Download just for the current symbol, Jun-2015 CL, and the problem resolved http://www.sierrachart.com/image.php?l=1429893039477.png |
| [2015-04-24 16:37:32] |
| bekitz3 - Posts: 83 |
|
For whatever reason, when I deleted all of the .dly files associated with CL and then did Chart >> Reload and Recalculate the CL-201506-CME.dly file was 1 KB. The instructions for this procedure as described here http://www.sierrachart.com/index.php?page=doc/doc_ContinuousFuturesContractCharts.html#ReDownloading should allow the deleting file method to work but it didn't. Only after doing the option mentioned in post #8 was the file CL-201506-CME.dly a more realistic size - 91 KB. |
| [2015-04-24 16:59:22] |
| User35326 - Posts: 57 |
|
SC: I think your charts are the best; however, I do think this needs to be fixed by you. I think there has to be a better way than for the trader to be deleting files in an attempt to get the correct data. I am a trader. I am not a chart file deleter. Please find a fix for this so that the user(trader) can focus on trading on not on correcting rollover files. Thanks in advance. |
| [2015-04-24 18:07:13] |
| |
|
We will check on this.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2015-04-24 18:15:54] |
| |
|
I think your charts are the best; however, I do think this needs to be fixed by you. I think there has to be a better way than for the trader to be deleting files in an attempt to get the correct data. I am a trader. I am not a chart file deleter. Please find a fix for this so that the user(trader) can focus on trading on not on correcting rollover files. The recent issue they had is related to this: Notice: Version 1248 and Higher and Continuous Futures Contract Charts Also, historical data downloads can fail for different reasons including network communication problems and sometimes the data has to be re-downloaded and there is no solution to this other than the user manually initiating that.. So there is nothing for us to fix. The functioning of everything is as designed and it is as best and reliable as it can be. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2015-04-24 18:17:21
|
| [2015-04-24 18:25:00] |
| |
|
I think there has to be a better way than for the trader to be deleting files in an attempt to get the correct data. Also, in regards to this, it is very rare that we ever give out a link to the procedure to delete files which is an alternative method to clear the existing data. We only give a link to the procedure to re-download the data.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2015-04-24 21:36:42] |
| bekitz3 - Posts: 83 |
|
If this is in relation to the messages from the other user, please that is a specific issue they are encountering and it does not apply to you if you are not having that same problem.
If we look back to the beginnings of this thread, we were both having the same problem....missing data in the [CBV] continuous contract for CL. I was using version 1225-ish and was experiencing missing data from early april to around the 17th (about 2 weeks worth). There was also a missing day around Jan-16th. When toggling to a [CB] continuous chart, the data was present. This would seem to indicate that the raw data is present in the .scid file(s) since I'm assuming that's what the intraday chart is referencing regardless of whether [CBV] or [CB] is selected. This still occurred even after performing Edit >> Delete All Data And Download per the documentation.The recent issue they had is related to this:
This is only after I upgraded to 1249 even though this was suggested in post #5: Notice: Version 1248 and Higher and Continuous Futures Contract Charts It should not matter, but update to the latest prerelease as well.
It did matter. It resolved the missing data issue but resulted in the charts as seen in post #6. Only after noticing similar issues with the historical charts (daily) and re-downloading those did the intraday charts appear normal again.I can't find any documentation in either of these links: Notice: Version 1248 and Higher and Continuous Futures Contract Charts http://www.sierrachart.com/index.php?page=doc/doc_ContinuousFuturesContractCharts.html#ReDownloading that indicates that if a user has a chartbook with both intraday & historical charts present, they will encounter chart display errors across both chart types if both .scid files and .dly are not updated under the new revision. It was my understanding that intraday & historical charts operate independently of each other, hence the different file suffixes. One would think that after I upgraded the software and re-downloaded the .scid files per the instructions, at least the intraday charts would look good....but they were messed up as seen in post #6. Only through trial & error (which I realize is always part of the equation as a user) did I discover that re-downloading (or deleting in my case) the .dly files resolved the intraday chart issue. I should also note that I purposely held off on upgrading to the newer versions because I knew you guys were making changes as it related to historical charts and it seemed as though users were having some issues with those charts not displaying properly. So I thought I will wait until the details were hashed out. But in the meantime, I experienced the data issues in my intraday charts....hence my involvement in this thread to begin with. it is very rare that we ever give out a link to the procedure to delete files which is an alternative method to clear the existing data. We only give a link to the procedure to re-download the data
The link you are providing http://www.sierrachart.com/index.php?page=doc/doc_ContinuousFuturesContractCharts.html#ReDownloading takes users to the section "Re-Downloading Continuous Futures Contract Data". And directly below that is the section "Re-Downloading Continuous Futures Contract Data By Deleting Files" which is the method I chose. Why? Because I had 4-5 years of .dly files that needed to be updated. I made the executive decision to choose the 'delete method' because it seemed more efficient for me....rather than the 're-download method' (which you seem to refer to as the preferred or more robust option) which would require me to acknowledge 50+ dialog boxes to get the new data (since CL rolls each month). If this 'delete method' is not something you feel is as good as the 're-download method', then there should be some sort of disclaimer or footnote that indicates that this 'delete method' is not the desirable method and may result in undesirable results. Like in my case which resulted in the front-month .dly file being 1 KB in size. Which I can duplicate by the way. I can toggle between the 2 methods and get a 91 KB file with the 're-download method' and a 1 KB file with the 'delete method'. So clearly something's up there.I'm not sure what User 35326 did to resolve the issue on their end. Maybe they were using my trouble-shooting methods to resolve the issue too. If not, maybe they can post their solution. Either way, there is some merit to the concept of spending this much time trying to resolve what is supposed to be a basic built-in function of the software is borderline ridiculous. When you guys first offered this functionality, I purposely held off and utilized the manual methods of Edit >> Adjust Data and Edit >> Join Intraday Data. While more tedious and time consuming (in the long run one would think), it only takes one instance of this back & forth banter to erase all of the saved time one would hope to realize with the built-in continuous contract feature of the software. I have been a long time user of the software both as a broker provided and direct license user. I used the software well before the continuous contract feature was offered. In fact, I believe I posted a few years ago under the user 'bekitz' inquiring about you guys offering this continuous contract feature but that wasn't high on the priority list at the time. Fine I thought, which is why I became adept at the manual method. I mentioned that I had a similar issue with data missing in Jan-2015. But since no one else was reporting a similar issue I took it upon myself to find a solution because quite honestly I felt my issue would be met with resistance and attitude. I found that if I loaded up the contracts in question as non-continuous charts, then did a re-download of those individually, then a reload & recalculate on the continuous chart, the problem was resolved. This resulted in a larger .scid file size (at least double or more) for those individual component months of the continuous contract chart I was trying to fix....compared to the normal 1.0 MB-ish file sizes you get (for CL - 1 min data) when you re-download within the continuous chart. I didn't know why? I didn't care? It seemed to work so I moved on. Only with this other user now reporting the same type of issue did I decide to chime in and get involved. Generally, 2 different independent users reporting the same exact issue would warrant the possibility that SC is not handling the data for continuous chart types consistently. If we are the only 2 users in the history of SC to experience these issues, then I guess we either have the same mis-understanding of how the feature works, or have similar corruptive bugs within our systems. And I get that the user is responsible for managing files because that is one of the great features of SC.....the data residing on the user's system for more efficient chart loading, etc. As a long time user, I have been in touch with the issues that arise on the support board. I can honestly say that within the last 6 months, there definitely seems to be more attitude from the support end. It seems that anything beyond a couple of paragraphs is too lengthy and complex (which means most likely no one will get to this portion of the post). If you're growing, that's great....but if this growth is causing you to feel overwhelmed.....then hire more support staff. Cuz eventually this will catch up to you. I like the software very much and realize issues pop up for everyone now & then and that everyone thinks their problem is the most important problem in the world. I don't consider myself to be a novice user and I don't think I've bothered you that much when you consider the # of posts I've posted relative to the amount of time I've used the software. This is just a general observation on my part...take it for what it's worth. |
| [2015-04-25 00:43:37] |
| |
|
If we look back to the beginnings of this thread, we were both having the same problem.... Our comments in post 12 were related to post 6 of yours. The problem you had there, the other user could not have had at all and was unrelated because they are using TransAct and you are using TT. The problem you had is related to price multipliers which does not affect TransAct users for most of the contracts. We apologize for the difficulties you had. These problems were unexpected and relates to internal changes to use a consistent symbol between Historical and Intraday charts. For some trading services which use price multipliers there is this unexpected side effect which we had not considered. But this is simply a one-off problem that will not continue once the data is re-downloaded. but if this growth is causing you to feel overwhelmed.....then hire more support staff. We are able to handle support. It is a policy of this business that only experienced developers are providing support. It is not possible for us to hire "support staff". And will only lead to a deterioration in support and greater operational frustrations.You also need to understand the Continuous Futures Contract functionality is very complex. After two years of experience with it, it is not possible for it to work 100% perfectly all the time without some manual user intervention of re-downloading data when a problem occurs. This is unavoidable. From what we understand, from the first post, the information we gave the user, resolved the problem and it was over and done with. That is usually how it works when there is a problem. There are no outstanding problems for us to fix with this functionality that we are aware of and it does work properly as designed. We only identified one problem in the last week when performing a manual re-download of the continuous contract data, where one day earlier than the contract beginning date was not being downloaded when it should be in case that data is needed due to the actual rollover date. It is not possible to look at problems reports on this board in regards this functionality and make any conclusions with it and claim that two problems are the same and there is something that needs to be fixed. Each case has to be analyzed carefully to determine what the source of the problem is. Although the first general step is to re-download the affected contract months before we do any further analysis. Problem reports with continuous contracts are common. It is sophisticated functionality, and there are numerous symbols and numerous trading services and numerous settings. It is inevitable that problems will happen and we will help users with them. We will respond to your other comments as soon as we can. Update: It was my understanding that intraday & historical charts operate independently of each other, hence the different file suffixes. One would think that after I upgraded the software and re-downloaded the .scid files per the instructions, at least the intraday charts would look good....but they were messed up as seen in post #6. There is a dependency upon the two different data files when using volume-based rollovers in Intraday charts.Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2015-04-25 00:53:06
|
| [2015-04-26 05:55:58] |
| bekitz3 - Posts: 83 |
|
*(1)* Our comments in post 12 were related to post 6 of yours. The problem you had there, the other user could not have had at all and was unrelated because they are using TransAct and you are using TT
Well I certainly cannot dispute this. Perhaps the user was simply expressing empathy as a fellow trader in post #10 after seeing all the steps I was taking to resolve the issue on my end. *(2)* From what we understand, from the first post, the information we gave the user, resolved the problem and it was over and done with. That is usually how it works when there is a problem.
The first post was the initial communication of the issue at hand!!! They couldn't have possibly executed your suggestion since that came in post #2. That doesn't mean they didn't coincidentally re-download data anyways based on Help/FAQ's or other documentation on your website or pure trial & error, but not due to your suggestion.....unless they are a prophet. They simply said this: I think I just figured out the answer. I changed the rollover rule to date based instead of volume based and it worked.
in post #1 before you even offered any advice. That doesn't seem to imply that a data re-download was their solution. After that, I don't see any other evidence that the problem was solved on their end other than this switch to a [CB] chart....which you have communicated is not the most accurate choice for CL. If they did perform a re-download and it worked for them...great. It did not work for me. Thus, a simple illustration where 2 anonymous users experience the same initial problem......then execute the 1st step suggested by support......and experience vastly different paths to get to the final solution....normal looking charts. This could be a plausible explanation for what appears to be shared frustration as expressed in posts #4 and #10 (and it should be noted post #4 is their first response after your suggestion)This still does not explain how when switching to [CB], the missing 2 weeks suddenly appear, and when switching back to [CBV] they disappear again. This tells me that the raw data does exist within the .scid and that the software is not accurately referencing this data based on a user selection within the software. Which would seem to indicate that re-downloading data that already has all of the information needed would not be a very successful solution to the problem. *(3)* You also need to understand the Continuous Futures Contract functionality is very complex. After two years of experience with it, it is not possible for it to work 100% perfectly all the time without some manual user intervention of re-downloading data when a problem occurs.
I can tell you from a user's perspective, it is not perceived to be complex. As a non-developer user, it doesn't mean that I'm correct in that perception though. But I can say from using other platforms that offer this functionality (Tradestation, NinjaTrader, MultiCharts, etc) I haven't noticed this kind of behavior. Maybe this is because they choose to handle this concept differently and that is why they are more successful when it pertains to this functionality of trading software.The one observation I notice between these 2 approaches is that the others tend to treat this continuous contract concept with a segregated symbol identifier or separate file. How they get there and do this....I wouldn't be able to tell you. It's possibly similar to the concept I mentioned earlier where when I did the continuous contract manually (adjusting & joining data), I was in essence dealing with one file with all of the individual contract data appended appropriately within this file and then assigned the front month identifier/designation. Then upon any future changes to data feeds, price multipliers, unforeseen events by developers, etc......the software would not have to break down or cross-compare all of the individual component contracts to determine the rollover points. But I already know you are not interested in this approach and have documented this on your site. You have your reasons to stick with the current method and keep all of the individual components of the continuous contract isolated from one another. Which leads to this suggestion: Perhaps you could consider the notion that when a user requests a re-download from within a continuous contract chart, the software will automatically treat these individual contract components as if they were being requested within a non-continuous contract chart. I referred to this observation in my last post: If I re-download on a continuous chart, the file sizes for the .scid's are smaller than if I individually re-download each of these contracts on a non-continuous chart. So there is a truncation of data taking place which seems like it would be more prone to potential gaps & holes when consolidating this data for continuous contract chart display purposes. Maybe the re-download event should aim to acquire all of the data for each .scid in the continuous chart, keep these .scid's intact, then the software can do the volume analysis to determine rollover points and then create new, separate truncated files (.scidc or .idc where the last c = continuous) for each contract and use those to build the continuous chart . It seems like the reverse is happening where the software thinks it already knows the rollover point (maybe from a previous user usage of this feature), and when requesting new data, it is discarding data that may be needed to result in more accurate/consistent performance of this continuous contract feature. *(4)* There is a dependency upon the two different data files when using volume-based rollovers in Intraday charts.
This is not intuitive. This should be noted inside the continuous contract documentation then. This would be analogous to upgrading MS Word from 2007 to 2013, and then finding out all of my .ppt or .xls files are jacked up. |
| [2015-04-26 08:41:46] |
| |
|
This thread is now closed. Please do not start a new one as it is not welcome and will provide no useful information. There will be no response to the prior post. Many of the comments are based on a misunderstanding of how the Continuous Contract Chart feature works and some of the information is not making sense to us. The type of information which is given is not really helpful to us. If there is a problem, communicate the particular problem that you have without going into excessive amount of details and conclusions which are not possible to determine unless you know how the code works. And then we will look into it. We always will consider ways to improve the feature. Also, it is particularly difficult for us when we have two different issues being presented in the same thread and 2 users commenting on each other, when the issues are unrelated. While the original problem may have been the same, the subsequent problem was completely unrelated and was not a problem which required fixing by us. There was one minor documentation update that was needed due to some recent changes and has been done. Intraday Continuous Futures Contract charts which use back adjustments or volume-based rollovers have a dependency on the historical daily data and when re-downloading data in those Intraday charts, there is also a re-download of the historical daily data and that data must be complete in order for the rollovers and back adjustments to be correct. And there were some additional comments we wanted to make in our prior post in response to the post before that, we will try to get to that when we can. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2015-04-27 01:10:22
|
To post a message in this thread, you need to log in with your Sierra Chart account: