Support Board
Date/Time: Mon, 25 Nov 2024 23:48:37 +0000
[User Discussion] - Version 1047: Millisecond time stamping
View Count: 5556
| [2013-11-18 13:40:23] |
| Hendrixon - Posts: 130 |
|
In the Changes Log you wrote: "One major exchange like the CME, does not provide millisecond timestamps on trades. Therefore, it cannot be known exactly when a trade occurred within a second. Sierra Chart does not support estimating milliseconds in these cases. As is previously said, Sierra Chart uses milliseconds as a counter." "Which is another reason, why Sierra Chart does not support milliseconds from external data feeds which do not necessarily accurately represent when a trade occurred, but only represent sending/receiving time of a message containing a trade." In a post few weeks ago you said you will give users the option to choose either the internal counter or the "millisecond time stamping" from the external feeds. I can't find any option like that in 1047 and from the text above its not clear if it will happen. Please clarify. Thanks. |
| [2013-11-18 16:12:44] |
| |
|
After further consideration we are not going to support that. It is difficult enough to support milliseconds. This was very complex to implement because there are so many small minute technical details which had to be considered. Supporting external millisecond time stamping only defeats the purpose of the reason milliseconds are being used in Sierra Chart, and will be hard to support. We are not able to accommodate everyone. There is a very small interest in external milliseconds time stamping. Sierra Chart only supports milliseconds in the way that it is useful for internal functionality. We also take the position, the data vendors who hype the fact that they have milliseconds, are only trying to take advantage of people who are ignorant as to what that is all about. We are not saying you are ignorant about this, not at all, but we are saying that most people do not need milliseconds. Milliseconds have no effect upon the accuracy of charts in any way. They never have. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2013-11-27 12:52:49] |
| jesslinn - Posts: 108 |
|
Do I correctly understand from the engineering response that SC truncates the incoming timestamp to the second and then increments the milliseconds for each tick from a particular instrument that truncates to the same second? Presumably the difficulty of keeping the vendor timestamp is that even at millisecond precision there may be duplicate timestamps which you want to avoid. If you want unique timestamps, since you are using a double to represent time, why not increment the microseconds or even nanoseconds. If you did that you could probably have unique timestamps for every piece of information across all instruments. There may be a great deal of interest in supporting external timestamps for vendors that provide them from the exchanges. I certainly would be interested in that. Data vendors who provide exchange time stamps are attempting to support the growing interest in automated trading and multi-instrument strategies that may depend on when a trade happens relative to another trade from a different exchange. You are correct that most people do not need milliseconds but an ever growing number want them. Milliseconds can have a great effect on the accuracy of charts and always have. The biggest problem historically has been comparing bars from vendors who decide to put there own timestamp on a tick rather than keep the exchange timestamp. If a trade happens close to a bar boundary it can easily appear in one bar or the other depending on how the exchange, the aggregator, the data vendor, and the charting software handled the timestamps. Even more important is the effect that improper timestamps can have on multi-instrument strategies. This is a blurb from iqFeed's website: Millisecond timestamps DTN IQFeed’s new millisecond timestamps show software developers and traders the exact time of each trade - as provided by the exchanges — with single millisecond precision. While most of our top third-party software partners are working to add support for our millisecond timestamps, you can see it today with Multicharts (www.multicharts.com) software — our first partner to release a display for this new level of IQFeed data. I recommend that Sierra Charts jump on this bandwagon. Exchange timestamps are much improved from the old days and if any vendor supplies them, it would be nice to use them. |
| [2013-11-27 16:49:41] |
| |
|
Do I correctly understand from the engineering response that SC truncates the incoming timestamp to the second No, this is not really correct because the IQ Feed protocol version that we use does not provide milliseconds. Sierra Chart does use the time stamping from IQ Feed which uses an exchange timestamp.Milliseconds have exactly 0% effect upon the accuracy of the charts within Sierra Chart . Sierra Chart bars are 100% accurate and do not depend upon milliseconds. Also, the double precision numbers that Sierra Chart uses to hold timestamps, have no room for precision beyond milliseconds. The biggest problem historically has been comparing bars from vendors who decide to put there own timestamp on a tick rather than keep the exchange timestamp. show software developers and traders the exact time of each trade We take issue with this. One major exchange the CME, does not use millisecond timestamps for trades. If you put reliance upon milliseconds to actually know the time of the trade, you are not going to necessarily get an accurate determination of this.Sierra Chart does not support external millisecond timestamps and has no intention of doing so at this time. Very simply, there are practicality problems with implementing milliseconds, and very very very few users would have any genuine need for them, and they have 0% upon the accuracy of chart bars. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2013-11-27 17:04:30
|
| [2013-11-27 17:05:12] |
| TastyRisk - Posts: 119 |
|
Milliseconds have exactly 0% effect upon the accuracy of the charts within Sierra Chart . Sierra Chart bars are 100% accurate and do not depend upon milliseconds.
This is totally wrong and shows that you don't understand how customers are using your software. If we have 15 trades at various prices within 1 second on one instrument - and 12 trades trades at different prices, at the same second, on another instrument... and we then compare the difference (arbing the ES/NQ spread for example); we will find that gross calculation errors are produced due to to the lack of millisecond support. Perhaps millisecond support would even be too slow but seconds is an eternity in today's world. edit; I want to add that multiple trades happening quickly is a normal occurrence not a rarity. In fact; when the market is slow it can often be because there are no decent profit opportunities. Therefore; when trying to produce an accurate study on comparative data series we will find that our spreadsheets / studies are wrong more often than accurate... and ironically when they are accurate there are no trades worth doing. Date Time Of Last Edit: 2013-11-27 17:11:23
|
| [2013-11-27 17:11:49] |
| |
|
OK, but you are referring to your own custom charts. Although not really certain how you are doing the comparison to a level under 1 second because the minimum fixed timeframe per bar is 1 second. The accuracy we are referring to is for all the standard chart bars within Sierra Chart. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2013-11-27 17:21:15] |
| TastyRisk - Posts: 119 |
|
One major exchange the CME, does not use millisecond timestamps for trades. If you put reliance upon milliseconds to actually know the time of the trade, you are not going to necessarily get an accurate determination of this.
This is an bogus answer and frankly doesn't not hold water. You get information from the exchange with millisecond stamping. As you boast 0% packet loss on your CME feed then it is evident that there is little congestion on the switch fabric. Therefore the CME time-stamp would be an excellent substitute for a true trade-time-stamp. edit; I want to add that even if there was packet loss or delay due to blocked switch fabric; it's likely that the stamp would still be valid if they are produced before the client distribution layer... which they obviously are. Date Time Of Last Edit: 2013-11-27 17:27:16
|
| [2013-11-27 17:23:59] |
| TastyRisk - Posts: 119 |
|
OK, but you are referring to your own custom charts. Although not really certain how you are doing the comparison to a level under 1 second because the minimum fixed timeframe per bar is 1 second.
Yes I refer to standard charts that are inaccurate because of single second resolution. The comparison is not accurate... but I'd like it be. |
| [2013-11-27 17:41:59] |
| TastyRisk - Posts: 119 |
|
If you put reliance upon milliseconds to actually know the time of the trade, you are not going to necessarily get an accurate determination of this.
Yes, it's true that an exchange will prefer to match trades over reporting the same. But it's also true that most of the exchanges - and the CME is no exception - have put much effort into fast and accurate data systems. Do you have any facts about any possible lag between matching engine events and the reporting the same ? You suggest that it's not worth implementing millisecond support because the CME might lag so much !! Wow - those hedge funds that fight for every millisecond must really have blown there cash on some wasted infrastructure then. |
| [2013-11-27 17:54:51] |
| TastyRisk - Posts: 119 |
|
You can see in this chartbook, that when overlaying an ES tick chart onto another ES tick chart and studying the difference; that it's incorrect as often as correct. Surely millisecond support would make this chart correct most of the time ? Date Time Of Last Edit: 2013-11-27 17:55:22
|
 ES vs ES.cht - Attached On 2013-11-27 17:54:33 UTC - Size: 49.14 KB - 548 views |
| [2013-11-27 19:15:13] |
| ganz - Posts: 1048 |
|
TastyRisk Hey. Perhaps millisecond support would even be too slow but seconds is an eternity in today's world.
Wow - those hedge funds that fight for every millisecond must really have blown there cash on some wasted infrastructure then.
Are you kidding?SC users are not competitors to HFT nor Hedge Funds. That guys are using CPU Core <---> CPU cache optimisation and 10Gbit co-location. You will be a loser by default in that area with any charting soft becuase of full circle execution latency: data latency + order routing latency + OS scheduler latency + .... charting latency? :) Ticks just should be in the proper order. That's all we really need. Date Time Of Last Edit: 2013-11-27 19:16:56
|
| [2013-11-27 20:08:17] |
| jesslinn - Posts: 108 |
|
I was going off the 8 byte time that appears in the SCID files, which has enough room for 300 years of nanoseconds, when I suggested keeping the exchange milliseconds and incrementing in microseconds, but you may use something besides 8 bytes internally. Certainly there are issues and cost in any case. Your mechanism for adding a millisecond as a sequence number is better than nothing. At least it keeps time marching forward which can be nice. The issue really is trying to maintain order between ticks for different instruments. Wouldn't it be nice, for example, to be able to notice that every time a 2000 share or larger ask is put up on ARCA that a 100 share up-bid on ISLAND appears 10 msec later. Detecting such automated strategies is interesting to some of us and is not possible without millisecond time stamps. Exchanges are improving their timestamps and some of your competitors are responding. We are just letting you know that some or your customers would find value in a solution. Just because we are not competing with HFTs does not mean that it is not useful to monitor the very delays that ganz mentioned. Wouldn't you like to know if you get your ticks 600 msec faster from one vendor than another or even more important, that a particular tick is greatly delayed from the average from a given vendor. This is not possible without keeping millisecond timestamps since all of the delays normally add to less than a second. |
| [2013-11-27 20:24:50] |
| ganz - Posts: 1048 |
|
jesslinn Just because we are not competing with HFTs does not mean that it is not useful to monitor the very delays that ganz mentioned. Wouldn't you like to know if you get your ticks 600 msec faster from one vendor than another or even more important, that a particular tick is greatly delayed from the average from a given vendor. This is not possible without keeping millisecond timestamps since all of the delays normally add to less than a second. The bottom line is you need to be very fast to use ms. But you are not. You can't operate using sub-seconds timeframe in a proper manner if you are not being co-located or using charts. The sub-second timeframe is another kind of business: other rules, requirements, speeds, TCO and so on. So you you will see a lot and you've never able to use it as it should be. This is just the same as to be a left-side-chart analytic. One could talk or could dream but unable to make any money there. Date Time Of Last Edit: 2013-11-27 20:34:09
|
| [2013-11-28 13:15:03] |
| TastyRisk - Posts: 119 |
|
ganz, quote: Ticks just should be in the proper order. That's all we really need.
If that's all you need then fine but please don't try to assert what I and other users need. I agree with you 100% that trying to compete with HFT is impossible using retail software and data feeds. We are not trying to beat the HFTs to market. What we're all trying to do is analyse past and current data in order to make decisions on the longer time frame. Basically that's the whole point of SierraChart software. However; the problem is that, without high resolution time support, sierra chart can't be used to perform accurate calculations on tick data. The problem is clearly shown in the attached image. Chart#1 is ESZ3 - 1 Tick. Chart#2 is ESZ3 - 1 Tick. (identical) I then overlay chart#1 onto chart#2 and perform a difference study... In my mind there should be no difference. It's clear that errors are being produced due to the lack of high resolution time support. |
 ESvsES.png / V - Attached On 2013-11-28 13:14:58 UTC - Size: 115.17 KB - 565 views |
| [2013-11-28 13:41:08] |
| ganz - Posts: 1048 |
|
TastyRisk If that's all you need then fine but please don't try to assert what I and other users need. Yeah, sure. That was just my opinion and nothing else/more. there should be no difference. Looks like the study has no support for a ms part of a timestamp. The same way as the crosshair tool jumps to a very first tick of a given second on a 1-tick chart. |
| [2013-11-28 13:43:42] |
| TastyRisk - Posts: 119 |
|
The study has no support for ms? The whole application has no support for ms timestamp ! There is only support for incrementing a counter. Date Time Of Last Edit: 2013-11-28 13:48:15
|
| [2013-11-28 13:49:14] |
| ganz - Posts: 1048 |
|
TastyRisk I meant a-la ms-part: internal-counter part of a timestamp. Date Time Of Last Edit: 2013-11-28 13:53:50
|
| [2013-11-28 13:50:50] |
| ganz - Posts: 1048 |
|
TastyRisk OK. Looks like the study has no support for "incrementing a counter" part. The charts are accurate and the same. I've checked one case. Date Time Of Last Edit: 2013-11-28 13:51:44
|
| [2013-11-28 14:04:13] |
| ganz - Posts: 1048 |
|
just a reminder to use a-la ms-counter FYI: ACSIL and Millisecond Timestamps Question | Post: 8109 |
| [2013-11-28 14:19:23] |
| Hendrixon - Posts: 130 |
|
ganz my friend I hope you know I respect you :-) There are MANY ways retail traders can use sub second time stamps to their advantage, OTHER ways than just going head to head with major HFT firms. Cool, you don't think on other ways, but it doesn't mean there aren't… It's not like adding ms support will make SC worse. Heck, let's ask SC to eliminate 1 second granularity and round it all too full minutes since almost NO ONE really uses seconds… oh and maybe add a counter for sub minute IDs. After SC promised full [REAL] millisecond time frame support, in anticipation for that I've tested things on Multicharts demo and in 3 weeks I found that several concepts I wanted to implement in my SC studies are spot on. Fabric Millisecond execution? Bars accuracy? None of my stuff relates to that, it's almost all doing checks on data integrity, latency and "freshness" (from lack of other word) at the moment of order entry, to KEEP ME SAFE from exchange/datafeed/network issues. None of that can be done with "a counter". |
| [2013-11-28 14:24:47] |
| ganz - Posts: 1048 |
|
Hendrixon my friend I hope you know I respect you too :-) real trades may occur in micro-, nano-seconds so why you are asking just for milliseconds ? |
| [2013-11-28 14:39:26] |
| TastyRisk - Posts: 119 |
|
Security X // BEGIN TIME = Second 1 and 001 ms: // event a // event b // event c // event d // END TIME = Second 1 and 100 ms: ... Security Y // BEGIN TIME = Second 1 and 700 ms: // event e // event f // event g // event h // END TIME = Second 1 and 900 ms: ... The implementation of the counter says that non-concurrent events in X vs Y align concurrently when in fact they are sequential. The counter solves nothing. Note that TIME is something SC cannot accurately record. |
| [2013-11-28 14:45:34] |
| TastyRisk - Posts: 119 |
|
ganz; he said that his concepts are working in Multicharts and he hoped to implement them in SC. Why does he have to explain himself to you ? I also respect what you add to the SC forum but your arguing against what others users want/need is now verging on trolling. Date Time Of Last Edit: 2013-11-28 14:47:26
|
| [2013-11-28 14:48:53] |
| Hendrixon - Posts: 130 |
|
lol I know :-) You again fall to the same trap... no retail trader is planing to front run the market on sub second time spreads (as MOST HFTs do). Not because it can't be done technically (from my +15 years of IT background I assure you it can be done) but because you need lots of capital to make it worth while... something like $100,000,000 to get started lol. Again... it's not about sub second HFT execution. I'll share with you this (To make it simple assume a perfect world): I'm located 80ms from Chicago, or 160ms round trip. if in the moment of order entry, all last 100 trades got to my end as 80ms old, that means I took a trade based on the most "fresh" information I could have at my end, right? What if a "safety study" finds that the data I have at my end is older than that? lets say 100ms old? it means something adds latency... Maybe my ISP? my data provider? a router along the way? a problem in the exchange? What if I find that it's not even 100ms stable, but with every new trade the data gets older? [100ms>105ms>107ms>112ms>130ms etc etc]? At this point my study will decide to either scratch the order or maybe compensate the entry by few ticks for safety... all based on rules. Does that have anything to do with HFT **execution**? Nope. |
| [2013-11-28 14:59:52] |
| ganz - Posts: 1048 |
|
Hendrixon Thanks for the case you provide. Your reasons are clear now. |
To post a message in this thread, you need to log in with your Sierra Chart account: