Support Board
Date/Time: Mon, 25 Nov 2024 20:53:51 +0000
[User Discussion] - Version 1047: Millisecond time stamping
View Count: 5554
| [2013-11-28 14:59:52] |
| ganz - Posts: 1048 |
|
Hendrixon Thanks for the case you provide. Your reasons are clear now. |
| [2013-11-28 15:00:24] |
| Hendrixon - Posts: 130 |
|
And to make it clear, yes I tested an idea on microseconds and it works! Will I have LOTS of use for microseconds? not seeing it in the near future, but will it hurt me if SC will support it??? Of course not. so why you are asking just for milliseconds?
Because milliseconds come before microseconds?And because if I'll ask for microseconds most chances SC will flag this as "user discussion"... |
| [2013-11-28 15:06:09] |
| ganz - Posts: 1048 |
|
TastyRisk Trolling? :) This was marked as a User Discussion but I can stop to talk w you. no prblm. Done. |
| [2013-11-28 15:08:20] |
| ganz - Posts: 1048 |
|
Hendrixon if I'll ask for microseconds most chances SC will flag this as "user discussion"...
OK. :)) ... It's clear. :) |
| [2013-11-28 15:08:46] |
| Hendrixon - Posts: 130 |
|
User Discussion??? When that happened? I'm speechless... Honestly I don't know what to say... |
| [2013-11-28 21:06:47] |
| |
|
Final responses in this thread: You get information from the exchange with millisecond stamping. As you boast 0% packet loss on your CME feed then it is evident that there is little congestion on the switch fabric. Therefore the CME time-stamp would be an excellent substitute for a true trade-time-stamp. Possibly. The trades do not have millisecond timestamps, but the fix message which contains multiple trades does have a sending time that has milliseconds. We do not use it not for the reason that it's not necessarily representing the milliseconds that the trades took place at, but because of the fact that Sierra Chart benefits from milliseconds by using them as a counter to uniquely identify trades.We have to make a decision as to what is beneficial for most of our users and not make things overly complicated to satisfy everyone. There are many programming considerations here. Lots of things are possible, but they start making the program more complicated, causing back compatibility issues, and performance problems. We will not do that to satisfy two or three people and we would rather lose you rather than creating excessive burdens on us. We are just being honest here. You do not realize all of the complex technical details and problems created with millisecond support. This is what took it so much time. And there's still more to do. You can always feed millisecond timestamps into Sierra Chart using the DTC protocol. Although as Sierra Chart processes that data, it will increment the timestamp to the next millisecond if it sees a duplicate. Up to a maximum of 999. In response to post 10 and 14 about overlaying one chart on another, the overlay studies have not been updated to take advantage of the new millisecond time stamping. We will work on that today. This is one reason we have added millisecond time support. We will first work on Study/Price Overlay study. We have not looked at the Chartbook and what configuration you have but if you are using that study it will be solved. And the problem will only be properly solved when each trade has a unique timestamp. This is the problem with exchange timestamps, they bunch trades together into a single message and many trades will have the very same timestamp when they probably executed at different times. You really have to go all the way down to the microsecond level to get unique timestamps. And the Sierra Chart 8 byte float type has no precision for that. Otherwise, we have no problem with supporting microseconds. That would be done for sure. No problem. I was going off the 8 byte time that appears in the SCID files, which has enough room for 300 years of nanoseconds,
Double precision numbers do not have enough digits of precision to represent this. The same way as the crosshair tool jumps to a very first tick of a given second on a 1-tick chart. If we mark a thread as a User Discussion, it's because we have said all we can, and we don't want to spend further time on it we have to get on with other things including the full implementation of milliseconds throughout the program. Do you know how much time is being spent as I go through this thread. About an hour. We do not have this kind of time. This is taking away time from everything else. And we are well aware of the limitations when trades do not have unique timestamps and this is what we have been intending to solve for some time now. After SC promised full [REAL] millisecond time frame support, in anticipation for that I've tested things on Multicharts demo and in 3 weeks I found that several concepts I wanted to implement in my SC studies are spot on. Then use multi-charts. We have to make a business decision here. We cannot be everything to everybody. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2013-11-28 21:10:47
|
| [2013-11-28 21:23:04] |
| |
|
The same way as the crosshair tool jumps to a very first tick of a given second on a 1-tick chart. Update: Checking on the code for this. We do not see how this ever was a problem. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2013-11-28 21:27:47
|
| [2013-11-28 21:36:07] |
| ganz - Posts: 1048 |
|
Hello SC Support. We have tested this. We do not see this problem. This has been apparently solved for some time now.
Update: Checking on the code for this. We do not see how this ever was a problem. The setup is: Two 1-tick charts are side by side and Global Cursor On is checked the crosshair jumps for an inactive window. |
| [2013-11-28 21:37:30] |
| Kiwi - Posts: 375 |
|
A vote for: making good business decisions for our intended customer base and platform integrity. Although I find the order aging issue interesting. Some of these ms issues could be resolved with a user written software although, just like for Sierra, a lot of it would be quite challenging. At least it wouldn't impact the SC application. A vote for: broker or exchange hosted OCO orders being much more important to the majority of customers than intra-second refinements (IB). Edit: Also a vote of thanks for the huge number of enhancements the SC team has made. I only dreamed of some of this stuff back in V39. And I have 100s of lines of code to do something that I need now but didn't want to reuse for age and safety reasons --- only to find that Sierra now has ACSIL functionality so I can do it with 10 or 20 new lines of fresh code. Well done guys. Date Time Of Last Edit: 2013-11-28 21:43:36
|
| [2013-11-28 21:37:46] |
| |
|
In response to post 32, yes we just realized this now. We will correct this.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2013-11-28 21:39:11
|
| [2013-11-28 21:48:06] |
| ganz - Posts: 1048 |
|
Kiwi +1024, also a lot of people will be happy to use the fresh session start and the histogram for Zig-Zag study. This will be the real and the useful statistics tool for everyone, imho Also I want to say thank you to Hendrixon for being so kind to provide some ideas. But these awsome goals may be achieved in other way than to make SC so complicated for those particular cases, imho^2. :) |
| [2013-11-29 00:35:45] |
| TastyRisk - Posts: 119 |
|
The trades do not have millisecond timestamps, but the fix message which contains multiple trades does have a sending time that has milliseconds.
I thought about this and came to realise that when the exchange receive a marketable order (can be matched) there exists the possibility that other trades could be triggered. For example; stop orders and some other other resting order types. I speculate that the above is the reason why information is sometimes "bundled" onto the egress of the MDP Gateway with the same time-stamp / into one message. In this case, all the trades effectively happened at the same time... Although I am interested to know if sequential order is maintained. So, If I understand correctly; the present data structures in SC would easily allow MS support but there would be no "room" for the existing sequence count without a major overhaul that would break compatibility ? OK, I'll wait for the new SC v2.00. 64-bit and Milli/Micro second support *with* extended sequential trade flag. :-) Regards. Date Time Of Last Edit: 2013-11-29 00:54:15
|
| [2013-11-29 03:21:07] |
| |
|
Date-Time values use this format: http://www.sierrachart.com/index.php?l=doc/doc_IntradayDataFileFormat.html#s_IntradayRecord Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2013-11-29 12:41:00] |
| Hendrixon - Posts: 130 |
|
TastyRisk, of course there are many trades who get the same time stamp. I don't know why SC think it is abnormal when it is normal, since any firm that trades size is trying to mask their activity and one of the ways is to break up a big lot to plenty of small lots with various sizes. Its easy to track institute traders when lots are big, but much harder when they break orders to small bits and *flood* the tape. Many "tape traders" use tools that try to "reconstruct the tape" and expose those masked big lots, but its not that easy because you need to establish rules to exclude big splashes of orders who are simply triggered stops, who may look like a "masked big lot" from an aggressor. So yea, there can never be a situation that every single trade match has a unique time stamp, not even in picoseconds... Date Time Of Last Edit: 2013-11-29 12:41:27
|
| [2013-11-29 13:03:34] |
| Hendrixon - Posts: 130 |
|
And to SC, if you still read this, here is a comment I got after exchanging few emails with a big HFT firm: In general we never rely on the timestamps provided by an exchange to indicate time of trade as it usually does not have good enough granularity and it usually is not representative of the time of the matching event and is subject to change.
We timestamp data to indicate when we received the messages and it could be argued that such a time is a better time as it actually represents the first time we could know about it (we presume that we could know about it when someone else outside the exchange could know about it). The fact that orders were matched at some time is probably not as important as when the rest of the world found out about it so they could take action on it. I'll add that I agree with this since the perfect time stamp should be the order arrival time. that represent the intent and action logic of the aggressors. Arrivals time stamp eliminates stop orders time stamps [who are totally meaningless] and is clean of any matching logic/latency or bundling with triggered stops. If we assume FIFO, with fixed matching latency, than time stamping the out going stream is pretty representative of arrival time. |
| [2013-11-29 14:51:11] |
| TastyRisk - Posts: 119 |
|
One message* can also contain trades for multiple security definitions, AFAIK. *having a single time-stamp Date Time Of Last Edit: 2013-11-29 14:54:14
|
| [2013-12-01 15:22:18] |
| jesslinn - Posts: 108 |
|
Supporting external millisecond time stamping only defeats the purpose of the reason milliseconds are being used in Sierra Chart, and will be hard to support.
I do not understand how anything is defeated. Presumably, the "reason milliseconds are being used in Sierra Chart" is as a sequence number to make sure that each tick for a given symbol has a different timestamp. How is this affected by supporting milliseconds? If you really want to encode a sequence number inside a time you can still do it and support external milliseconds. If sequential ticks come in during the same millisecond just increment by 1/100th of a millisecond. Even with the unfortunate implementation of time that SC has, there is room for that. It is also not easy to guess why it would be hard to support. The only place that need be affected is the place where the timestamp with counter is created. Everywhere else is ignoring all those wasted bits anyway but they would get written to the files for our use.Somewhere in this discussion was the suggestion of time-stamping on arrival at the platform. This is not useful for measuring probable delays from exchanges and vendors but it would actually be better for reconstructing the datafeed from the individual scid files, which would be very nice for back-testing. In any case, it is much easier and better to measure datafeed delays before the ticks ever come to the charts and strategies. |
| [2013-12-01 22:07:44] |
| |
|
If sequential ticks come in during the same millisecond just increment by 1/100th of a millisecond.
Double precision floating-point numbers have 15 to 17 digits of precision. It would make more sense to go out to microseconds and not 1/100 of a millisecond. However, this is exceeding the digits of precision when you consider that 5 digits to the left of the decimal place is needed to represent the days, but we have looked into this, and there is still enough precision to distinguish microseconds. So we will give it a try. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2013-12-01 22:08:22
|
| [2013-12-01 23:50:57] |
| jesslinn - Posts: 108 |
|
Thanks. It would certainly be valuable to me for you to give it a try. I suggested hundredths of a millisecond for the sequence number because it lets you have 2 centuries, but if you limit the number of days since the epoch to 65535 (179+ years) then, as you say, you should be able to support a microsecond sequence counter just fine. |
| [2013-12-07 12:12:52] |
| ganz - Posts: 1048 |
|
Hello SC Support there is still enough precision to distinguish microseconds. So we will give it a try.
Will it be for CME Group data source?http://www.cmegroup.com/confluence/display/EPICSANDBOX/Market+Data+-+Message+Specification UTCTimestamp YYYY = 0000-9999, MM = 01-12, DD = 01-31, HH = 00-23, MM = 00-59, SS = 00-60 (60 only if UTC leap second), sss=000-999 (indicating milliseconds). UTCTimeOnly HH = 00-23, MM = 00-59, SS = 00-59. sss=000-999 (indicating milliseconds) I can't see microseconds resolutions there or am I missing something? Thank you. Date Time Of Last Edit: 2013-12-07 12:32:13
|
| [2013-12-08 01:59:32] |
| jesslinn - Posts: 108 |
|
No, ganz. You are not missing anything. The point is that SC wants to make sure that every tick (for a given symbol) has a different timestamp, which can be useful for processing. If two ticks come in with the same millisecond timestamp, then it will be possible to use the microseconds as sequence numbers, as is now being done with milliseconds. The milliseconds would be a valid timestamp and the microseconds would ensure strictly increasing times. |
| [2013-12-08 04:15:05] |
| Hendrixon - Posts: 130 |
|
So... are we back to have real time based millisecond support? |
| [2014-03-12 11:28:25] |
| Perseus - Posts: 30 |
|
As a future feature, I would also appreciate true mS timestamps as discussed above. As an alternative to using uS might it instead be feasible to use a unique tick-identifier such as provided by for example IQFeed to distinguish between two identically mS stamped ticks?
|
To post a message in this thread, you need to log in with your Sierra Chart account: