Support Board
Date/Time: Thu, 26 Dec 2024 14:30:16 +0000
[User Discussion] - A Better Implementation of Interactive Broker's Data Feed
View Count: 11736
| [2013-04-05 16:07:58] |
| Bad Dog - Posts: 17 |
|
A lot of postings deal with volume issues with Interactive Broker's datafeed. SC developers have indeicated that the data feed is missing trades, so the volume is understated. To compensate, SC adds phantom trades to make up for this missing volume. The SC developers have suggested that alternate brokers or alternate data feed providers be used if the quality of IB's data feed is inadequate. Unfortunately if you reside in Canada (especially British Columbia), there are very few alternate brokers available to replace Interactive Brokers and provide comparable trade commissions. I have never been convinced that IB's data feed could not be improved, so I created my own data feed using IB's API. The resulting feed data differs quite significantly when compared to Sierra Chart's version of the IB data feed. To measure the accuracy of the data feed, I have compared the feed to Sierra Chart's two data feed backfill services: SC Historical Data & SC Futures Backfill Data. Using the crude oil futures symbol, CL, I created 10 minute bars using the four different data feeds, and compared their volumes: 11:10am 11:20am 11:30am 11:40am 11:50am SC real-time feed using IB data 15878 9574 1141 1146 2228 My custom real-time data feed using IB data 10649 3460 980 917 463 SC Historical Data (backfill) 10572 3462 979 908 463 SC Futures Backfill Data (backfill) 6796 3997 940 868 445 I found that none of the data feed matched exactly, including Sierra Chart's two historical data feeds. However the closest match is my custom data feed and SC historical data. Sierra Chart's r/t feed using IB data does not come close to matching any of the other feeds, and is consistently overstated. I believe the addition of phantom trades by SC is the cause of the large volume numbers. I have attached two screen shots of the 10 minutes charts using the various data feed, showing the market profiles of the bars. The shots show that my custom data feed matches SC futures backfill data fairly closely, while Sierra Charts data feed using IB data does not. I hope this information illustrates my point that a better data feed using Interactive Broker's data feed and API can be created. My programming skills as rather crude, so I am sure the SC developers would have not problem improving upon my attempt. I can submit details about the coding of the custom data feed if anyone is interested. -BD |
 customVersusSC.jpg / V - Attached On 2013-04-05 16:03:50 UTC - Size: 688.2 KB - 861 views IBversusSC.jpg / V - Attached On 2013-04-05 16:04:12 UTC - Size: 692.05 KB - 750 views |
| [2013-04-05 17:23:23] |
| |
|
This posting is very puzzling to us. We have addressed this problem for you and for others and made it very clear here: https://www.sierrachart.com/supportboard/showthread.php?t=42481 Once again here is a restatement of what we previously said: We will add an option to not insert trades from Daily Volume messages and then you can evaluate what works best for you. There is a good reason why we do that processing because sometimes there is trading that occurs, but TWS will only send daily volume messages.
There used to be a way to disable this, but we force this feature on because if someone inadvertently disables it, then we would receive feedback from them asking why their chart bars look sparse and incomplete. So unless IB has changed the data feed, we would likely think that you would keep the option on. Although, we want to provide you the capability to do your own evaluation. The problem with this is that if you find you need to keep this on, we might as well take the option out. This is now supported in the latest Sierra Chart version: Quote: When using the Interactive Brokers Trading service, this functionality to simulate a trade in order for the Intraday chart volume to match the Daily Volume messages, has shown to cause volume spikes in Intraday charts. Therefore, this can be disabled for Interactive Brokers. To disable this, uncheck Global Settings >> Data/Trade Service Settings >> More Interactive Brokers Settings >> Simulate Trades to Match Daily Volume. Actually, there is a way to disable this. Enable Global Settings >>Data/Trade Service Settings >> More Interactive Brokers Settings >> Record True Real-Time Data in Intraday Charts. More information about this here: http://www.sierrachart.com/index.php?l=doc/IB.php#TrueRealTimeData Unless you use Record True Real-Time Data in Intraday Charts, that we advised you to. Why do not you use it? Here is the documentation once again for this: http://www.sierrachart.com/index.php?l=doc/IB.php#TrueRealTimeData How is it that you overlooked and disregarded what we said. Please explain. There is nothing new here. We provided solutions. We assumed you followed our suggestions and you continued to indicate there was a problem with the data feed, and we said this is an IB problem , which is correct if you continued to have an issue after following our instructions, so there was nothing for us to do. And there still is not. We have for years, based on user feedback heard about incomplete data from IB. So we know there is and has been a problem. Maybe they have made some improvements which seem to be indicated in your testing. There was a good reason why Sierra Chart would simulate trades to match the daily volume messages. This was implemented many years ago based upon how the data feed from Interactive Brokers worked many years ago and this information was explained to us by a user. What TWS would do, is when the price does not change they do not send a new price but instead just increment the daily volume. So the only accurate way to chart it was to do processing with the daily volume messages. We are just providing some background here. This may not be necessary currently or in the CL Futures market. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2013-04-05 17:50:54
|
| [2013-04-05 19:17:08] |
| Bad Dog - Posts: 17 |
|
Thanks for the quick reply. I have not ignored your suggestions about using the TrueRealTimeData option. I recognize it as being very innovative and a significant improvment. Although I do not fully understand the details of how it is implemented, I see that it provides accurate volume. However, its Ask/Bid data is very inaccurate. Since the TrueRealTimeData relies on the tick data feed from IB to get ask/bid volumes, I believe the TrueRealTimeData could benefit by an improved implementation to the tick data feed. Sometime in 2011, Interactive Brokers updated its API to include a new tick type called "RTVolume". This was added to help provide accurate tick volume data. I used it to create the custom data feed. I respectfully ask that your development team take a look at IB's updated API and the RTVolume tick type. It may help you improve the quality of IB's data more as you have already done with the TrueRealTimeData option. -BD |
| [2013-04-05 20:21:35] |
| |
|
OK, we looked this up and we see it. Clearly we have not been using this and did not consider the usefulness of it. So we have learned something here. Acknowledged. We will have to see how best to implement this.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2013-04-09 04:54:29] |
| |
|
We have implemented the use of RT volume, but we could not test it. This is in version 962. Let us know how it works. But we wonder if it's going to work properly with all symbols. In any case, after considering this some more, while it may be helpful with volume, it still does not solve the problem of IB providing incomplete Tick by tick data. You really have to use the True Real-time data for complete bars.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2013-04-11 08:40:17] |
| Tonkadad - Posts: 235 |
|
Today my volume charts with IB as data source are very condensed,I have 2 charts CL and Euro and they both are 1/5th the normal size (number of bars showing) something must have changed when you made the changes with IB.
|
| 989 vol Euro.PNG / V - Attached On 2013-04-11 08:40:08 UTC - Size: 68.69 KB - 628 views |
| [2013-04-11 08:49:08] |
| Tonkadad - Posts: 235 |
|
When I delete the intraday data and it reloads the data again it seems to be ok.
|
| [2013-04-11 09:20:14] |
| |
|
Please update to the very latest version. The changes have been removed. We are not even clear if there is any practical solution at this point. We will have to review it later on.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2013-04-11 20:51:26] |
| Bad Dog - Posts: 17 |
|
I have tested the IB data feed using verion 962 and found the same problem as reported by Tonkadad. Only about 18% of the actual volume is retieved using this version of IB's data feed. I continue using my custom version of IB's data feed using an implementation of "RTVolume" to retrieve volume information. I have calculated that this custom datafeed is 99.98% accurate, so it is possible to create a more accurate data feed using IB's API. BD |
| [2013-04-11 21:17:52] |
| |
|
This is difficult for us to test because we do not have access to a standard Interactive Brokers data feed. Also how do you use RT Volume? It is a completely separate value independently sent. In our testing we did not receive any data for it. So we are just making guesses about it. The way we were using it was the same the standard Daily Volume messages. And we were no longer processing the Daily Volume messages. When requesting market data how did you set generic ticks. We were using "233": // Generic ticks. 233 = RT_VOLUME If you tell us specifically what you are doing, we can just implement the same thing and then we should be good to go. Otherwise, because we do not have access to the actual live feed it is hard to evaluate how we should do this. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2013-04-11 21:24:07
|
| [2013-04-11 22:08:38] |
| Bad Dog - Posts: 17 |
|
Here is what I used to code the data feed: reqMktData() - Call this function to request market data. The market data will be returned by the tickPrice() and tickString() events. void reqMktData(TickerID id, const Contract &contract, CString genericTicklist, bool snapshot) The String, generericTickList is passed as string "233" 1.Use event tickPrice() to retrieve the latest askprice (ticktype=1) and bidprice (ticktype=2). 2.Use event tickString() to retrieve rtVolume events (tickType=48). IB's documentation mistakenly says that RTvolume is passed by tickGeneric() but this is wrong. RTVolume is a string and is passed by the tickString() event. Fields in struct s_IntradayRecord: double DateTime.............RTVolume(lastprice, volume, dateTimeStamp, TotalVolume, VWAP, Single trade flag) float Open .................RTVolume(lastprice, volume, dateTimeStamp, TotalVolume, VWAP, Single trade flag) float High .................RTVolume(lastprice, volume, dateTimeStamp, TotalVolume, VWAP, Single trade flag) float Low...................RTVolume(lastprice, volume, dateTimeStamp, TotalVolume, VWAP, Single trade flag) float Close.................RTVolume(lastprice, volume, dateTimeStamp, TotalVolume, VWAP, Single trade flag) unsigned long NumTrades.....1 unsigned long TotalVolume...RTVolume(lastprice, volume, dateTimeStamp, TotalVolume, VWAP, Single trade flag) unsigned long BidVolume.....tickPrice() passing ticktype=2. If lastprice<=bidprice then BidVolume=volume else 0 unsigned long AskVolume.....tickPrice() passing ticktype=1. If lastprice>=askprice then AskVolume=volume else 0 BD Date Time Of Last Edit: 2013-04-11 22:13:13
|
| [2013-04-11 23:03:45] |
| |
|
2.Use event tickString() to retrieve rtVolume events (tickType=48). IB's documentation mistakenly says that RTvolume is passed by tickGeneric() but this is wrong. RTVolume is a string and is passed by the tickString() event. This is the issue. We will solve this.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2013-04-12 00:47:40] |
| |
|
For tick type 48 on the TickString event, we are getting the last trade price. We are testing ES June 2013 futures and getting a number like 1547. Or sometimes 1547.25. Highly doubtful we are doing anything wrong with the data processing. This is on the "edemo" username. Could you confirm that you get the same thing? Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2013-04-12 03:46:22] |
| Tonkadad - Posts: 235 |
|
Nice job Bad Dog. If you have time would you mind posting a 200V chart of the CL-201305-NYMEX so I can compare with mine. thanks again for following up with this. |
| [2013-04-12 04:54:54] |
| Bad Dog - Posts: 17 |
|
When I use IB's demo account (edemo) I also get numbers like 1545 and 1544.75. However, when I use my IB account, I get more realistic data with prices around 1585. Looks like you are now retrieving the proper info from the data feed. -BD |
| [2013-04-12 05:51:33] |
| Bad Dog - Posts: 17 |
|
Tonkadad: 200V CL-201305-NYMEX chart for 4/11/13 as requested. Pacific Std Time http://www.sierrachart.com/image.php?l=1365746113917.png http://www.sierrachart.com/image.php?l=1365745766150.png -BD Date Time Of Last Edit: 2013-04-12 05:55:37
|
| [2013-04-12 06:03:00] |
| |
|
When I use IB's demo account (edemo) I also get numbers like 1545 and 1544.75. However, when I use my IB account, I get more realistic data with prices around 1585. Looks like you are now retrieving the proper info from the data feed.
What we are saying is that for tick type 48 on the TickString event, we are getting the last trade price. Not volume. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2013-04-12 06:03:19
|
| [2013-04-12 07:32:26] |
| Tonkadad - Posts: 235 |
|
Thanks BD. I attached a pic of my 200v chart. |
| tonkadad 200v CL.PNG / V - Attached On 2013-04-12 07:32:21 UTC - Size: 46.18 KB - 713 views |
| [2013-04-12 14:21:51] |
| Bad Dog - Posts: 17 |
|
RTVolume returns a string containing the following information: Last trade price Last trade size Last trade time Total volume VWAP Single trade flag (true or false). See link for further info and example: http://www.interactivebrokers.com/en/software/api/apiguide/tables/rtvolume.htm |
| [2013-04-12 15:30:31] |
| |
|
We have been looking at an older User guide: http://www.interactivebrokers.com/php/apiUsersGuide/apiguide.htm Interactive Brokers is very disorganized. They now have at least 2 different user guides. Why is it also that they are transmitting a different set of information through a text string compared to their numeric data feed coming through the other events. This is not making sense. They state "RTVolume is the API equivalent to opening the Time and Sales Window ". This is exactly what we would expect to be part of their standard data feed but it is not. None of this is making any sense. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2013-04-12 15:55:45] |
| Bad Dog - Posts: 17 |
|
Agreed, working with Interactive Brokers can be a challenge. Unfortunately I have not yet found a cheap broker that will provide futures services in British Columbia, Canada. If you have any suggestions, let me know. Do you have any idea, how many of your customers use IB? -BD Date Time Of Last Edit: 2013-04-12 15:56:32
|
| [2013-04-21 04:59:23] |
| |
|
In version 967 Sierra Chart now processes the Time and Sales data feed known as RT_VOLUME by IB. We did some basic testing and it seems to work just fine. If this data feed is not available, Sierra Chart will continue to process the standard data feed. The standard data feed gets ignored for a particular symbol as soon as we receive the first RT_VOLUME from TWS. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2013-04-21 05:00:11
|
| [2013-04-30 22:45:58] |
| |
|
We have disabled support for this new data feed and 969 because of the problem reported here: IB Data with big spikes in DAX Future We might add an option for this data feed, but this is really getting confusing. Now there are 4 different data feed possibilities with Interactive Brokers: Time and Sales Data Feed 5 Seconds True Real-Time Data Standard API data feed without simulated trades to match daily volume Standard API data feed with simulated trades to match daily volume From support perspective, this is really getting difficult. In the end, does this new time and sales data feed make any sense? Clearly it does not make sense to use it as a default because it has serious problems. So in the end it appears as though our original position is correct. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2013-04-30 22:54:09
|
| [2013-10-20 22:06:49] |
| katek - Posts: 28 |
|
Do we have any news regarding this issue? What is the best way toe get accurate bid /ask volume data from IB. I have been using IB True real-time data, but even this feed does not give accurate bid/ask volume. On this feed trades in the 5 seconds period of "real-time data" are considered either "bid volume" or "ask volume". And this is certainly not the case as most 5 seconds periods usually have trades at bid and at ask prices(to check this use a 5 second bar chart with bid/ask volume bars). |
| [2013-10-20 23:31:38] |
| |
|
Yes. We are able to deliver a solid solution to this. We have a new data feed available that you can subscribe to starting around the beginning of next month: Notice: New Futures Real-Time Data Feed We will be putting no further time into interactive brokers data feeds. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
To post a message in this thread, you need to log in with your Sierra Chart account: