Support Board
Date/Time: Tue, 26 Nov 2024 12:34:39 +0000
Post From: Problem with market depth desapearing
| [2015-03-31 02:16:29] |
| i960 - Posts: 360 |
|
Yes of course. And we will say it again, the problem is on the Interactive Brokers side. It is caused by out of order data because of faulty transmission of market depth data from TWS.
Absolutely not. Anyone writing serious event driven code knows order is *never* guaranteed. It's a given by definition of standard concepts of asynchronous programming. Your code is not resilient against inserts, updates, and deletes arriving out of order while the API makes no guarantee of such a thing (nor should it). There's absolutely zero reason to assume that all inserts happen first or all updates happen first. In correct code they can arrive in any order and the final result should be the same always - just the same as if I add 2+3-7+4 the result is the exact same as -7+4+3+2. Order is not guaranteed with event driven architectures! This code has potential issues, you should be ready to accept that. Let's use a hypothetical value for inserting at a level of 2 with size of 114. case INSERT_DOM: { s_SCDomStructure TempDomStructure = DomToUpdate[Level]; The above is a copy of DomToUpdate[2], which is fine, because it's potentially going to be used for a Swap(). for (int UpdateLevel = Level; UpdateLevel < MAX_NUM_DOM_LEVELS - 1; UpdateLevel++) { if (TempDomStructure.Price == 0) break; // no more DOM Levels Swap(DomToUpdate[UpdateLevel + 1], TempDomStructure); } DomToUpdate[Level].Price = Price; DomToUpdate[Level].Volume = Size; } break; The above "0 as a sentinel for nothing left" conditional is completely suspect. It assumes that everything always arrives in a specific order of completely non-zero updates for every single level beforehand and relies on a sentinel value that isn't even guaranteed not to be sent by the datafeed, and worse, terminates the processing of the remaining levels being moved if inserts arrive out of order before a full update of all levels has occurred (it's entirely possible partial updates could occur followed by inserts) or an explicit value of 0 has already been updated from the feed. It's particularly prone as I bet the structure is being initiated with zeros beforehand. The following will completely break: 0: 123 1: 89 2: 77 3: 0 4: 55 Request to insert at level 2, with value of 114 will result in this afterward: 0: 123 1: 89 2: 114 3: 77 4: 55 Level 3 got shifted with the old value of level 2 correctly, but the old value of 0 for level 3 just got lost and now level 4 is not what the feed thinks. This is important because the feed has it's own idea of levels that the client structure must follow 1:1 at all times. A correct insert would look like: 0: 123 1: 89 2: 114 3: 77 4: 0 5: 55 It doesn't matter if 4 has a value of 0, as it doesn't necessarily mean "terminate further shifting of levels past the thing we're inserting." It's not a true sentinel value because it's not guaranteed the feed cannot send it nor is it guaranteed all levels are populated with non-zero values before inserts even start arriving. The correct thing to do here to avoid always processing all values up to MAX_NUM_DOM_LEVELS and also allow 0 as a valid existing price at any level is simply not rely on this false sentinel but keep a simple high watermark int in one of the parent structures. Said int points to the highest level processed either by an insert, update, or delete. An insert for level 0, means last_level is incremented. An update for level 7 means last_level = MAX(last_level, MIN(MAX_NUM_DOM_LEVELS, Level)). A delete for level 6 means last_level is decremented. It's a latching variable that just bounds the amount of processing the for loop has to do but more importantly gets the 0 value out of the picture. Obviously set it with a min() macro such that it's bounded by MAX_NUM_DOM_LEVELS, e.g. last_level = MIN(Level, MAX_NUM_DOM_LEVELS) so that values on the wire are safe. Then change the for loop to use UpdateLevel < last_level rather than UpdateLevel < MAX_NUM_DOM_LEVELS - 1. You're going to have to find a place in either m_BasicData to place this last_level for both Bid and Ask DOMs, otherwise the DOM structure itself is going to have to be something other than a simple array. Alternatively, one could just comment out the "if (TempDomStructure.Price == 0)" check and process every single index up to MAX_NUM_DOM_LEVELS. That will achieve the same thing as not breaking on legitimate levels that already have 0, not require any structure changes, but will require more iterative processing. Since the amount of inserts and deletes is probably far less than updates, this may be an acceptable tradeoff as it's only a single line change to make the problem go away. The following will also totally break: 0: 123 1: 89 2: 0 3: 77 4: 55 Request to insert at level 2, with value of 114 will result in this afterward: 0: 123 1: 89 2: 114 3: 77 4: 55 On the surface, that may look right, but that's now potentially wrong because the feed's idea of 3 was 77 *before* the insert was sent. It's existing view probably looks like this: 0: 123 1: 89 2: 114 3: 0 4: 77 5: 55 Once again, 0 is not a safe sentinel value. Also, making an equality comparison to 0 using a float will probably work safely here, but in general floating point comparison is a known problem area. Aside from that, it's just not safe here to use 0 for anything other than 0 as a value, and not a special control value. Given these existing values, the code will work right with an insert at level 2, with value of 114: 0: 123 1: 89 2: 77 3: 55 Resulting in: 0: 123 1: 89 2: 114 3: 77 4: 55 Which is the assumed normal case. case UPDATE_DOM: { DomToUpdate[Level].Price = Price; DomToUpdate[Level].Volume = Size; } break; The above code looks fine although remember, it's entirely possible (although it might be weird) for the feed to send a value of 0.0 for price and the feed is going to expect it's idea of the levels being worked with is followed by your client using them. If a bunch of initial updates happened with a wide spread or possibly a level without any bids/asks (this *does* happen), you're going to end up with a level that's already 0 if the structure was initialized to 0's beforehand or explicitly updated by the feed with a price of 0.0. Remember, this is then going to break the insert processing because it thinks 0 (or 0.0) means "end of the array" when that isn't necessarily the case, and as a result it won't shift the remaining levels to the right positions that the feed thinks they should be at. case DELETE_DOM: { // Clear the single level memset(&DomToUpdate[Level], 0, sizeof(s_SCDomStructure)); The above isn't 100% portable, although it'll work probably everywhere. It's also safer to use "sizeof DomToUpdate[Level]" incase it's type changes. However, it's not even necessary because as long as Level is < MAX_NUM_DOM_LEVELS - 1 it's going to get stomped by the DomToUpdate[UpdateLevel] = DomToUpdate[UpdateLevel+1] below. Additionally, when Level is == MAX_NUM_DOM_LEVELS - 1, it's going to get unconditionally stomped by the block which is following the for loop anyway. for (int UpdateLevel = Level; UpdateLevel < MAX_NUM_DOM_LEVELS - 1; ++UpdateLevel) { DomToUpdate[UpdateLevel] = DomToUpdate[UpdateLevel+1]; // //if (DomToUpdate[UpdateLevel + 1].Price == 0) // break; // no more DOM Levels } //if (DomToUpdate[MAX_NUM_DOM_LEVELS - 1].Price != 0) { DomToUpdate[MAX_NUM_DOM_LEVELS - 1].Price = 0; DomToUpdate[MAX_NUM_DOM_LEVELS - 1].Volume = 0; } } break; Again, 0 is not a safe value to use to mark the end of the levels. Only a separate variable holding the last unused index is safe. It's perfectly sane to zero out all of the Price and Volume members here for reasons of a placeholder value (as it's not like any other value makes sense, and I'm sure downstream code knows to display an empty DOM level if the value is 0.0), but it's not safe to do it just to manipulate the insert code's idea of what the end of the array is. Also, this code has to process every level up to MAX_NUM_DOM_LEVELS on any delete. Using a last_level variable as I described previously would mean it wouldn't have to do all of that on every single delete. It just deletes up to that point because there's no way any levels beyond that have ever been touched. Here's another example DOM that's busted due to these ordering issues with how the code is handling it.... Date Time Of Last Edit: 2015-03-31 03:08:14
|
 sc_busted_dom_cl.png / V - Attached On 2015-03-31 02:13:36 UTC - Size: 90.39 KB - 442 views |