Login Page - Create Account

Support Board


Date/Time: Thu, 26 Dec 2024 12:18:54 +0000



Live data versus replay data problem

View Count: 2055

[2016-02-16 15:35:54]
User55519 - Posts: 37
Hello Support,

I have a bit of a strange problem I need help with. I have written a few custom studies that partially take into account the number of times a price has changed per bar. It works by counting how many times the price moves up or down from the last time a tick was received, which is stored in the persistent variables. If I apply the study to a chart, there is no data at all for this function. However, if I use the replay function, it fills in the data properly - this is fine by me. The problem I'm having is if you look at the replay data and compare it to the live data coming in real time, for the exact same time period, the information is completely different.

For example: maybe on average there are 20 price changes per 10 seconds. On a replay this will be mostly the same, but there will be spikes up to 40, 85, 100, etc. Those spikes do not occur on live data feed.

Can you help me figure out what is wrong here, or how I can get this fixed?

Thanks
Date Time Of Last Edit: 2016-02-16 15:36:49
[2016-02-16 17:01:20]
User55519 - Posts: 37
I just realized I should add that I just read in the support documentation that using "Accurate Trading System Back Test Mode" will not show data tick-by-tick. I have been using that. When I use "Standard Reply" the data is definitely not calculating accurately. So I guess, a better question to add to this would be - How can I access the data that is being used for "Accurate Trading System Back Test Mode" when running a chart live / real-time?

Thanks
[2016-02-16 17:59:41]
User55519 - Posts: 37
Ok I have been pouring through the documentation on replays to make sure I am correct when explaining this problem. What my study is doing (partly) is that every time the study is recalculated it checks to see if a certain float persistent variable is 0. If it is, then it makes the value of that variable, the LAST price instead. When all subsequent calls to calculate are executed, it checks if the current LAST price is different than the one in the persistent variable. If it is, it increments a counter that is stored in a integer persistent variable. If it is not different, it doesn't do anything at all. It then checks to see if this is a new bar. If it's a new bar, then it plots that counter on a line in region 2, and zeros out the integer persistent variable to start over.

When I have my data/trade settings to download 1 tick data, and run the "Accurate Trading System Back Test Mode" is what I'm interested in. In this scenario, what data is it using / how etc? That's what I need to understand. Thanks for your patience with all of my attempts at explanation here.

If I download 1 sec data instead, the results are totally different, and also, not what I'm looking for. Hopefully that explains more about what I'm trying to figure out.

I'd also be happy to pay for a phone call to help get this figured out.

Thanks
Date Time Of Last Edit: 2016-02-16 18:05:57
[2016-02-16 18:08:30]
Sierra Chart Engineering - Posts: 104368
The explanation for what you see is explained here:
https://www.sierrachart.com/index.php?page=doc/doc_ACS_ArraysAndLooping.html#WhenFunctionCalled



So what you need to do is use the Time and Sales data:
https://www.sierrachart.com/index.php?page=doc/doc_ACSIL_Members_Functions.html#scGetTimeAndSales
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
[2016-02-16 18:20:30]
User55519 - Posts: 37
Thanks, I'm looking through these documents now. Can you please explain though, when a calculation of a study is called if the data is set to download by 1 tick, and using the "Accurate Trading System Back Test Mode"?
[2016-02-16 18:25:32]
Sierra Chart Engineering - Posts: 104368
This is explained here:
https://www.sierrachart.com/index.php?page=doc/doc_ReplayChart.html#ReplayMode
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
[2016-02-16 18:49:11]
User55519 - Posts: 37
Ok so I think the main thing I was looking for was in that last link:
"When you use Accurate Trading System Back Test Mode, then this means that the study functions on the chart are going to be called/calculated whenever the High, Low, or Last prices of the latest bar currently loaded into the chart change"

The problem I'm having is this; The calculations are very different when using the accurate mode, and data collection is set to 1 sec versus 1 tick.

How can I get my study to be called the same way / frequency as the replay behaves when set to 1 tick data collection on accurate mode?

Thanks
[2016-02-16 20:34:08]
User55519 - Posts: 37
To add to this, if I'm going to programmatically force a study to update similar to an accurate replay - What is the LAST price actually using - the bid, the ask, or some combination? That will be very helpful to know here.

I have gone through the documents and it seems that the accurate replay using tick data only calls a study when one of these occurs:
New bar
Last, high, or low price changes

And in live mode the study is only called more frequently than the Chart Update Interval set in Global Settings >> General Settings when:
Using sc.UpdateAlways = 1
OR
Setting sc.OnExternalDataImmediateStudyCall

Am I understanding this correctly then, that if I use sc.UpdateAlways or sc.OnExternalDataImmediateStudyCall, and have my study check to see if the last, high, or low price has changed before executing any code, then I would be mimicking the replay behavior?


Additionally, this quote from the documentation makes me ask another questions about the process:
"To visually observe a tick by tick replay, Accurate Trading System Back Test Mode must not be set. This setting is meant for back testing and the chart will not be redrawn frequently but instead infrequently."

What is infrequently? Is it the time set in 'Intraday Data Storage Time Unit'?
If so, how frequently will the chart bars be drawn if the setting is '1 tick' or is there a setting for this?


Thanks
Date Time Of Last Edit: 2016-02-16 20:57:39
[2016-02-16 22:04:30]
Sierra Chart Engineering - Posts: 104368
What is the LAST price actually using - the bid, the ask, or some combination? That will be very helpful to know here.
The last price is the last price. Not the Bid or Ask.


How can I get my study to be called the same way / frequency as the replay behaves when set to 1 tick data collection on accurate mode?
The proper solution in all cases whether during a replay or real-time updating, is that when the study function is called, iterate through the Time and Sales data:
https://www.sierrachart.com/index.php?page=doc/doc_ACSIL_Members_Functions.html#scGetTimeAndSales

This is the proper and only solution to use. All of your other questions and analysis relating to what you want to do are either irrelevant or off course.

Do not use:
sc.OnExternalDataImmediateStudyCall

It is impossible for us to say what "infrequently" is. There are too many variables. You just have to observe this and see what it is.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
Date Time Of Last Edit: 2016-02-16 22:06:35
[2016-02-16 22:06:45]
Sierra Chart Engineering - Posts: 104368
Some corrections were made to the prior posting.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
[2016-02-16 22:28:40]
User55519 - Posts: 37
Ok, thank you for the information.

I do have a question about the follow up though:

If the last price is not the bid or ask, how do I get that information like what is stored in the intraday data storage?

For example, in the Intraday Data Editor, in each entry (like 16:59:58.009, 16:59:58.010, etc) there is a 'last' price there. That's the data I want to see.

Thanks
[2016-02-16 22:51:55]
Sierra Chart Engineering - Posts: 104368
The Time and Sales data contains the trade prices as well as the Bid and Ask prices at the time of the trade.

It is that data you will be working with. That is the only accurate way to do this.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
[2016-02-16 22:56:28]
User55519 - Posts: 37
Ok understood. Now I'm curious though, how is the 'last' price calculated for intraday data storage?

Thanks
[2016-02-16 23:07:31]
Sierra Chart Engineering - Posts: 104368
There is no calculation for this. Sierra Chart uses what is received from the data feed.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
[2016-02-17 09:34:46]
User55519 - Posts: 37
So the last price is recorded by Sierra Charts, from the data feed, on a tick by tick basis, but cannot be accessed by studies on a tick-by-tick basis unless in replay mode?
[2016-02-17 09:50:55]
Sierra Chart Engineering - Posts: 104368
No, this is not what we said at all.

We have been consistently saying more than once is to use the Time and Sales data which contains the recent trades:
https://www.sierrachart.com/index.php?page=doc/doc_ACSIL_Members_Functions.html#scGetTimeAndSales

At this point, you are to have to start working with the code and figuring this out. Further questions are not making any sense to us.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
[2016-02-17 10:52:54]
User55519 - Posts: 37
You guys are awesome! Thanks for the timely and in depth responses, as always. I'll see what I can work out from here.

I'm not sure what doesn't make sense about the question though, it's extremely simple:
Where does the "last" price come from in the intraday data records, or rather, how could that be accessed from the data feed in real time by ACSIL?

I'm asking for access to the 'last' price, and you are pointing me to Time and Sales, which provides 4 sets of data, but you are also saying that none of those are the 'last' price that is stored for intraday data:
SC_TS_MARKER // Indicates a gap in the time and sales data
SC_TS_BID // Trade is considered to have occurred at Bid price or lower
SC_TS_ASK // Trade is considered to have occurred at Ask price or higher
SC_TS_BIDASKVALUES // Bid and Ask quote data update

It seems like an easy question, not sure if I'm missing something else.
[2016-02-17 17:00:28]
User55519 - Posts: 37
I'm still having a really hard time figuring out how to achieve what I want to do. Is there any way you can help me sort this out?

If you are saying that the 'last price' for each tick, that is located in the intraday data storage is not accessible to ACSIL on a tick basis in real-time, then what is an alternative?

I think it would be this; If I cycle through the Time and Sales data, then I'm want to examine the bid and ask price of each tick. Since the last time my study was called. I think I have the code (below) to cycle through Time and Sales data. But I'm missing the basic code to retrieve the bid and ask price during each iteration of the while loop. There is not much in the documentation on this. Can you provide assistance on this?

Thanks


  if ( sc.Index < sc.ArraySize-1 ) return;

  SCTimeAndSalesArray TimeSales;
  sc.GetTimeAndSales(TimeSales);

  if (TimeSales.GetArraySize() == 0) return;

  int TSIndex = TimeSales.GetArraySize() - 1;

  while((TSIndex > 0) && (ColorBar[sc.Index]!=1))
  {
  SCDateTime TradeDateTime = COMBINE_DATE_TIME(TimeSales[TSIndex].Date, HMS_TIME(TimeSales[TSIndex].Hour, TimeSales[TSIndex].Minute, TimeSales[TSIndex].Second));
    TradeDateTime += sc.TimeScaleAdjustment;
    
    if (TradeDateTime < sc.BaseDateTimeIn[sc.Index]) break;
  }
  

[2016-02-18 02:18:13]
Sierra Chart Engineering - Posts: 104368
The time and Sales records that contain trades are as follows:
SC_TS_BID // Trade is considered to have occurred at Bid price or lower
SC_TS_ASK // Trade is considered to have occurred at Ask price or higher

s_TimeAndSales::Price contains the trade price and would be considered the "last" trade price at each trade.

We will put together a code example to iterate through the time and sales data that the study has not yet examined.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
[2016-02-18 04:50:15]
Sierra Chart Engineering - Posts: 104368
There was a small addition we had to make to the code and that is now done. So here is the proper code example for what you want.

This has not been tested.

SCSFExport scsf_TimeAndSalesIterationExample(SCStudyInterfaceRef sc)
{
  if (sc.SetDefaults)
  {
    // Set the configuration and defaults
    
    sc.GraphName = "Time and Sales Iteration Example";
    
    sc.StudyDescription = "This is an example of iterating through the time and sales records which have been added since the last study function call.";
    
    sc.AutoLoop = 1;
    
    sc.FreeDLL = 0;
    
    return;
  }

  //Only do this processing on the last bar
  if (sc.Index == sc.ArraySize-1)
  {

    __int64& LastProcessedSequence = sc.GetPersistentInt64 (1);

    // Get the Time and Sales
    SCTimeAndSalesArray TimeSales;
    sc.GetTimeAndSales(TimeSales);
    if (TimeSales.Size() == 0)
      return; // No Time and Sales data available for the symbol

    if (LastProcessedSequence != 0)//only iterate the time and sales if have a prior sequence number
    {
      // Loop through the Time and Sales.
      for (int TSIndex = 0; TSIndex < TimeSales.Size() ; ++TSIndex)
      {
        if(TimeSales[TSIndex].Sequence < LastProcessedSequence)
          continue;

        //only interested in trade records
        if (TimeSales[TSIndex].Type == SC_TS_BID || TimeSales[TSIndex].Type == SC_TS_ASK )
        {

          float TradePrice = TimeSales[TSIndex].Price;
          float BidPrice = TimeSales[TSIndex].Bid;
          float AskPrice = TimeSales[TSIndex].Ask;
        }
      }
    }

    LastProcessedSequence = TimeSales[TimeSales.Size()-1 ].Sequence;
  }

}

Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
[2016-02-18 11:32:47]
User55519 - Posts: 37
Amazing! You guys are great! Thanks, I will test this out today.
[2016-02-18 11:51:05]
User55519 - Posts: 37
REMOVED - I added a response here, showing part of my code that was not working correctly with the example provided. This problem was 100% related to my code only.
Date Time Of Last Edit: 2016-02-18 12:03:03

To post a message in this thread, you need to log in with your Sierra Chart account:

Login

Login Page - Create Account