Login Page - Create Account

Support Board


Date/Time: Mon, 25 Nov 2024 12:21:25 +0000



[User Discussion] - "accurate" backtest results change when multiple charts opened

View Count: 2491

[2014-02-02 04:09:00]
hairyMug - Posts: 31
when I have one chart open and do an "accurate" back-test my profit is $3072.
after opening another 30 charts and running the same back-test on just the same chart with the exact same settings gives a result of less than $2000.

all of the charts are completely separate and do not reference any other charts and this is on 5 minute bar charts.

Why does my result differ so significantly???

And this is not limited to just this chart; ALL the charts, when tested alone with no other charts return vastly different numbers as soon as multiple charts are opened.

This is an ACSIL strategy and there are NO compilation errors.

Please explain.
Date Time Of Last Edit: 2014-02-02 04:14:21
[2014-02-02 04:28:59]
Sierra Chart Engineering - Posts: 104368
We cannot explain this unless we were to run a test using your Chartbook and custom study. It makes no sense based upon how you describe it. What kind of back test are you running? Is it a replay or bar based back test. And if it is a replay back test, do you have For All Charts in Chartbook enabled on the Replay Window?

And what happens if you just open one additional chart? And not 30.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
[2014-02-03 02:28:01]
hairyMug - Posts: 31
1. type backtest is "accurate trading system replay backtest"
2. YES, the results change when just 1 chart is added.
3. "For All Charts in Chartbook" is not selected; If I do select it, I get an even different result...

Also, the "bar-based" back-test gives an even different result.

None of this makes any sense as I only trade when "sc.GetBarHasClosedStatus() == BHCS_BAR_HAS_CLOSED"...

I use three "persisted" variables in my function and observed that the location of the declaration of the seems to have changed the results also. The only reason I mention this is that I also had a problem where the "persisted" values caused a continuous "CPU exception" error which was cause by assigning a float value to a double "persisted" value without "casting" it to double...

I have spent a great amount of time finding "undocumented features" (bugs) of ACSIL and I am really starting to question if SC is even capable (or stable enough) to handle an "auto-bot" trading system...

[2014-02-03 05:14:00]
Sierra Chart Engineering - Posts: 104368

Also, the "bar-based" back-test gives an even different result.

3. "For All Charts in Chartbook" is not selected; If I do select it, I get an even different result...
This makes sense and should be expected. Refer to:
https://www.sierrachart.com/index.php?l=doc/doc_Backtesting.php

I have spent a great amount of time finding "undocumented features" (bugs) of ACSIL and I am really starting to question if SC is even capable (or stable enough) to handle an "auto-bot" trading system...

What are these? From our perspective there are none. And we take exception to this. Rather than saying this, provide us a simple example of a problem so we can reproduce it and resolve it. Otherwise, nothing is accomplished here. At this point you have provided no evidence of a problem. There has been only one recent ACSIL issue and it was identified and resolved and documented in the What Is New notes for 1086:
https://www.sierrachart.com/index.php?l=doc/Whats_New.php

We cannot help you unless we get a simple example to test.

I use three "persisted" variables in my function and observed that the location of the declaration of the seems to have changed the results also

This makes no sense. This could not possibly have any effect and if it does, this must be a code issue on your side. This could not be a cause.

The only reason I mention this is that I also had a problem where the "persisted" values caused a continuous "CPU exception" error which was cause by assigning a float value to a double "persisted" value without "casting" it to double...

This makes no sense either and we just tested this and encountered no exception at all. Here is the actual code tested:

  //Persistent variable test
  double &DoubleReference= sc.PersistVars->Doubles[0];
  float TestFloatValue = 3.0f;
  DoubleReference= TestFloatValue;//No exception occurs

If you are encountering an exception with code like the above, it is a very strong indicator, something is wrong with your code or the compilation of the code. There should be no trouble with this whatsoever. None at all. This could not be a Sierra Chart issue at all. There is 0% chance of that. Sierra Chart has no involvement of the compilation or running of that code, other than actually calling the function. The compilation is handled by a separate compiler.

Keep in mind if you are corrupting memory, you can run into all kinds of unusual problems that occur later on in your code and effect Sierra Chart code and are extremely hard to find. And we suspect this could be what is going on in your case. It does seem like there is a bug in your own code and then it appears like a problem within Sierra Chart. We cannot debug your own code. This is why you need to work with very simple code examples to reproduce a problem and then if there is no problem, then you know that the problem most likely will be in your own code.

Start off with a simple example then gradually add to it, until there is a problem. Then determine where the source of the problem is. If it is in your own code, resolve it . If you believe it is within Sierra Chart, then give us a simple example to duplicate the problem.


If you need help or think there is a problem, provide us a simple example to work with. It does not help at all to say "Please explain" to something programmatically you are doing and obviously is complicated, and not giving us anything to work with. And then you make comments about problems with out any code to back them up. We cannot respond to this differently than we have.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
Date Time Of Last Edit: 2014-02-09 11:42:27
[2014-02-03 06:10:34]
ejtrader - Posts: 688
hairyMug - Could be a different issue - but in case if you are using persistent variables in some kind of global variables/pointers form "AND" using freedll=1, you are likely to run into CPU exception issues. I run into similar CPU exception issues ( can reproduce at will ) - but freedll=0 would resolve the issue in my case.

Thanks
[2014-02-03 06:14:50]
Sierra Chart Engineering - Posts: 104368
Post 5 is absolutely correct. Additionally, without us having a simple example to work with, we have no idea what is going on.

You need to establish a very clear procedures on your side to handle problems. Use a very logical step-by-step debugging approach. Start with something simple and then gradually add to it until you encounter a problem. It is up to you to clearly isolate the source of a problem. At this time we are not aware of any outstanding problems. We provide a very very solid custom study and trading system interface. It may not have all of the functionality some may want. And back testing is not really going to give you results that are the same as live trading, but it is most definitely solid and works as designed. And there definitely are no bugs which would cause any kind of stability issues.

Contemplating this more, we do believe the problem is in your code. It really does not make any sense whatsoever, that opening another chart is going to affect the back test results especially if you have Accurate Trading System Back Test Mode selected and evaluating on bar close. You should get a consistent result each and every time. What you are describing is indicative of memory corruption. Although all of this depends on what your trading is doing.

And if you do believe there is a problem on our side, provide us a simple example code to duplicate the problem along with steps.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
Date Time Of Last Edit: 2014-02-09 11:36:08
[2014-02-09 03:10:20]
hairyMug - Posts: 31

Congratulations!

You have lost a customer...

Make no mistake; You are NOT the only solution and your poor support attitude has done the job...

I have already found a MUCH better tool with more features and a polite and helpful support staff.

... Enough said ...

BTW: If you software is so bug-free, Why do you do a new release every day???


[2014-02-09 04:02:28]
Sierra Chart Engineering - Posts: 104368
Very good. Please move on. Clearly that is what is best for you. We do not have a poor support attitude. We helped you as best as we can and also provided you with a procedure to isolate the source of the problem. With a lot that you have said here, we really could not do much to help because you did not give us any code examples and steps to duplicate the issue.

Nevertheless, we did run tests for two of the problems mentioned and did not encounter them . We also considered them carefully to see if there could be any problem we were not able to test for as well. We determined that there were no problems.

Please go ahead and use your better tool.

The issue is on your side. Not ours. We are very helpful on the support board. And has been said, if you do not give us the ability to help you, we cannot help.

We do not necessarily release a new version every day. Sierra Chart is being continuously improved and there are new features added, and this is why there are frequent releases. That does not mean with functionality you are using, there was a bug. That has not even been established.


BTW: If you software is so bug-free, Why do you do a new release every day???
Even a comment like this, is reckless, is not even accurate, and is inferring an incorrect conclusion without basis.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
Date Time Of Last Edit: 2014-02-09 04:47:00
[2014-02-09 04:33:33]
Sierra Chart Engineering - Posts: 104368
Maybe we should not focus too much on the persistent variable issue you mentioned, but that we know absolutely is an indicator something is wrong with your own code or the compilation of the code. Here is a complete test:


//This is the basic framework of a study function.
SCSFExport scsf_SkeletonFunction(SCStudyGraphRef sc)
{
  // Section 1 - Set the configuration variables and defaults
  if (sc.SetDefaults)
  {
    sc.GraphName = "Persistent Variable Test";
    
    // During development set this flag to 1, so the DLL can be rebuilt without restarting Sierra Chart. When development is completed, set it to 0 to improve performance.
    sc.FreeDLL = 1;

    sc.AutoLoop = 1; //Auto looping is enabled.
    
    sc.Subgraph[0].Name = "Name";
    sc.Subgraph[0].DrawStyle = DRAWSTYLE_LINE;
    sc.Subgraph[0].PrimaryColor = RGB (0, 255, 0);
    
    sc.Input[0].Name = "Float Input";
    sc.Input[0].SetFloat(0.0f);
    
    return;
  }
  
  
  // Section 2 - Do data processing here
  
double &DoubleReference= sc.PersistVars->Doubles[0];
float TestFloatValue = 3.0f;
DoubleReference= TestFloatValue;//No exception occurs
sc.Subgraph[0][sc.Index]= DoubleReference;
  
}

This code has been run extensively without any exceptions. And there should not be any exceptions. It would make no sense if there are exceptions. We also did a test using the Sierra Chart provided compiler. That passed the test without any problems. We know what we are talking about here.


I have spent a great amount of time finding "undocumented features" (bugs) of ACSIL and I am really starting to question if SC is even capable (or stable enough) to handle an "auto-bot" trading system...

And with a comment like this without any kind of basis (code examples and steps to reproduce), we will defend our position, look into as best as we can the issues described and confirm that there are no problems. And we did confirm that there was not a problem with the described issues. And why do you not point out these particular "bugs" with a simple code example and steps to reproduce. If you consider all of this a "poor support attitude" then so be it because what could we do if you do not provide a simple code example demonstrating an issue, or demonstrating a relatively simple question for us. Obviously we cannot debug your code we cannot even see and we cannot debug a complex function you have created, that would be your responsibility.

In regards to this particular issue described in this thread with consistency, the primary consistency issue lies within your own code and not Sierra Chart. Refer to this section here:
https://www.sierrachart.com/index.php?l=doc/doc_Backtesting.php#BackTestConsistency

If you test the Sierra Chart provided trading systems you will find consistency between them every time. What does this show you?
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
Date Time Of Last Edit: 2014-03-08 06:03:36

To post a message in this thread, you need to log in with your Sierra Chart account:

Login

Login Page - Create Account