Support Board
Date/Time: Tue, 26 Nov 2024 09:49:57 +0000
[User Discussion] - Replay is skipping days?
View Count: 1391
| [2023-12-06 15:29:41] |
| Sawtooth - Posts: 4120 |
|
Some very brief responses: 1) If a pattern or mathematical relationship or market structure repeats based on past historical context, please help me understand why there is such a high rate of failure across retail algo traders. Every algo based on a pinbar or a 123 pattern or whatever should make millions based on how appealing they look when we scroll the chart backwards :-
Your eye only sees the good entries when the conditions are met, but the algo sees all occurrences when the conditions are met.2) What is the root cause of algos to stop working after a while if they are based on such proven statistical edges?
Because the market is dynamic and the algo is rigid.3) How long and how often should an edge have a repeatable and statistical frequency for it to be considered a robust edge?
A repeatable edge could end at any time.At a holistic level, what gives us confidence that a strategy would succeed?
An autotrading strategy will always eventually fail, so the confidence of success is to use its signals with discretion, not with automation.
|
| [2023-12-08 22:53:18] |
| j4ytr4der_ - Posts: 938 |
|
BTW, with regards to replay speed... here's one that is obvious but I totally just accidentally overlooked and it cost me hours of chasing my tail. If you're using a spreadsheet system, ALWAYS set it to as few rows as you can possibly get away with. I am always set to only 2 rows, but earlier today I was exploring something on a historical chart where I set it to 20,000 rows just so I could quickly scroll back and look at some things. I forgot it was on 20,000 and actually started running some replays, and the speed was absolutely useless - like worse than realtime on Every Tick replay, and barely better than realtime if set to Accurate System. I finally remembered that I had 20,000 rows enabled, and as soon as I put it back to my usual 2 rows... all the speed & performance (such as it is) that I normally expect, came back. Silly mistake but an easy one to make! |
| [2023-12-16 00:27:22] |
| User61168 - Posts: 403 |
|
the confidence of success is to use its signals with discretion
Thanks Tom. Sadly, for me, Jury is still in deliberation mode when it comes to algo trading. Any human discretionary action still have some rules engine taking certain set of parameters as input/filters which could be programmed. All such discretionary techniques that I know of and tested show failure or performance decay. I don't know what the right or wrong answer whether to continue or end my automation journey. There has to be a better approach in strategy development than to expect algos to decay over time requiring life-long tweaks or "calibrations". I went into automated trading in search of freedom away from starring the charts and it is turning out to be much more hard work than manual/discretionary trading lol p.s. I have yet to get into all the advanced order flow, TPO, valume/market profile related techniques so maybe that's the next step for me but I am literally scared to get sucked into the complexities of footprint and volume profile stuff for another few years to gain mastery :-( Date Time Of Last Edit: 2023-12-16 00:28:27
|
| [2023-12-16 00:47:21] |
| j4ytr4der_ - Posts: 938 |
|
I think you're overlooking a critical point that has been made above. NO non-discretionary system will just keep always working. So if your idea of "success" is "A system that will make money forever without touching it again" then I'm afraid you're on the wrong path. Also I think you may have a great misunderstanding of what order flow is. Forget what all the books and websites say, as they totally overcomplicate the whole thing. You don't need to know all that much really, and footprints are SO much easier to understand than candlestick and price patterns etc. (not to mention, they are actually meaningful with regards to what's happening in the market!) If you're trying to build systems around patterns of price, price-based indicators, averages/means, etc. then I believe you are in for a world of frustration. Order flow is literally the only thing that moves the futures markets. Volume, at price. Without order flow, there is no movement, and if you learn just a handful of things you will see your entire worldview change as if someone removed a blindfold. I say this in all sincerity as that was my own experience and I could never go back. For the first time in over 20 years, I can see "light at the end of the tunnel" so to speak, and I'm finally able to create setups that have an actual, objectively measurable edge, are automate-able, and work in live markets. If you need a pointer, send me a DM and I'll gladly share it. But I would say you're shying away from exactly the things which could make all the difference for you. |
| [2023-12-16 03:33:13] |
| User61168 - Posts: 403 |
|
"A system that will make money forever without touching it again" I am approaching markets with a research mindset of finding a universally acceptable solution that works in both trending and ranging markets...i.e. in both Divergence from Mean and Reversion to Mean from a price level or setup or indicator or whatever technique that is available out there. Before diving into strategy development I tried to understand why strategies stop working and my conclusion is the fact that strategies fail because1) there is no easy non-lag way to identify the transition between the two. Atleast not that I am aware of. 2) reliance of history i.e. looking at the past to predict future. After testing all the (historical) price based indicators out there, my conclusion was to narrow down and focus primarily on the current bar and ohlc of previous bar with no price based indicators or derivatives of price. And I have approached volume analysis but only at the rudimentary level using VWAP/VAH/VAL/Cumulative Delta/Large volume study whatever is readily available in the standard package of SC. My efforts did not reveal any positive edge with these volume based studies. With that said, my definition of "order flow" (powered with limited knowledge) is to look at the buy and sell orders at the centralized order book i.e. price ladder and the resting limit orders, stare at the rapid price moving up and down the ladder and derive "some" theory or logic on what type of volume drives the price up and down via exhaustion and absorption of resting orders taken out by market orders. I have tried staring at the price ladder and physically could not stare for more than 5 minutes without feeling dizzy with flashing numbers everywhere (rapid moving numbers did not go well with my eye sight and concentration so there is a physical human limitation in my ability). If you need a pointer, send me a DM and I'll gladly share it. But I would say you're shying away from exactly the things which could make all the difference for you. ... I agree and also realize the thick-headedness in my chosen approach in this journey. I would love to take you up on the offer to share your knowledge on orderflow especially if it has proven benefit to your own growth. If you can show why a bar opening at price x has the potential to move to x + y ticks in a given direction or the signature in orderflow to depict a reversal after x+ zticks, that would be truly impressive :-)
Date Time Of Last Edit: 2023-12-16 03:34:32
|
| [2023-12-16 06:16:27] |
| j4ytr4der_ - Posts: 938 |
|
I'm going to be especially blunt here, I hope you will take it as constructive criticism =) 1. There is no system that works in both "trending" and "ranging" markets. By definition, this is impossible. They are two entirely different situations that require entirely different ways of trading. If this is your quest, it's no wonder you've struck out! This will never work. 2. Price cannot revert to an average. EVER. The average converges on price. The average is an output OF price, not an input TO price. Price generates averages. Averages do not generate prices. Really think about this if it's not instantly obvious why "mean reversion" is a terrible idea to trade mechanically. 3. It is absolutely possible to say with a certain likelihood, that a given level will trade. This is routine in the order flow world (well, for those who understand it anyway). There are certain setups that happen just about every single day (a few days per year are the exceptions) where you can clearly see the level in question, understand why it is most likely to trade, and then time your entry by watching the volume at price in order to gauge where the passive and aggressive traders are doing business, and join in at an appropriate time. If you want to automate a trading system, this is how you want to focus your efforts. Happy to point you (or anyone) to some helpful resources, just shoot me a message, that invite goes to anyone reading this. |
| [2023-12-16 10:07:34] |
| User61168 - Posts: 403 |
|
Thanks. I welcome "constructive criticism" #1 - Unless I can prove this to be wrong, I have no choice but to agree with you #2 - Agree and I do not deal with averages. #3 - There are certain setups that happen just about every single day I also only look for setups i.e. market behavior that happen each and every day. Taking every signal results in DD as frequency is unpredictable so volume might be the missing key to filter out the bad ones. Good discussion.
|
| [2023-12-16 17:25:09] |
| j4ytr4der_ - Posts: 938 |
|
Just to clarify, it was your mention of "mean reversion" which I took to mean you're using at least a mean, if not normal moving averages. Mean reversion is an illusion. The mean converges on price, not the other way around. The times price actually DOES move towards a distant mean (which is also continuing to converge on price), is the illusory trap.
|
| [2023-12-17 03:55:39] |
| User61168 - Posts: 403 |
|
I understand what you are saying. It's the movement of a "mean" that causes illusion that is the reason why I do not use averages. I like to use a constant i.e. point of reference or atleast a fixed reference which could be anything you can imagine to be not moving with price. A level, Session open, a recent or new HH/LL, etc. With that definition of median, price is always moving away from or towards it. Price is always either bouncing off or breaking thru this predefined level hence the diverging or reverting terminology with no consideration of whether price is retracing or pulling back or continuing as those are all contextual. my use of the term divergence or reversion refers to the inherent move that price has to make all the time to support the ebb and flow of orders from market participants. Hope this makes sense.
|
| [2023-12-17 06:03:33] |
| j4ytr4der_ - Posts: 938 |
|
Gotcha. Usually when people say "mean reversion" they literally mean the expect price to return to wherever the mean of price happens to be *right now*, without regard for the fact that it's moving and price may very well reach it - because it moved to price. I would say you're on the right track with looking for static levels that price will revisit. This is now the cornerstone of every bot idea I develop. What is a level that I can prove has a reasonable frequency of being reached, that can also have order flow to support it. Things like the IB high/low, overnight high/low, certain TPO artifacts, etc. It starts from the target, and if the historical data supports the idea that the level in question does get re-traded with high frequency, then it becomes worth looking deeper and developing a setup to trade it. |
| [2023-12-17 21:05:36] |
| User61168 - Posts: 403 |
|
Glad we are on the same page now lol if the historical data supports the idea - I do not look at History or do any historical statistical analysis or stare/analyze charts looking for patterns (either recurring more frequently or trying to find some unique secret pattern etc). This is the only way I found to avoid biases resulting in curve fitting. That is the reason why I run my replays with only one fat bar on the chart and believe me, it is time consuming and I have run atleast 5000+ replays primarily focused only on NQ. Started little bit of ES this year. Before Denali came out, I was working on only couple months of replays data with CQG and that really sucked. As I said, this journey has been cyclic with bouts of intermittent success but nothing glorifying to share without consistency. As they say, making money is the easy part. keeping your winnings has been the most difficult part in my trading endeavors. That is why whatever I see on the net does not impress me at all unless I see a consistent record of performance (which, sadly, no one wants to disclose to gain credibility as step1). Just like you, even I have been dabbling in stocks since 1998 and spot forex crap from 2010-2015 and futures from 2018-till date and I can happily share that I have made several million in profits just to give it all back lol. The only way, in my experience, to not lose your winnings is to stop trading at the high watermark for the given duration. I eventually got out at breakeven after wasting 5+ years in forex. Lets hope I am able to break this cycle with futures lol. Being an intraday swinger, every single strategy I have tried in futures has to come with a daily PnL stop at session close i.e. forcing me to exit losers and keep my daily profits. It's my key requirement to control greed/fear. No matter what anybody says, this sh*t is not easy... simple for sure but not at all easy to generate income solely from trading consistently year after year. Anyways, I have shared enough about me :-( Date Time Of Last Edit: 2023-12-17 21:07:54
|
| [2023-12-17 21:09:23] |
| j4ytr4der_ - Posts: 938 |
|
You misunderstand me. I'm not suggesting looking for "patterns" in historical data. I'm saying look for events that happen with a measurable frequency. Breaks of the IB high/low, breaks of the overnight, etc. Those are "true" levels, meaning they are defined by the market participants of the day. And they break with frequency that doesn't change much over many years. This is the basis of setups that can be gamed with a statistical edge. Forex was a waste of your time from day 1 because it is a 100% manipulated market (retail forex that is) and your broker is your enemy. I believe you are currently chasing your tail and looking at things from a flawed perspective, I very strongly suggest you check out the resources I DM'd you about. Seriously... you should change your thinking IMO. |
| [2023-12-17 21:27:27] |
| User61168 - Posts: 403 |
|
resources I DM'd - just to say it, I am looking for specifics with using orderflow and what you shared is part of your paid/subscription services. I am getting close and looking for just one specific thing which I am unable to solve with just price. I feel I might need volume analysis for exit trigger. I have my edge discovered with 1) entry trigger, 2) trade direction and 3) pre-defined stoploss. Where I need some work is in two areas: 1) optimize for maximum profit taking (and avoid leaving profits on the table). 2) trade frequency to address/reduce consecutive losses to keep my max DD under a certain limit. (p.s. I have achieved all this without using any indicator/derivative of price or using historical data beyond the previous and current ohlc...I feel my approach to markets is rock solid but ultimately, it's the outcome that matters. Until then, this remains a good research project in data science lol) Date Time Of Last Edit: 2023-12-18 03:13:05
|
| [2023-12-17 21:38:59] |
| j4ytr4der_ - Posts: 938 |
|
And for the price of 2 ES ticks, you could have information which you can implement immediately and might just change everything for you. I'm just trying to lead you to what you claim to be seeking, but if you want everything to be free... that's ok, I wish you well. You appear to know what you need to know so, go do it.
|
| [2023-12-18 00:02:55] |
| User61168 - Posts: 403 |
|
I appreciate it. It's not the cost that I am avoiding or expecting stuff for free. I just fear dipping my toes into the complexities as I came into futures trying all kinds of complex strategies. My #1 objective is to keep it simple that is why I am using only simple alerts and stayed away from spreadsheets or ascil or advanced package. I feel footprint and profile charts will suck me into another bundle of strategies which I find unnecessary at this point in my journey. There is already lot of free stuff regarding orderflow on youtube and paid courses on udemy for under $50. If and when I decide to step into order flow, it will be with my blinders on, looking for every specific, targeted and unique information to address my needs. It won't be about learning order flow or associated strategies. Hope this makes sense :-) Date Time Of Last Edit: 2023-12-18 00:14:36
|
| [2023-12-19 18:25:31] |
| sgne - Posts: 105 |
|
I find it worthwhile to use replay to iron out order placement logic, but to generate possible trading results I like to run code I wrote on bar tick files I assemble. I like to assemble my own tick files because that way I'm not stuck with exchange bid ask values or with SC calculated bid ask values. I can set or assume any theoretical bid ask value I want. Also, bar tick files can be made as complex or simple as desired. With very simple bar tick files containing only ticks making new highs and lows during a bar, trading code can run through months of data very quickly, so much quicker than SC replay ever could. And even bar tick files that are more typical can be run through quite a bit quicker than SC replay. And there won't be skips, unless your code is wrong. Date Time Of Last Edit: 2023-12-19 18:27:14
|
| [2023-12-19 18:34:01] |
| User61168 - Posts: 403 |
|
Great idea sgne. I tried doing something similar in excel but ran into limitations with max number of rows. Just so I understand what you mean by “bar tick”, is that the same as using 1tick range bar setting in SC and then exporting data into text files? Also, could you shed some direction on what tool or coding language are you using to speed u the testing process? It would be greatly beneficial to have some faster alternative process than waiting for hours and days to complete market replays.
Date Time Of Last Edit: 2023-12-19 18:35:24
|
| [2023-12-19 19:09:58] |
| sgne - Posts: 105 |
|
Yes, I think it will be quite hard to use Excel for this task. Here are some more details on what I do. First I extract the tick data to large text files from binary 1-tick SCID files, using either Python (see the relevant thread on this board), or the language I'm most familiar with, which happens to be AHK. This gives you large files for each 3-month period (in the case of NQ or ES). Then, based on the bars of the strategy I'm working on, I make text files from the extracted tick data, one text file for each bar. That's what I mean when I use the term bar tick file. So now we have many smaller text files with tick data. As of now I use a custom chart bar DLL file to write bar data to a file, but the Write Bar Data study can do this as well. The price values and timestamps are used to assemble matching bar tick files. Then I write code to simulate trading replay, which reads and runs through the bar tick files. |
| [2023-12-20 09:58:58] |
| User61168 - Posts: 403 |
|
Great insight. Thanks for sharing. In essence, you have your own trade sim execution engine outside of Sierra reading bar tick files and running your strategy outside of SC.. Interesting technique
|
To post a message in this thread, you need to log in with your Sierra Chart account: