Search | User Listing Vidsite | Calendars | Quotes
Home Page ->  Optimizing OmniVest -> Development Discussions -> View ThreadLogon (or Register, or Join TradeTight)

You are logged in as a limited-access Guest.To join TradeTight, first read the info in the Organization & Content room, then click the link above. 

Matthew 25:16,21 (ESV) … He who had received the five talents went at once and traded with them, and he made five talents more … his master said to him, ‘Well done, good and faithful servant. You have been faithful over a little; I will set you over much. Enter into the joy of your master.’


Pitfalls of Automated Overtuning
Jump to page : 1
Now viewing page 1 [50 msgs/pg]
Jump to room :
JimDean
Posted 7/6/2015 3:16 AM (#6376)
Subject: Pitfalls of Automated Overtuning



Owner/Admin

Posts: 3579
20001000500252525
Location:
USA: GA, Lawrenceville
Amongst other methods, one reliable way to identify when a strategy has been "over-tuned" (ie so curve-fit that true unseen out of sample "past the HRE" future will *not* echo past incredible returns) is to watch for very very low frequency of trades with very very high hit rates. That condition can (but may not) be indicative of a combination of rules that in effect have been "taught" to seek out sweet spot trades during the historical window that won't necessarily repeat. That is, a crystal ball is in play.

This can happen *without your realizing it* when zillions of permutations through large focus lists with large numbers of systems are used. It is hard to emphasize this too strongly, as we obtain more and more tools such do thousands or millions of iterations "in the background" without much "thoughtful considered decision-making" on our part.

I am not claiming or even warning that this is happening with some of these very high-performing portfolios - but the characteristics and procedures that I'm hearing about seem to make them good candidates for this effect.

If you are concerned about this being the case, the wise course is to stop, step forward, and NOT squint … that is " SFAF" (Step Forward and Focus) … ask yourself "WHY" is this working so well? What rules and methods are in play that actually choose the entries and exits, that logically should be randomly repeatable, under foreseeable and understandable and likely-to-repeat market conditions?

And - keep in mind that repetitive cycles of "back" testing with measurements and decisions made using "forward" results is effectively including the forward test period in your curve fitting process. That is, using the same forward test period and symbol set repeatedly will (fairly rapidly) destroy the effectiveness of the forward test as an unbiased metric.

Selah.
Top of the page Bottom of the page
JimDean
Posted 7/6/2015 3:34 AM (#6377 - in reply to #6376)
Subject: Pitfalls of Automated Overtuning



Owner/Admin

Posts: 3579
20001000500252525
Location:
USA: GA, Lawrenceville
I believe one key component to avoiding this is the use of Dynamic Lists based on personality-pattern algorithms. By their very nature, they are seeking out symbols which are "friendly" to the trading-logic they are paired with. They offer slam-dunk-sure means of knowing that future past-the-forward-test performance will be to some degree repeatable, since in those UFO's (unknown future occasions) the Dynamic List rule will still be in play, seeking out symbols with similar past personality profiles.

The Market States rules are just the beginning of this (though an incredibly powerful and flexible start). They are intentionally "generic", so that they can be applied (with THOUGHT-fulness) to most any kind of strategy or instrument.

But the personality-pruning-process (P3) can be adapted much more - rather than seeking out a "state" that is "conducive" to a strategy, rules can be created to seek out "price-progress-patterns" (P3 again) to help assure good reward/risk potential for specific types of strategies. This is a concept I've worked with for years (you'll find lots of conceptual posts about it in the distant-past threads here and in the N forums and yes even way back in the Worden forums under the "Sir Tanstaafl" nom-de-plume) … and now that I've developed a mechanism for applying it in a testable fashion for the full spectrum of platforms, I'm looking forward to resurrecting it for general consumption.
Top of the page Bottom of the page
JimDean
Posted 7/6/2015 3:46 AM (#6378 - in reply to #6377)
Subject: Pitfalls of Automated Overtuning



Owner/Admin

Posts: 3579
20001000500252525
Location:
USA: GA, Lawrenceville
Another key component to this is the use of smart, robust and flexible EXIT rules.

Strategies with "simplistic" exits sometimes can be highly tuned to perform well, by seeking out sweet spots in the historical period. But as has been stated earlier, the better this looks during the repetitive back+forward test period, the danger is that the "sweet spot" patterns that produced those ideal entries might not repeat.

A robust exit is one that keeps a TIGHT but SMART reign on trades, whether they start off in a sweet spot or not. Please read that sentence a few times and memorize it. We hear so much about "not worrying about stops" and "letting the trade breathe" and "letting your profits run" … that we forget how great a burden that approach puts on getting "crystal ball" entries.

I'd far far rather have a strat with an understandable entry approach that finds trades fairly often, with a "big brother" Market State filter to keep me from being blind to macro conditions, and a "big sister" P3 scan to keep me from making the wrong "friends" as to my choice of FL symbols, all coupled with a SMART exit that lets things move *when* they are moving well, but "guillotines" them quickly if it turns out that the system entry was not-so-good on that occasion.

This is "my style" of trading and of strategy development - "eyes wide open".
Top of the page Bottom of the page
Jump to page : 1
Now viewing page 1 [50 msgs/pg]
( E-mail a Link | Printer Version | Search Room )

Owner of site: Jim Dean -- Forum content is confidential, and may not be distributed without written permission.