Tools, Techniques, and Training for Nirvana Platforms 
OmniTrader, VisualTrader, OmniFunds & OmniVest 
 
 


John 6:27 (NIV) ... Do not work for food that spoils, but for food that endures to eternal life, which the Son of Man will give you. On him God the Father has placed his seal of approval. 
Nine Market States  Simplified Formulae  

JimDean 
 
Owner/Admin Posts: 3889 Location: USA: GA, Lawrenceville  The goal of this thread is to propose and refine simple OmniScript formulae which can be used to identify Symbols that fit into nine market states. It's useful for OmniScan, FilterBlocks, Color Charts, and of course OmniVest Dynamic Lists. Introductory background remarks about this topic are provided in the OmniTrader forum. Click here to review them. Please post "contributions" to the thinking process in this forum ... I will try to answer questions that are posted both here, and on the various Nirvana forums, in this thread, so that everything remains centralized for future reference. There are several other threads that are conceptually supportive and provide useful background for this one ... other links will be found in subsequent posts, that provide supporting info relevant to those discussions: Click here for another thread about Market States. Click here for another thread about Market Sentiment. Click here for a "map" of history by different Market "modes", for backtesting. Click here for more info about the MTV tools (incl purchase options), and the Scripts they generate, and see 6/4/15 post below regarding the MTV "solution" (with slides) that I presented at the 2015 Bash ... The "DeanMarket" algorithms presume that useful Market States are permutations of: Three Trend states: Bull, Bear, Flat ("flat" = short for "not Bull or Bear") Three Volatility states: Wild, Calm, Norm ("norm" = "not Wild or Calm") Thus there are nine States defined by the permutations of those: BullCalm, BullNorm, BullWild, FlatCalm, FlatNorm, FlatWild, BearCalm, BearNorm, BearWild Note that the "natural progression" between states is NOT in any particular order, but that we might reasonably expect that trends OFTEN move from Bull to Flat to Bear to Flat to Bull, and volatility OFTEN moves from Calm to Norm to Wild to Norm to Calm. Sometimes the intermediate states (Flat / Norm) are so short as to be effectively nonexistent, and sometimes they can be very extended. Please, in this thread, let's stick with that terminology and paradigm ... there certainly are other ways of looking at it, but here let's channel the work along those lines. There are FIVE MORE posts in the initial presentation of the procedure ...  
JimDean 
 
Owner/Admin Posts: 3889 Location: USA: GA, Lawrenceville  My research in this works with perspectives on the Moving Average of Price, based on the principle that we don't want to get too far away mathematically from "simple price action". === MA Calc Method === Virtually any MA calc method could be used ... I'm partial to WMA's or EMA's over SMA's, and I've come to appreciate DCMA's (Ehlers DeCycled MA's) over ZMA's, at least for medlarge periods. WilderMA's provide very good smoothing but have significant lag. Click here and review items #15 regarding the relationships between the MA types. For simplicity's sake, I'll use standard EMA's for the preliminary discussions here, but the final solution might home in on a different type. Note that Wilder, EMA, and DeCycled all can be calculated using appropriate variants of the EMA, so they are all OmniScriptable. === How Many MA's === I think that a much better measure of market state comes from comparison of multiple MA's rather than trying to find a "single best" MA to act as a bellweather. I've had success with sets of three, and sets of four ... with four, the calcs are more involved but the principles are the same, and four actually can be used with appropriate logic (OLanglevel complexity) as the basis for a fairly decent Entry/Exit strategy ... but that's not what we're focusing on here. So, for simplicity's sake, I'll orient my initial discussion around a set of THREE MA's. I definitely feel that more than four does not add any benefit, and less than three can create too much chatter. We will also use the Price of the bar as yet another MAlike dataseries to include in the mix (for Stacking only) ... I'm strongly recommending that the Guts price (H+L+O+O+C+C)/6 be used in that case ... it is demonstrably the smoothest and most representative price which can be derived from a single bar, without any lag. Maybe we will use it, maybe not ... but by using Guts vs Close or Typical, etc, we will help prevent logicwhipsaws in our State determination. === MA Characteristics === I believe that there are TWO primary characteristics of MA's that are useful for Market State evaluations: Slope and Stacking. Slope is found from a short Linear Regression of the MA, and Stacking examines whether "faster" MA's are consistently above/below "slower" MA's. I've looked at a lot of other attributes such as MACD (relative velocity of the MA's  ie the distance between them), MACDH (relative acceleration of the MA's  the rate that the distance between them is changing) ... but both of those are too unstable for State calculations. Other things like Relative Strength vs an index or Industry Group, or ROC (rate of change) versus some arbitrary point in history could be considered but they add additional degrees of freedom that move us away from the "stay close to watching only what price is doing" mantra. === Lag vs WhipSaws === Some might argue that MA's are lagging rather than leading. I believe that by using Slope evaluations, we include a degree of "projection" that offsets the lag, sometimes. If lag truly ends up being an issue, then we can gravitate to DeCycledMA calc's, or ZMA (though ZMA's are pretty "bumpy". However, FOR THE PURPOSE OF FINDING MARKET STATE, I don't think that lag is as much of a "boogeyman" as it can be, when trying to find good entries and exits. The true "dastardly dan" in the market state search is WHIPSAWS ... that is, spurious flips from one state to another and back again. This often happens when we try to wring all the lag out. === What are we trying to accomplish? === To repeat ... this is NOT a trading system ... we are not trying to find entries and exits. Rather, we are trying to identify Symbols whose priceaction indicates it's in one "domain" or another ... once we know that domain, we can find an appropriate OVest trading system to use with it. In fact, we might even attach the SAME STATE RULE as a condition to the System, based on SPX or something like that. But that's another thread :~)  
JimDean 
 
Owner/Admin Posts: 3889 Location: USA: GA, Lawrenceville  The Slope and Stacking concepts can be applied to analyze BOTH the Trend and the Volatility components of the Market State. To analyze Trend, we look at the Slope/Stack of MA's of the PRICE. To analyze Volatility, we look at the Slope/Stack of MA's of ATR. As I mentioned earlier, there are a lot of different possible schemes ... especially re the volatility categorization. I am trying to keep this proposed solution SIMPLE both to understand (for "intellectual comfort level") and SIMPLE to implement in an OmniScript formula. OScript is much more powerful than most people realize, but it does have practical limits since it's all just one long line of code, not multiple lines. Let's define some terms for the MA's. The example will use three MA's, and ignore Price (as a simplified starting point). There are three MA's of Price: FastPrc, MedPrc, SlowPrc ... and three of Volatility: FastATR, MedATR, SlowATR. We are not actually defining variables here ... OScript has no variable assignments. But the "handles" will help explain the logic. Also, GutsPrc is simply the Guts Pricepoint for that bar (defined earlier). Stacking Rule for BULL Trend: GutsPrc >= FastPrc and FastPrc >= MedPrc and MedPrc >= SlowPrc ... that is, the MA which has the smallest #periods is "on top", the MA with the largest #periods is "on bottom", and the other one is inbetween them. Stacking Rule for BEAR Trend (simple reversal): GutsPrc <= FastPrc and FastPrc <= MedPrc and MedPrc <= SlowPrc ... note that "FLAT" trend Stacking is any situation that does NOT satisfy either the Bull or Bear rules. Slope Rule for BULL Trend: LRS(FastPrc) >= 0 and LRS(MedPrc) >= 0 and LRS(SlowPrc) >= 0 ... where "LRS" is the LnReg_Slope function in OT, using a small window of 35 bars duration (best=3 as long as MA is not "bumpy") Slope Rule for BEAR Trend: LRS(FastPrc) <= 0 and LRS(MedPrc) <= 0 and LRS(SlowPrc) <= 0 ... note that "FLAT" trend Slope is any situation that does NOT satisfy either the Bull or Bear rules. === COMBINING THE TREND RULES: BULL: GutsPrc >= FastPrc and FastPrc >= MedPrc and MedPrc >= SlowPrc and LRS(FastPrc) >= 0 and LRS(MedPrc) >= 0 and LRS(SlowPrc) >= 0 BEAR: GutsPrc <= FastPrc and FastPrc <= MedPrc and MedPrc <= SlowPrc and LRS(FastPrc) <= 0 and LRS(MedPrc) <= 0 and LRS(SlowPrc) <= 0 FLAT: any time that both the Bull and Bear rules fail === PARALLEL RULES FOR VOLATILITY: WILD: ATR >= FastATR and FastATR >= MedATR and MedATR >= SlowATR and LRS(FastATR) >= 0 and LRS(MedATR) >= 0 and LRS(SlowATR) >= 0 CALM: ATR <= FastATR and FastATR <= MedATR and MedATR <= SlowATR and LRS(FastATR) <= 0 and LRS(MedATR) <= 0 and LRS(SlowATR) <= 0 NORM: any time that both the Wild and Calm rules fail (Reminder ... the ATR variables above represent the stacking and slopes of MOVING AVERAGES of the ATR, not the ATR itself. The "raw dataseries" for ATR MA's is ATR(14) ... or some other number of periods ... but the 14 is not the thing that varies from Fast to Med to Slow (I tested that idea and it did not matter).  
JimDean 
 
Owner/Admin Posts: 3889 Location: USA: GA, Lawrenceville  OK ... the prior posts have set the stage and provided the core formulae and logical concepts. Now I'd like to discuss some of the "research decisions" needed for implementation. FOREMOST: What is Fast? What is Med? What is Slow? That is, what are the periods of the MA's that form the basis for all the Slope and Stack logic? The periods need to be DIFFERENT enough to make a difference in the logic  they can't be too close together (tends to create a lot of lag, or a lot of whipsaws). And, the ratios of the periods need to be geometric, not algebraic. That is, not something like 10,20,30,40 (if we used four MA's) but better, 10, 20, 40, 80 (doubling with each step up). Again, I am proposing this based on the conclusions I've drawn for a LOT of R&D in this area. FURTHERMORE, given a "doubling" relationship or something geometrical like that, WHAT SHOULD BE THE SLOWEST (or FASTEST) periods in the group? This is one of the junctures that we have to consider how many we are using (three or four), and what we're trying to accomplish, and what kind of MA calc method is in play. The answer will differ depending on those factors. But I said the example would use Three EMA's, so let's go with that. I suggest you try plotting three MA's on the SPX chart as a sandbox ... green, yellow and red for fast, med and slow ... and keep their periods in the relationship that Med/Fast = 2 and Slow/Med = 2. Try 10, 20, 40 ... no good ... too reactive to define any extended trend. Now try 40, 80, 160 ... this is much more stable, but has a lot of lag in it. How about 30, 60, 120, or 20, 40, 80? Both of those might be likely candidates. Let's DECIDE, for the example, to use the middle ... 25, 50, 100. Season to taste. If you use WMA's or DCMA's instead of EMA's, make the periods bigger. If you use SMA's or WilderMA's, make the periods smaller. =========== The second major consideration is nowhere near as obvious as the number of periods chosen for the MA's, but it can have a very significant impact on how the rules "pan out" when they are applied ... THE GOAL HERE is to reduce STATEWHIPSAW effects, allowing a given State to remain "in play" even if a brief "glitch" hits to circumvent one of the six clauses that make up the rules. There are TWO different ways to do this ... they can be used separately or in combination. There are pros and cons to each. I call them "Voting" and "FuzzFacs". Voting is fairly simple to understand, and thanks to "Boolean Math", it's fairly simple to implement. Click here for an extensive description of how it works, with some nifty examples. Nutshell: we have SIX RULES that together comprise the definition of a Bull or Bear or Wild or Calm state. Prior to this, we've required that ALL SIX rules be true at the same time, for the State to be identified. Imagine that you're rolling along in a Bull state, and a sudden weird bar causes the GutsPrc to drop, and the FastPrc MA to dip, without much affecting the MedPrc MA ... then say that it recovers quickly a bar or two later. That one disturbance affects the slope of the FastPrc MA, but it may or may not make it go negative. Whether the slope goes negative or not, depending on the current slope of the MedPrc MA, it might flatten the FastPrc MA enough to briefly cross a little under MedPrc. Or, the GutsPrc might drop below the FastPrc MA. So, either the Slope rule or the Stacking rule, or both, or neither, might be affected. That is, requiring ALL SIX rules to be true would force us from Bull to Flat to Bull (StateWhipsaw) ... but if we could allow just FIVE of the rules to be adequate, then maybe that whipsaw would be avoided. Another example (very different) ... let's say that a new Bull trend is starting with a strong thrust up. GutsPrc takes off, and the slope of Fast and Med MA's turn up quickly, and Fast crosses above both Slow and Med. Then, let's say that either Med crosses above Slow, OR Slow slope turns up, but not both. Eventually, both will occur, but often the lag can be significant, such that we miss the majority of the first powerful move of the new trend ... which is often followed by an exasperating pullback that a late entry traps us in. Once again, we have a situation where ALL SIX rules are not true, but FIVE of them might have given us a more timely "StateID". PUTTING IT TOGETHER ... in both of those scenarios, we posit that FIVE rules of the SIX might give us a more "healthy" trend definition. BUT, there are several combinations of WHICH five are needed ... some on the fast side, some on the slow side, depending on circumstances. Boolean Math solves this in a very nifty fashion  our BULL TREND rule becomes: (GutsPrc >= FastPrc) + (FastPrc >= MedPrc) + (MedPrc >= SlowPrc) + (LRS(FastPrc) >= 0) and (LRS(MedPrc) >= 0) + (LRS(SlowPrc) >= 0) <= 5 Note that each relational rule is in parentheses ... this forces OScript to convert it to 1 or 0 for True or False ... SEE the link I provided. ALSO note the "5" at the end ... this says that ANY COMBO OF FIVE of those six rules being true, defines a Bull Trend. Or, you can take the story further if you want, and change the 5 to 4, allowing only FOUR of the rules to be true. I've found that this usually is counterproductive in my testing ... but it might not be, for some choices of MA periods. The point is, it's EASY to change. Similar method for Bear Trend rules, and for Wild and Calm Volatility rules can be used. This is the FIRST of the two "secondary considerations" that I mentioned ... next post discusses the other one.  
JimDean 
 
Owner/Admin Posts: 3889 Location: USA: GA, Lawrenceville  In the prior post, I said:
We then looked at one of two ways of accomplishing that, using Voting and Boolean Math. Now, let's look at another approach that can be used separately or in combination with the Voting method ... this all still works just fine in OScript, btw. The "FuzzFac" method seeks to "soften" any "arbitrary thresholds" that our rules might be using. I've applied this kind of logic to hundreds of situations ... done properly, it ALWAYS helps. In this case, we have a clear "arbitrary threshold" when we see the number "0"(zero) repeated three times for each of the State Sloperules. The FuzzFac method replaces that arbitrary, fixed, sudden threshold with a "fat fuzzy line". To do this, we need to ask what values of the slope (ie price change per bar) are VIRTUALLY "the same" (for purposes of our State definition) as a "pure zero" slope. That is, if the slope is "a little teensy bit down", can / should we treat it as "effectively the same as zero"? If the answer to that is yes, then the question becomes "how big should the Fuzzyequivalence zone be?" That second question can be dealt with in several ways ... but for our purposes, it is important that the FuzzFac "equivalence" zone be NORMALIZED ... you can't pick any arbitrary "points" that will match all symbols. You can use percent of the basis for the slope (price or ATR), but the "appropriate" value for percent tends to vary from symbol to symbol. Futhermore ... take a bit of time to look at the green yellow red lines that you fiddled with earlier. Note how the fast green lines have a much wider range of slopesteepnesses than the slow red lines do. SO, it stands to reason that the FuzzFac effectivelyzero equivalence zone should be a different "width" for the green line, which varies a lot in slope compared to the red line, which is much more sedate. I fiddled with a lot of different ways of doing this, and found that using the Standard Deviation (SD) of the slope provides the best normalized metric. That is, some multiple of SD(LRS(FastPrc MA)) defines the FuzzFac zeroequivalence zone for the FastPrc slope rule, and SD(LRS(SlowATR MA)) sets it for the SlowATR slope rule, and so forth. StdDev in OScripteze is simply "STD(data,pds)" ... typically 30 periods are considered a statistically acceptable minimum (sorta), so let's use that as a starting point. The MULTIPLIER for the StdDev amount will be a FRACTION, probably something like 0.2 to 0.5 ... smaller mult for the more dynamic FastPrc and FastATR, and a bit larger mult for the lazy SlowPrc and SlowATR (may be different mults for Prc and ATR btw). PUTTING IT ALL TOGETHER, our Slope rule triad for a Bull Trend: LRS(FastPrc) >= 0 and LRS(MedPrc) >= 0 and LRS(SlowPrc) > 0 ... becomes ... LRS(FastPrc) >= FprcMult*SD(LRS(FastPrc)) and LRS(MedPrc) >= MprcMult*SD(LRS(MedPrc)) and LRS(SlowPrc) > SprcMult*SD(LRS(SlowPrc)) ... note that the multipliers are NEGATIVE ... we make the line "fuzzy" on the opposite side of zero, to allow a "very slightly negative" slope to "pass" the rule. Similar modifications are made to the ATR Wild Slope rules, and the mod's to Bear Price and Calm ATR rules are the same but the negative sign is removed. COMMENTS: 1. Yes, I realize that the formula is getting pretty long and complex ... but OScript can handle it 2. But, since OScript is not very efficient compared to OLang (no variables , no progressive iterative calcs, etc), this OScript might take a while to calc 3. If speed is a problem, use the Boolean Math Voting approach instead of the FuzzFac approach. 4. If speed is not a problem, you can if you want combine the Voting and FuzzFac nuances together (very powerful). 5. To determine what the appropriate multipliers are for each of the four different states (Bull, Bear, Wild, Calm), you have to set these up as an OLang indicator and plot them to see how it works ... yes, that takes a lot of time ... but, having spent that time, I can say for sure that it is worthwhile to consider.  
JimDean 
 
Owner/Admin Posts: 3889 Location: USA: GA, Lawrenceville  OK ... three hours of continuous typing ... that covers pretty much all the groundwork so that you know the building blocks that I've developed. This is all a part of what I call the "DeanMarket" State analysis. My full implementation of this is much more robust, taking various indexes and Industry groups into account, as well as additional statelogic ... but that stuff is highly impractical (or in some cases impossible) to try to fold in to a transcontinental OmniScript formulae ... the CPU would really balk at the inefficient execution of it too. I have coded all this up in OLang, with a lot more bells and whistles, and eventually plan to release it in one form or the other. All the necessary information is here for you to apply the core concepts to OmniScans or Filter Blocks ... you'll need to make choices along the way, for periods to use, and FuzzFac mults or Voting thresholds, etc. Please consider these first six posts of the thread to be '"particularly" copyrighted (C) 2014 by James D Dean (actually, all the stuff I post on this forum is, but since I may well publish this in book or article form in the future, I'm specifically noting it here). If you want to reproduce them somewhere else, please write me for permission. I've spent countless hundreds of hours on this over the years. Thanks ... I hope you find it beneficial!  
WesSmith 
 
Veteran Posts: 102 Location: Canada: , London  Wow Jim!! A lot to digest and a lot of good work ! Thanks and please get some rest !! Cheers Wes  
JeremyLane 
 
Spectator Posts: 8 Location: : ,  Great work Jim, thanks.  
BillLeake 
 
Regular Posts: 59 Location: USA: VA, Stafford  Jim I love the way you defined and discussed your 9 market states. Great stuff Some things to discuss 1) do you think one should define the market state on something like $SPX or $RUT vs the individual stocks? 2) do you think the market state determination should be against current +1 vs current? Daily for RT and weekly for EOD? or 60min for those trading at 15min and below? You could use the market state for both current and current+1 for extra confirmation 3) determining leading pointers that the market state is changing could be very useful  if you could tell it was about to leave the flatnormal state you may ant to allow long trades for example  get in early. If you were in a trade and the market state was giving indications of changing maybe your stop could tighten up Just thoughts on your excellent discourse  
JimDean 
 
Owner/Admin Posts: 3889 Location: USA: GA, Lawrenceville  Thanks for the encouragement, all! Bill brings up some great points ...
I'll try to incorporate these things in the Sat afternoon session ... will try to prepare appropriate Scan formulae to illustrate simple weighting with Group and index, and also formulae that provide a "leading" flavor ... TIME permitting. Thanks for the good suggestions!  
JimDean 
 
Owner/Admin Posts: 3889 Location: USA: GA, Lawrenceville  Introductory background remarks about this topic are now provided in the OmniTrader forum (with Nirvana's permission). Click here to review them. Please post "contributions" to the thinking process in this forum ... I will try to answer questions that are posted both here, and on the various Nirvana forums, in this thread, so that everything remains centralized for future reference.  
LDNewby 
 
Friend Posts: 37 Location: USA: VA, Vienna  Jim Thanks for sharing the Market State Research. I love Simple and Elegant Solutions! Would it be possible for you to comment on my implementation of your concepts in OmniScript. I don’t want to get too far down the research path without knowing if my understanding is correct. If I wanted to make sure that an Omnivest System only operates in a BullWild market(using the condition manager) or to insure that a system runs againsts stocks in the Bull AND Wild mode (dynamic list) the code that I would use is: (ie.. Research Decisions made were MAs – 25/50/100, Voting and Fuzz Factors) Ie Bull and Wild ((O+O+C+C+H+L) >= EMA(25)) + (EMA(25) >= EMA(50)) + (EMA(50) >= EMA(100)) + (LNREG_SLOPE(EMA(25),3) >= .2*StD(LNREG_SLOPE(EMA(25),3),30)) + (LNREG_SLOPE(EMA(50),3) >= .35*StD(LNREG_SLOPE(EMA(50),3),30)) + (LNREG_SLOPE(EMA(100),3) >= .5*StD(LNREG_SLOPE(EMA(100),3),30)) <= 5 AND (ATR(14) >= EMA(ATR(14),25)) + ( EMA(ATR(14),25) >= EMA(ATR(14),50)) + (EMA(ATR(14),50) >= EMA(ATR(14),100)) + (LNREG_SLOPE(EMA(ATR(14),25),3) >= .2*StD(LNREG_SLOPE(EMA(ATR(14),25),3),30)) + (LNREG_SLOPE(EMA(ATR(14),50),3) >= .35*StD(LNREG_SLOPE(EMA(ATR(14),50),3),30)) + (LNREG_SLOPE(EMA(ATR(14),100),3) >= .5*StD(LNREG_SLOPE(EMA(ATR(14),100),3),30)) <= 5 If there is a conceptual error please advise. Some preliminary testing on my part has indicated that RTM system profitability can be greatly enhanced by matching specific triplets… for example In a Flat Wild market (as determined by the ETF IWM), VBX3 works well with stocks in a BearCalm (profit factor of 14) or BearNormal (profit factor of 5) State. There are just not a lot of trades (I ran the test using OT on the R2K). The possibility of running it on OV against the entire stock market is enticing. I also believe that this paradigm is well suited for the use of a GA with the market states as genes. However – there is a lot more testing that needs to be performed. Thanks again! LD  
JimDean 
 
Owner/Admin Posts: 3889 Location: USA: GA, Lawrenceville  Hi, LD: It's great that you are charging ahead with this ... I'll be providing formulae later on, but I love to see other people take off and run with ideas ... that's what this forum is all about! I can't take the time to research the specific combo of thresholds, periods, etc that you've chosen, but here are a few comments that might help ... I rearrange it slightly for readability of the two sets of six rules:
1. (O+O+C+C+H+L) is missing the "/6" that defines what the "Guts price" is  this test would likely fail every time without the /6 ... added it in for you. 2. I removed one unneeded pair of paren's around the priceset of rules 3. Slopedeviation multiples fast = .2, med = .35, slow = .5 are sensible choice ... usually, the faster MA's have much steeper effective max slopes than slower MA's, AND the wiggles in fast are bigger than for slow, so a "steeper minimum" for fast than slow makes sense. Note that the negative value (which I suggested in writeup) "loosens up" the rule ... to tighten the rule, use positive numbers. 4. It's likely that the slopes of the ATR's have different wiggliness than that of the slopes of the prices, and different relative variations of steepness for different lookback windows ... so, I'd suggest you consider using a different set of SDmultipliers for ATR slope fuzzyfactors than for the Priceslope fuzzy factors. 5. To figure out a good starting point for those six SDmults, use OLang to create a bunch of simple indicators that match the "scalar" LnRegSlope(EMA) values used in each of the rules. Plot the three price indic's on one pane together (with a common scale, using one OLang routine), and do the same (separately) for the three ATR indic's together. That will give you a visual starting point. Then you can modify those routines to change line colors whenever the slopes exceed the candidate input thresholds (input param's as mult's of the SD's of the slopes). 6. You can if you wish apply fuzz factors to the stacking rules, but since they are already dynamic comparisons on both sides of the relational operator (ie no scalars used), it didn't make much sense to me to do so. GREAT START! Jim  
JimDean 
 
Owner/Admin Posts: 3889 Location: USA: GA, Lawrenceville  Note re speed testing in OT: I've run a simple test using the OT filter block, to see if the OScript provided above "kills" the execution time or not. Configuration I used is designed for max possible calc speed. I ran it twice, once with the BullWild filter, and once without. Details: Dynamic Lists start with a universe of Optionable Stocks + All ETF's ... 4645 symbols in all (today, per OScan). But ... I didn't want to sit there waiting a week for the tests to complete, so I used the DJ30 instead. OT is a lot, LOT slower doing this than OVest, which is trimmed down to focus on doing ONLY the bar by bar filtering, not full strategy processing. I modified the BullWild filter VERY slightly, to make sure that the timetests were legit. That is, I wanted the filter to run for EVERY signal, fully evaluating the formula, but have it PASS all the signals so the number of trades in the with/without runs would be identical. To do this, I simply changed the two "5" threshold values to "0". Worked like a champ. Dynamic Lists processing goes back to the year 2000. That's about 3800 bars, plus 200 or so for warmup ... 4000 bars. Turned off BT and FT periods, and unchecked the "save data" Test Settings box, for max speed. No other custom focus lists defined or strategies active. Strategy timeframe set (bottom right of StratBldr pane) to Custom > Daily only. Used All Longs System to force max possible bars to be checked, with Nbar=0 exit, Enter on Close and exit Market (Orders Block) ... note that this strat fires a signal on *every second bar* (DL's run on EVERY bar). Since each trade only lasts one bar (plus a signal bar "gap"), the state of the filter does not substantially change the number of bars being tested from one run to the next ... so, the BullWild filter is being run for all the same trades in both cases. The INCREMENTAL TIME DIFFERENCE between the two test runs, for 4000/2 = 2000 bars, and 30 symbols: Ran the first test as described above  took 0:10:10 to complete Ran again, with BullWild filter turned on  took 0:12:30 to complete ... SO, the BullWild filter created an additional 2:30 = 10 seconds per symbol, doubling it to get to 4000 bars. (on my machine, that means the 4645symbol list would have run 12 hours longer than the nonfiltered list)  
JimDean 
 
Owner/Admin Posts: 3889 Location: USA: GA, Lawrenceville  Followup to the prior post re timetesting BullWild in an OT filter block ... I decided to run a relativecomparison test on OT. To do this, I grabbed the filterformulae from ALL the current "dyamic" lists offered in OT, and combined them all with "and" statements. Then I wrapped paren's around that, and prefixed the whole thing with "True or". The net effect should be that it has to process all the logic of the scan filter rules from ALL the existing logic, to complete the formula, but because of the "True or (longcombo)" component, the answer will always PASS the signal (like setting 5 to 0 in the BullWild formula). Here is the combination of all OVest's existing filter rules (left out ones that just had "V" as the rule): True or ( HHV(250)  C > (HHV(250)[1]LLV(250)[1])/2 and C > EMA(55) AND C > EMA(100) AND C > EMA(200) and LNREG_SLOPE(20)>0 AND LNREG_SLOPE(20)[21]>0 AND LNREG_SLOPE(20)[41]>0 AND H > HHV(40)[1] and LNREG_SLOPE(20)>0 AND LNREG_SLOPE(20)[21]>0 AND LNREG_SLOPE(20)[41]>0 AND MACD_Hist(12,26,9) < 0 AND MACD_HIST(12,26,9)[1] < MACD_HIST(12,26,9) and V > 1000000 and RS(31,"$SPX") > ZMA(RS(31,"$SPX"),5) ) I plugged this into the same test strategy that I used for the previous post, with all the other settings the same for the environment, bars loaded, focus list, etc. The BullWild filter was turned OFF for this run. Then, I did it one final time, with BOTH filters turned on, and filter block requiring both to be true to pass the signal, so it forced each to run every time. Results: Test with OVest DLcombo filter turned on, BullWild off, took 0:11:40 to complete Both OVest DLcombo filter and the BullWild filter on, took 0:12:00 to complete Argh. This means that other stuff is going on in the background, and the timing tests cannot be compared with any degree of refinement. I can only conclude that the filter takes a noticeable amount of time to calculate ... but cannot estimate how much time.  
JimDean 
 
Owner/Admin Posts: 3889 Location: USA: GA, Lawrenceville  "Possibly good news" for OmniVesters: I have done a few timetrials on OVest, comparing the BullWild filter to the OVestCombo filter described in the prior post. OVest if much Much MUCH faster than OT when doing this! It took about 5.5 minutes to complete the BullWild test in OVest, running on the full Opt+ETF list of ~4600 symbols (no ranking or sym limits). … recall that the OT test showed ~2min for the filter, for half as many bars and about 0.7% as many symbols! The OVestCombo test took 44 minutes total, to do the work of six separate DL filtering rules at once (about 7.3 minutes apiece on the average) Probably the really long EMA's and long HHV & LLV tests are the killers. SO, it's reasonable to conclude that NINE of the market state formulas * 5.5 min apiece = 50 min total ... and that since those nine take only a 6 minutes longer to execute than the SIX existing rules, the new ones are adequately efficient. I will pass this info on to Ed, hopefully to get a preliminary approval that adding the 44 minutes extra CPU time will not be too much. If he says no problem, then it makes sense to charge ahead and work on tuning the formulae.  
JimDean 
 
Owner/Admin Posts: 3889 Location: USA: GA, Lawrenceville  Here are the Strategy and Profile that I used for the OT testing described a couple of posts ago. You'll want to use them in combination. And, a snapshot of the screen: (BullWildTimeTest.png) Attachments BullWildTimeTest.png (27KB  10 downloads)BullWildTimeTest.zip (1183KB  22 downloads)  
JeffMartin 
 
Spectator Posts: 2 Location: USA: NM, Los Alamos  I'm just becoming familiar with your methodology, so my apologies for taking this discussion back in time. I'm not taking issue here, just emphasizing an important point. As you noted the natural progression between states is not in any particular order. It would seem that the use of this approach should not depend on any expected or "natural" sequence." I view it, as the name implies, as an approach to determine what the state is at any given point in time and market sector of interest, recognizing that this is a best effort to approximate that. Markets change in a flash between any two states. So I would be reluctant to structure my logic upon expected ordered progressions between states.  
JimDean 
 
Owner/Admin Posts: 3889 Location: USA: GA, Lawrenceville  Exactly true! Everybody wants a crystal ball. This isn't one. Rather, it is a "categorizer of the present".  
JeffMartin 
 
Spectator Posts: 2 Location: USA: NM, Los Alamos  I'm confused by the Section on MAs in "What are we trying to accomplish?" As I was reading i thought the MA' function was to be used to determine the State of the Market. In this sub section you noted the we're trying to determine which state a symbol is in. If we're truely trying to narrow down to deciding the state (domain) of a symbol, would we simply apply the MA logic to the history of the symbol rather than to the history of a market sector of interest. Then you can decide what strategy to apply to the symbol based on the identified state. The other approach is to determine the state of the market and apply related strategies to the market to select symbols. The former would seem more reliable. Transferred by on 5/21/2015, Original Author: Jim Dean  4/16/2014, Original Title: #5786  in reply to #5785  
JimDean 
 
Owner/Admin Posts: 3889 Location: USA: GA, Lawrenceville  Sorry about confusion. The applicability is very broad. You can use this for single symbols, or for MG groups, or for overall market indices ... or a weighted combination of them. There are many possibilities. If you feel that a symbol's likely ongoing price action is driven, for instance, 50% by the symbol's past price action, 30% by its MG Group, and 20% by the SPX, you could apply the formula 3x and weight the results together accordingly. Requires OLang. Or, create three Filters for each of those categories, with appropriately tighter or looser constraints for each. Many possibilities. This presentation's main purpose is to present the core concepts and show examples of how they might be applied.  
JimDean 
 
Owner/Admin Posts: 3889 Location: USA: GA, Lawrenceville  ... and the beat goes on ... In the year+ since I first posted this, I've continued very extensive development work on it. At the 2014 Bash, I presented an enhanced version of the ideas and formulae ... and I was intending to distribute that. Those of you who attended will surely remember the very busy and colorful slides ... one OmniScript formula per slide, with fairly small print. I had to use color coding just to explain it. While it would have been possible for y'all to implement that manually, it would have been a real pain, and would likely have fostered a lot of errors and frustration along the way. So, after returning from that Bash, I decided to check out a different approach before distributing anything. The investigation panned out better than I expected ... the concept was to create the MS model using an indicator in OmniLanguage, with friendly userinputsliders that corresponded to the various parameters for the MA's, LinReg's, SD's, fuzzy factors, etc. The absolutely HUGE advantage here was that I could "play with" the settings and SEE the impact on the chart ... ie when the formula identified each of the three Trend and Volatility states. The second half of the experiment was much more difficult ... an Excel spreadsheet that takes the input parameter settings used by the OLang indicator, and automatically creates a valid OmniScript formula that *exactly* (bar for bar) duplicates the results of the OLang calc's ... but using a form that can be ported easily over to OmniVest for Strategy Conditions and Dynamic Lists (or use in OT/VT anywhere that OScript is accepted). This development took a lot of Excel experimentation, but finally was successful. So, being even more encouraged, I decided to push this new approach as far as I possibly could ... ie, "load up" the OLang Indicator with extra options and capabilities, using concepts more advanced than the presentation above ... until I reached the "practical limits" of what was possible with OScript ... what turned out largely to be the max number of characters possible in a scriptstring (dotNET limit is 32,767 chars ... there are actually some rare cases where the Excelgen'd string can exceed that). This allowed me to fold in some of the other valuable lessons I'd learned in the creation of the "beallendall" State filters (QuadMAplus) that I was also working on last year. Finally, it's done. There is now an OLang indicator called "MTV" (Market Trend & Volatility) which I'll be making available in the next week or two. The short presentation I'm making on Monday at the 2015 Bash will use that tool to explain the importance of Market States ... I will cover it in depth during the Saturday afternoon presentation. Of course, there will be some costs associated with this but I've come up with a means of doing so that should keep it very, very affordable for everyone, regardless of how they might want to use it. I'll publish more about that later in this forum. So ... the journey is nearly complete. But the beat goes on ... For more info about the MTV and ProMTV tools, and the Scripts they generate, CLICK HERE  
WesSmith 
 
Veteran Posts: 102 Location: Canada: , London  Sounds great Jim. Thanks in advance for this. Have fun at the Bash.  
JohnWolfraad 
 
Veteran Posts: 208 Location: Australia: , Church Point, NSW  Jim, WELL DONE! THANK YOU for being SO FAR AHEAD and allowing us to be part of your AMAZING ACHIEVEMENTS! Best Regards, John  

Owner of site: Jim Dean  Forum content is confidential, and may not be distributed without written permission.  