Search | User Listing Vidsite | Calendars | Quotes
Home Page ->  OT ToolBox Modules -> Strategy Wizard -> View ThreadLogon (or Register, or Join TradeTight)

You are logged in as a limited-access Guest.To join TradeTight, first read the info in the Organization & Content room, then click the link above. 

1 Tim 6:18-19 (ESV) … (the rich) are to do good, to be rich in good works, to be generous and ready to share, thus storing up treasure for themselves as a good foundation for the future, so that they may take hold of that which is truly life.


Improvements in SW Outputs
Jump to page : 1
Now viewing page 1 [50 msgs/pg]
Jump to room :
JimDean
Posted 10/18/2011 7:14 AM (#3111)
Subject: Improvements in SW Outputs



Owner/Admin

Posts: 3925
2000100050010010010010025
Location:
USA: GA, Lawrenceville
With PreRelease 4C, Nirvana has made some great improvements in the Strategy Wizard outputs, and in the related ToDo > Performance-Analysis Reports that they are drawn from.

==================

The "performance analysis" report from a standard ToDo backtest/forwardtest run, which is viewable in Files > Print Reports, is a "limited normalized" form of backtesting.

The limitation is that it takes every trade that the Strategy generates, and calculates its statistics based on ONE SHARE per trade. Regardless of the price per share or anything else.

Some very meaningful statistics can be drawn from that kind of analysis ... for example, Hit Rate is a measure of whether a given trade is profitable or not, so it doesn't matter what size the position is. Other "safe and useful" numbers in prior versions have been NT and Prof%. There was, however, some question as to the units-basis of PPT and APR and StdDev ... but AvgWin and AvgLoss were pretty much useless ... BUT - see below ...

The statistics related to the Performance report have recently (THANKS, NIRVANA) with this 4C update been GREATLY IMPROVED. Before this version was released, several of the Analysis Report stats were in units of Dollars (for example, AvgWin) - or (probably - it's unclear) they were derived from raw dollars (such as APR and PPT). Since the Analysis study is based on only one share per trade, the "dollars" of profit or loss ends up being HEAVILY WEIGHTED to more expensive stocks ... which made the statistics in those columns less useful than they could have been.

With the 4C release, Nirvana has made a big improvement to the Performance Analysis reports. They've modified all the output results so that if they are had been in units of "dollars' ... they now are "normalized" into units of PERCENT of share-price. This means that ALL the outputs from this report offer solid statistical measures of the strategy's performance, disregarding the relative differences in per-share prices.

EXAMPLE: if you have a $10 stock that had a profit of $1, and a $100 stock that had a profit of $1 ... instead of reporting the average of those two as being $1 ... the improved calculations will be MUCH more meaningfully representative of the profitability of your strategy, since it will first determine the PERCENTAGE profit for each ... $1/$10 = 10% and $1/$100 = 1% ... then it will average the percentages ... (10% + 1%)/2 = 5.5%

This "carries over" into the Strategy Wizard Analysis Reports, since SW uses the normal output from the Performance Analysis BT/FT runs. So, the SW results are MUCH MORE USEFUL now. Before, you had to pick and choose pretty carefully what stat's to use as a valid basis of comparison. Now, Gianluca has fixed all the outputs that previously were $$-based, to be properly normalized as percentages for those runs. This improved the usefulness of the AvgWin, AvgLoss, and AvgWin/AvgLoss columns, and probably also the PPT, APR columns as well.

==========

IN CONTRAST ...

When you run Portfolio Simulator ... either separately or via Strategy Wizard ... it determines exactly how many shares & dollars are TRULY related to each trade ... not just one share per trade. Therefore, the "share-price bias effect" from Analysis runs is gone. So, when you use PortSim, the outputs for those stat's I mentioned earlier are in DOLLARS, not percent.

If you think about it, ROI (return on investment) and WDD (worst draw down) are meaningless unless the entire cumulative portfolio is modeled, taking into account the "balance" of how your cash actually has been spread across the various stocks. That's why ROI and WDD and ROI/WDD columns are BLANK in the SW Analysis output, but not in the Portfolio output.

PortSim provides a wide (and cool) variety methods of allocating your funds to the various trades ... anything from Fixed $ to Sharpe Ratio to Neural Net driven (for NClub members ;~) ... which is intended to emulate your actual practice when trading.

ANOTHER important thing that PortSim brings into the picture is logic which LIMITS how many trades can be taken at a time, based on how much money you have available at the time. The Performace-Analysis reports don't have any such limitation ... but your real-life account does.

PortSim also has user-configurable logic to determine WHICH symbols to take positions in on any given bar, if there are inadequate funds to take all such trades. Among other reasons, this is why the Performance run and PortSim run might have different total NT (numbers of trades) or HR (profitable-hit rates).

Each of these two reports brings something new to be considered ... and Nirvana's recent modifications have only made it BETTER!

==============

So, if you want to see statistical measurements concerning how a strategy performs "in isolation", unrelated to limitations of real dollars (this is usually the "early testing phase"), I advise use of the Performance reports. Once you've done the preliminary adjustments to the strategy and want to "fine-tune" it, you might want to focus more on the PortSim reports.
Top of the page Bottom of the page
JimDean
Posted 10/18/2011 7:19 AM (#3112 - in reply to #3111)
Subject: Improvements in SW Outputs



Owner/Admin

Posts: 3925
2000100050010010010010025
Location:
USA: GA, Lawrenceville
Here are a series of output snapshots from the NEW Strategy Wizard Performance Analysis reports ... note that:

I ran the test with all the bars assigned to the BackTest period, to keep the output simple.

The Output table columns for ROI, WDD and ROI/WDD are blank, since they are meaningless without real $$$.

The other columns (after BT HR) are mostly all single-digits. This was run on the DJ-15, over a two-year period. This is because they are NORMALIZED percentage values now, rather than raw dollars (the final column is normalized differently - it's the number of standard deviations in the Gain/Loss results.

The bottom snapshot is the text-file output, suitable for loading into Excel. NOTE that Nirvana added percent signs to make it clearer what the basis for the analysis is, now ... also note that here, too, the ROI and WDD-related outputs are zero.

(New SW Performance Output.png)



Attachments
Attachments New SW Performance Output.png (147KB - 4 downloads)
Top of the page Bottom of the page
JimDean
Posted 10/18/2011 7:40 AM (#3113 - in reply to #3112)
Subject: Improvements in SW Outputs



Owner/Admin

Posts: 3925
2000100050010010010010025
Location:
USA: GA, Lawrenceville
Here are a series of output snapshots from the NEW Strategy Wizard Portfolio Simulation reports ... note that:

I ran the test with all the bars assigned to the BackTest period, to keep the output simple.

The PortSim allocation method chosen is Fixed $ ... that is, regardless of the price per share, PortSim takes an equal stake in dollars, in every trade that it manages. This is a good way to do SW runs imho ... it's a form of "normalization" that prevents the share-price from biasing the results.

The Output table columns for Prof% and APR are blank ... I'm not actually sure why N did this ... it would have been interesting to see these values based on the PortSim run.

The ROI, WDD and ROI/WDD columns are NOT blank. Since real dollars are involved, the portfolio return on investment and worst draw down info is meaningful (and interesting).

The AvgWin & AvgLoss columns are NOT single-digits any more ... they are DOLLARS in this report (compare to the snaps in the prior posting). These are menaingful since a true portfolio-investment model has been run.

I'm not certain why the final-column Gain/Loss Std Deviation values are much smaller in this report than in the Performance output ... I suspect that it's calculated versus the equity curve rather than as an average of each trade. If so, that's interesting ... and a different viewpoint than the Performance run provides.

The bottom snapshot of the text-file output shows that Nirvana added percent signs AND dollar signs, to make it clearer what the results are, now ... also note that here, too, the PPT column is blank ... and the APR column is actually missing (not sure why, but no matter, since it's blank anyway).

(New SW Portfolio Sim Output.png)



Attachments
Attachments New SW Portfolio Sim Output.png (177KB - 3 downloads)
Top of the page Bottom of the page
MarcBekaert
Posted 10/21/2011 4:15 AM (#3130 - in reply to #3111)
Subject: Improvements in SW Outputs



Friend

Posts: 33
25
Location:
Belgium: , Waardamme (Bruges)
Hi Jim
I noticed that the column BT APR and FT APR is missing in the output TXT file when running the the perfomrance report
It is available in the output tab, but not in the output txt-file

Marc
Top of the page Bottom of the page
JimDean
Posted 10/22/2011 6:30 AM (#3135 - in reply to #3130)
Subject: Improvements in SW Outputs



Owner/Admin

Posts: 3925
2000100050010010010010025
Location:
USA: GA, Lawrenceville
Good catch, Marc! I will report that to Barry - thank you!

==== followup ====

Barry told me that Gianluca will add APR to the performance output text file, presumably in the next PR, but that's a guess. APR is not available in the native PortSim report, so it won't be in the SW output.
Top of the page Bottom of the page
Jump to page : 1
Now viewing page 1 [50 msgs/pg]
( E-mail a Link | Printer Version | Search Room )

Owner of site: Jim Dean -- Forum content is confidential, and may not be distributed without written permission.