Search | User Listing Vidsite | Calendars | Quotes
Home Page ->  General Discussion -> Operating System & Software -> View ThreadLogon (or Register, or Join TradeTight)

You are logged in as a limited-access Guest.To join TradeTight, first read the info in the Organization & Content room, then click the link above. 

Ecclesiastes 7:8 (NKJV) ... The end of a thing is better than its beginning; the patient in spirit is better than the proud in spirit.


Sticky Using VM Clones to Speed OT Testing
Jump to page : 1
Now viewing page 1 [50 msgs/pg]
Jump to room :
JeffBrown
Posted 5/21/2018 12:43 PM (#9093)
Subject: Using VM Clones to Speed OT Testing



Friend

Posts: 45
25
Location:
WI: ,
How to create a copy (clone) of an OT VM – Testing with ease
Attached PDF has the contents of this thread's presentation

We all have different ideas on what market indicators, strategies, and symbols will make up the most profitable trades. We also have different styles and time frames that we like to trade. OT offers so many different customization options and parameters that it makes our heads spin. Add to it tools like Strategy Wizard that help us iterate through various settings and it can be Herculean! How can we possibly conduct a lifetime of analysis in a shorter period of time?

Cloning can help. I’m sure we’ve all thought about how we could clone ourselves and get more done. While we can’t do that (yet), we can clone a VM – make an identical copy so that we can do twice or three times as much. VirtualBox and VMware support cloning capability. For my example, I’ll use VirtualBox. But before we get into how to clone, let’s first talk about the type of clones.

Full Clone vs Linked Clone
Virtual Machines consume hard disk space. The base Windows VM that we created in another thread as an example (OT-Demo), consumes 16 GB before we added OT to it. Once we add OT, our strategies and plugins, data, and a few other useful pieces of software, our VM could easily grow to 60 GB. Multiply that by the number of VM’s you want to use for testing and suddenly you are out of disk space.

Once you get comfortable using virtualization, you may want to create specialized VM’s for specific tasks such as live trading vs strategy testing. We call this VM sprawl. This also consumes disk. So, the first challenge you are likely to face is running out of disk space.

Creating a Full Clone will not help you conserve disk space, but it does help you save time. For instance, when we built the OT-Demo VM, the installation of Windows probably took 20 minutes. Adding OT and all the plugins could take another hour. Add in updates and reboots and we spent say 2 hours creating that one VM. Now we want to create another one to use for testing or live trading. A full clone will give us an exact copy of an existing VM. So, if we cloned OT-Demo that occupies 16 GB, the clone will take 16 GB. But since OT-Demo already exists, it will take a matter of minutes instead of hours to create. That’s the time saver of a Full Clone – fast create of a new VM based on an existing VM.

A full clone is a copy of a VM. They are not connected in any manner meaning you can move it, delete it, change it, and there is nothing used or shared with the original VM.

But I said our problem is likely to be shortage of disk space, so why would I choose a Full Clone? I’ll cover that in a moment. First, let’s describe a Linked Clone.

A Linked Clone is a method of cloning/copying that links the new VM’s hard disk file to the original VM’s hard disk file (the VM that you used as the base). This method saves valuable disk space. In the example of OT-Demo where its hard disk file is 16 GB, a linked clone to OT-Demo (let’s call it OT-Clone) may only take up 2 MB – yes, 2 MB vs 16 GB. Why the difference and how can I fit a full OS in only 2 MB of space? The answer lies in the use of snapshots.

A linked clone takes advantage of the snapshot capability of the virtualization software. It begins by taking a snapshot of the current state of the base VM (OT-Demo) to preserve that point in time. It then creates a new disk and attaches that to the new VM (OT-Clone). This new disk will hold only the changes made to OT-Clone that are different from what is found in the OT-Demo point in time snapshot. So, since OT-Clone was just created, there is essentially no difference between OT-Demo and OT-Clone. As you use OT-Clone, this will not be true and the 2 MB disk of OT-Clone will grow in size. But you can already see the advantage from both a time to create perspective and an amount of storage consumed perspective.

So, a Full Clone makes an exact copy of an existing VM that becomes essentially a new standalone VM. A Linked Clone creates a new VM but shares a virtual hard disk with the original base VM – they are linked together at the storage level through the use of snapshots.

Now, back to the question of why would I choose a Full Clone since it consumes more storage? The simple answer is for performance and resiliency. Here’s what I mean, and sorry, this is going to get a little technical.

When you make a Linked Clone, a snapshot is made against the main disk of the base VM (OT-Demo). There are actually a minimum of 3 files introduced. The first is a point in time snapshot of the base VM’s disk. A second snapshot is created to capture changes caused by running OT-Demo from this point forward. Finally a third snapshot is created to capture changes caused by running the clone (OT-Clone) from this point forward.

When you run OT-Clone and it reads the virtual disk, the virtualization software needs to first check to see if the disk blocks of the OT-Clone snapshot are different from the point in time snapshot of the original OT-Demo disk. If they are different, the virtualization software returns the data found in the OT-Clone snapshot, otherwise it returns the data from the original OT-Demo disk. It is essentially a cascade of checking first the content of OT-Clone’s disk and if not found there, then check OT-Demo’s original disk. This takes time. OT isn’t terribly disk I/O intensive so you won’t see much of a performance impact (likely imperceptible – we are talking about milliseconds of time).

As for resiliency, we have to consider what might happen if that one base disk (the original OT-Demo disk) becomes unavailable. Maybe we saw it in the directory and it had an old timestamp on it and decided to delete it. Or perhaps the physical disk drive failed and the original file now has a bad set of blocks. Now, let’s run through the cascading logic – check the clone first and the block isn’t found there so let’s move on to the original disk and… BAM! It doesn’t exist or is faulty. What happens to the OT-Clone VM? What happens to the OT-Demo VM? Essentially, they both may not function. You may then need to rebuild both of the VM’s.

On the other hand, if you had created OT-Clone as a Full Clone, the only VM affected would be the OT-Demo VM since it had the bad virtual disk. So, performance and resiliency are the main reasons you’d consider using a Full Clone.

So, I should use a Full Clone for OT testing? No – use a Linked Clone unless you intend to move the new VM to a different computer. File damage is rare. It can happen, but it is rare. The newer your computer, the less likely you are to encounter file problems. You can also take a backup of that original file and restore it if it becomes corrupted. Save space and use a Linked Clone.

How about the amount of maintenance? Is there less maintenance to the OS or OT if I use a Linked Clone? Short answer – No. Once you make the clone, from the view of each VM, you have a running OS. It needs to be patched/updated. OT enhancements become available and you need to install the new version. In both cases, you’ll perform both of those activities in each VM – twice as many VM’s means twice as much maintenance. So, if you notice you aren’t using a VM, delete it. Free up that disk space. You can always create a new clone within minutes.

Alright, enough on Full Clones vs Linked Clones. Let’s get into preparing a VM to create clones for testing.


Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing


Attachments
Attachments UsingVMClonesToSpeedOTTesting.pdf (986KB - 9 downloads)
Top of the page Bottom of the page
JeffBrown
Posted 5/21/2018 12:44 PM (#9094 - in reply to #9093)
Subject: Using VM Clones to Speed OT Testing



Friend

Posts: 45
25
Location:
WI: ,
OT Testing with Multiple Clones

You’ve got that great idea, but it will take time to run the tests. You want to continue trading but you also want to run a few tests. Clones to the rescue.

Step 1: Prepare OT for the testing in the VM you want to clone – the “base VM”
It’s a good idea to install all the strategies, plugins, and load the data you want for your tests before you clone the VM. As discussed in the prior section, this will help minimize the amount of hard disk space.

Step 2: Create a Linked Clone (unless you really need a Full Clone).
To create a Linked Clone, go to the VirtualBox Manger, right click on the VM you want to clone, and select Clone from the popup menu.

(Clone-1.png)




Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing


Attachments
Attachments Clone-1.png (153KB - 3 downloads)
Top of the page Bottom of the page
JeffBrown
Posted 5/21/2018 12:47 PM (#9095 - in reply to #9093)
Subject: Using VM Clones to Speed OT Testing



Friend

Posts: 45
25
Location:
WI: ,
You will then be greeted with the window below:
• Change the Machine Name to something different if you’d like.
• Select Linked Clone
• Check the box to Reinitialize the MAC address. This isn’t required but it helps avoid network trouble if you leave the network setting active. We will be deactivating the network shortly.
• Click Clone.

10 seconds later I have a new VM. That was easy!

Do you want another clone? Repeat the steps above beginning with right clicking on the original VM again. Be sure to give it a unique name. This time, 1 second later I have a second clone.

(Clone-2.png)




Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing


Attachments
Attachments Clone-2.png (141KB - 3 downloads)
Top of the page Bottom of the page
JeffBrown
Posted 5/21/2018 12:48 PM (#9096 - in reply to #9093)
Subject: Using VM Clones to Speed OT Testing



Friend

Posts: 45
25
Location:
WI: ,
When you are done cloning, you will see the original base VM (OT-Demo) and the 2 clones I created. Looking at the Snapshots detail, you can see the cloning process made a snapshot for each of the clones. You don’t have to do anything with these, but it confirms what you read above about Full Clones vs Linked Clones.

(Clone-3.png)




Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing


Attachments
Attachments Clone-3.png (113KB - 3 downloads)
Top of the page Bottom of the page
JeffBrown
Posted 5/21/2018 12:50 PM (#9097 - in reply to #9093)
Subject: Using VM Clones to Speed OT Testing



Friend

Posts: 45
25
Location:
WI: ,
Before we use the clone, we will deactivate the network for each of the clones. You don’t necessarily need to do this and in some cases you may want the network active. But, here’s the consequences.
• When Windows starts, it looks for a network connection. If it finds one, it may decide to conduct maintenance (download the latest Windows Update, update the virus definition…). This consumes network bandwidth and CPU. If it downloads data, it will save it to the virtual disk – consuming your valuable hard disk space. And since you will remove these Clones when you are done with your testing, do you really want Windows to waste time doing maintenance?
• When Windows does maintenance, it also check for Product Key registration. Windows will likely treat this temporary VM as a new installation of Windows and using an existing registration key. It usually doesn’t impact your base VM since you registered it with Microsoft, but why take the chance that you’ll have to talk with Microsoft to clean up its license registry when you delete the clone?
• OT itself is licensed software. Nirvana gives us and ability to run 5 instances. If you add a plugin or get your OT license keys, you are likely to exceed the 5 at some point (maybe not immediately, but if you create and destroy clones with network enabled, you will eventually do this). You then need to make a call or email the Nirvana support team to have your license keys reset. That wastes time – both yours and Nirvana’s.

So, let’s deactivate the network.
• Click on the Clone to select it.
• Click on the Settings (Red box) or you can click on the bold Network in the VM detail section of the VirtualBox Manager.
• Select Network if it isn’t already selected.
• Change the “Attached To” from NAT to “Not Attached”
• Click OK.

(Clone-4.png)




Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing


Attachments
Attachments Clone-4.png (126KB - 3 downloads)
Top of the page Bottom of the page
JeffBrown
Posted 5/21/2018 12:53 PM (#9098 - in reply to #9093)
Subject: Using VM Clones to Speed OT Testing



Friend

Posts: 45
25
Location:
WI: ,
“WAIT,” you say. “I forgot to download all the data I needed… How do I activate the network again?”

If your VM is not running, just reverse the process – change the network back to NAT or Bridged.

Both of these options (NAT or Bridged) will connect you to the internet. NAT stands for Network Address Translation. It will essentially use your host computer’s IP address for communicating with the network. If you select Bridged, then your local network (home network) will assign your VM a unique IP address for your local network. That's just a little more technical jargon for you, both will get you to the internet.

Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing

Top of the page Bottom of the page
JeffBrown
Posted 5/21/2018 12:55 PM (#9099 - in reply to #9093)
Subject: Using VM Clones to Speed OT Testing



Friend

Posts: 45
25
Location:
WI: ,
What if my VM is running… Can I change the Network Setting?

You certainly can! Some settings require you to shut down the VM before making the change. This means shutting down the OS inside the VM. However, the network is not one of those changes that need the OS shut down. Let's activate the network on a running VM:

• Right click the Network icon in the bottom right of the VM window (Red box).
• Select Network Settings.
• You will be greeted with the same dialog box as above. Change the Attached To: to either NAT or Bridged.
• Press OK.
• Your network will now be attached. Remember to detach it once you are done collecting your data.

(Clone-5.png)




Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing


Attachments
Attachments Clone-5.png (174KB - 2 downloads)
Top of the page Bottom of the page
JeffBrown
Posted 5/21/2018 1:02 PM (#9100 - in reply to #9093)
Subject: Using VM Clones to Speed OT Testing



Friend

Posts: 45
25
Location:
WI: ,
Other Helpful Tips

Snapshots are your Friends - learn to love them

OT is a powerful tool, but it can be fragile. Occasionally, it will crash or throw an error. This is okay and you can easily restart OT, but what if it occurs in the middle of a long Strategy Wizard run? Maybe you are 1 week into the run and OT crashes. That’s devastating – a week lost.

Here’s a helpful tip, create a snapshot periodically. If OT crashes, simply restore to the most recent snapshot, start the VM and you are back running. Yes, you did lose all the work that was done from the point of the snapshot to the point of failure, but at least it isn’t all gone.

When I’m running something that I know will take a week or month to run, I introduce automated snapshots – I make my host periodically run a command that will create a snapshot every 6 or 12 hours. Since snapshots take up space, I go through and delete the earliest snapshots every few days to reduce the number back to 1 snapshot for each testing VM. How do I do this? I’ll give you the basic VirtualBox command, but you’ll have to create your own script and google how to set up the schedule on your host computer.

I place the following command in a file I call vmsnap.bat

“C:\Program Files\Oracle\VirtualBox\VBoxManage.exe” snapshot “OT-Clone-1” take Auto --description “Snapshot During Analysis” --live --uniquename Timestamp


• This is one line.
• The dashes are double dash.
• Change “OT-Clone-1” to the name of your VM.
• Quotes should be typed as well – they solve the problem of embedded spaces.
• If you have multiple VM’s you are using for testing, just duplicate the line and change the name of the VM.
• When the bat file is executed, it will then snapshot both of the VM’s.

The same command format works for Linux and OS X. The command basically instructs VirtualBox to take a snapshot to be named Auto and adds a description. It is to take the snapshot against a live (running) VM and include the Timestamp in the name if it is needed to make the name unique.

Now use the Windows Scheduler to run the vmsnap.bat file every so often. You’ll have to google how to do this for your version of Windows.

Happy testing!


Use Cloning to Create a Template

This is perhaps a more advanced topic. A template can be considered a base VM that you want to use to use as the base for other VM's.

For instance, during the create of the VM with just the OS but before you installed OT, you have a pristine copy of a VM with Windows. If you plan to create non-OT instances, this would be a good spot to conduct a full clone of the VM -- maybe call it "WindowsTemplate". You might do the same after you installed OT with your plugins and call it "OT-Template".

This gives you a known VM upon which to spawn other, independent VM's.



Edited by JeffBrown 5/21/2018 1:25 PM


Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing

Top of the page Bottom of the page
SalimHira
Posted 9/3/2020 8:44 PM (#9827 - in reply to #9093)
Subject: Using VM Clones to Speed OT Testing



Veteran

Posts: 183
100252525
Location:
USA: MD, Columbia
Hi All,

Thank you for taking the time and explaining the above as it has helped clear many questions, and responded to others I'd not have thought about.

From attempting to understand thoroughly, I have a following question of "Full Clones vs Linked Clones" pertaining specifically to OT install.

In a Full Clone, its easy to follow-thru the OT install as its independent from all other VM installs, upgrades, data-feeds, etc. as explained in the forum.
I presume that is a correct assumption ?

But in a "Linked Clone" of OT - I am attempting to visualize how OT would work "linked" to another VM ? I am struggling to understand while "linked" how "one-bird is attempting to make a two-kill at same time" ...

Am I missing, or maybe I have forgotten something ?

Secondly, I'd like help for suggestions how best to setup (allocate memory and HD) for a new VMware machine between the Host and VM's that has AMD Ryzen 3700x, 32gig memory, 1-TB hd, with 4-display graphics card specifically to trade intraday Live and EOD, and rest (i.e., possibly 2-VM for Strategy Wizard development), etc... I'd still have 1-extra OT license that I'd use on my laptop as backup, I presume without needing for resetting ?

Thank You.


Edited by SalimHira 9/3/2020 8:59 PM


Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing

Top of the page Bottom of the page
SteveLuerman
Posted 9/4/2020 5:18 PM (#9828 - in reply to #9827)
Subject: Using VM Clones to Speed OT Testing



Elite

Posts: 414
100100100100
Location:
USA: CO, Boulder
Hi, Salim.

A Linked-Clone is a VM which shares the static (i.e., unchanging) part of the original VM (most of a Windows OS installation are executables that are "static"). As soon as you start using the Linked-Clone VM, then anything unique to that VM will not be shared with the original VM, and anything the original VM changes after the Linked-Clone was created will not be shared with the Linked-Clone VM.

Personally, I would only use a Linked-Clone for a VM that is "disposable." You use a Linked-Clone to run some short term testing. And, I would do as Jeff advises, and disconnect it from the network. In this clone VM without network access, you will not be able to make any changes (i.e., can't update data or install new plugins), but you can run tests. NOTE: This implies that you create the clone only after completing your OT install, setup and current data download in the Original VM (the VM you are going to clone).

A Full-Clone VM is basically a copy of the original. If you intend it to be used long-term, it will need it's own Windows license and OT license (one of 5). Creating a Full-Clone is simply a short cut that avoids installing everything (e.g., the OS, OT and all OT's plugins) from scratch.

OT is single-threaded. So, OT really can't effectively use much more than 1 "processor" at a time. I find that due to OS overhead, you should allocate at least 2 processors for each VM, and you might see a little bit of a performance improvement by adding a third. You have a Ryzen 3700x which has 8 cores and 16 threads. In VMware terminology, that is 16 virtual processors. I would allocate 2 or 3 to each VM (it's easy to experiment with and change). And, 4GB of memory per VM. So, if you allocate 2 processors and 4GB per VM, you could theoretically have 7 VMs active (leaving a couple threads/processors and memory for your host system). Personally, I wouldn't plan on running 7 concurrent VMs. I'd limit it to 4 VMs with 2 or 3 processors per VM. (But, as you'll find out, it's easy to make changes to your VMs to suit your preferences.)


Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing

Top of the page Bottom of the page
PaulWaijers
Posted 9/5/2020 3:51 AM (#9829 - in reply to #9093)
Subject: Using VM Clones to Speed OT Testing



Spectator

Posts: 6
0
Location:
Netherlands: , Voorschoten
Can anyone share some statistics how much faster OT will perform for example on strategywizard 5. What is the speed performamnce on your machine using vM Clones or regular? Just exploring whether I want to do the extra effort for setting up a VM clone.

Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing

Top of the page Bottom of the page
SteveLuerman
Posted 9/5/2020 2:33 PM (#9830 - in reply to #9829)
Subject: Using VM Clones to Speed OT Testing



Elite

Posts: 414
100100100100
Location:
USA: CO, Boulder
Paul,
I don't have any statistics to offer. But anything on a Virtual Machine (VM) will run slightly slower than on the Physical machine hosting the VMs. There is always some overhead associated with a VM. Jeff's purpose, in creating this thread, was to suggest a way to accomplish more testing by running multiple instances of OT in parallel on VMs. And his explanation of how to setup and use a clone is a way to run more VMs than your 5 OT instance limit, albeit with some limitations.

Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing

Top of the page Bottom of the page
RolandVanGiesen
Posted 9/6/2020 5:29 AM (#9831 - in reply to #9093)
Subject: Using VM Clones to Speed OT Testing



Spectator

Posts: 5
0
Location:
: ,
@Salim: I have processor AMD A8 3.9-4.2 GHz, 4C+6G cores, 32GB DDR3, 1TB HD.
Currently troubling with speed especially during calculations with GA and now also with StratWiz5. I replaced HD Seagate 1TB memory drive with same capacity Samsung 860EVO SSD SATA connection. It got ~3 -5 times faster but still "slow".
The advice I got, willing to have equipment that calculates with high speed, is to replace motherboard / processor to Intel i7 6700K or 7700K 4.2-4.6 GHz, DDR4 workmemory min 32GB or better 64GB, SSD SAMSUNG 970EVO M2 connection and RAM supporting (gamer) videocard. That should increase speed another 3-5 times. Important is also the ability of the motherboard to having a M2 connection, SATA SSD is compared to M2 SSD super slow (factor 5-6).
Maybe this would help?


Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing

Top of the page Bottom of the page
MarkGerber
Posted 9/6/2020 5:40 PM (#9832 - in reply to #9093)
Subject: Using VM Clones to Speed OT Testing



Friend

Posts: 31
25
Location:
USA: WA, Issaquah
Roland,

I have just begun to spec out a new computer for ARM6 GA and SW5 data crunching, so thank you for the performance suggestions.

Interestingly, your base (AMD A8 3.9-4.2 GHz, 4C+6G cores, 32GB DDR3, 1TB HD) appears to be somewhat close to my current computer, too.
Question: Confused by the acronyms . . . if the '4C' refers to 4 physical CPU cores, what does the '6G' refer to?
Are you somehow using 6 GPU cores for data crunching too?
If so, what hardware are you using, and how are you able to do *that*? I think I had dipped a toe in these waters some years back but have long forgotten how I did it.
What do you think the relative additional throughput improvement using the GPUs might be?
Further, what hardware improvement (graphics card) would that suggest as an efficient generator for GPU data crunching?

Question: The performance improvement suggestions from your base would seem to be 1TB hard drive --> 860 EVO SATA results in ~3-5 x throughput improvement
And then further improving 860 EVO SATA --> 970 EVO M2 resulting in ANOTHER ~3-5 x throughput improvement . . .
Would these improvements suggest a final (3-5 x) X (3-5 x) = 9-25 throughput improvement over your base??


My older son is a professional programmer. We had begun our discussion with the idea of using one of the new AMD *many* cored CPU constructs. But he is now leaning toward a top-of-the-line Intel speed demon. Any thoughts on this?

Is there a generally accepted software program to effectively benchmark the kind of data crunching throughput we need to do for ARM6 GA and SW5?



I began this reply mostly just to thank you for your suggestions, but considering that I had just begun to speck a new computer, my mind just went bananas thinking about the possibilities. Besides, it is better to ask questions now, then to regret some decisions later :)


--Mark G.






Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing

Top of the page Bottom of the page
SalimHira
Posted 9/6/2020 6:03 PM (#9833 - in reply to #9093)
Subject: Using VM Clones to Speed OT Testing



Veteran

Posts: 183
100252525
Location:
USA: MD, Columbia
Steve, thanks for your feedback, quite helpful in planning.

Another question that comes to mind is Data-feed both RT and EOD, i.e., esignal, iqfeed, etc. Normally, for RT feed, they want subscription for each computer. Technically speaking, with VM - its' one computer with a single internet connection coming in... does it become an issue running OT in multiple RT or EOD instances in VM-world ?

Hi Roland, thanks for the great suggestions. For the life of me, I've been an Intel-guy, 40+ years supporter, with no issues w/ Asus motherboards. My gamer teenage son, last month highly pitched AMD, company has really turnaround, etc., etc. and I put him to the task :-). I will look upgrade to 64GB DDR, but what stings me, for now at least, is I thought 1TB HD was more than enough (to work w/ an OT-only computer), but diving into the VM-world (ideas are popping up beyond OT-only), I am really thinking it may not be enough.

I'd like to pull from all those computers, harddrives lurking on my bookselves, attic, and basement and dump each into its own VM ... and all my work, antics, etc. would be in one or two large HD and its own repository for easy access. Bear in mind, older HD were not larger than couple hundred megs :-) back than.

I really appreciate Steve and Jeff, and others chiming in for having taken the initiative to help us all here in N-community. As the costs for hardware keep coming down, with the increase of its speed and efficiency, its worth diving into the VM-world, imho.


Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing

Top of the page Bottom of the page
JimDean
Posted 9/6/2020 6:35 PM (#9834 - in reply to #9833)
Subject: Using VM Clones to Speed OT Testing



Owner/Admin

Posts: 3925
2000100050010010010010025
Location:
USA: GA, Lawrenceville
Fwiw, here are my hardware recommendations:

1. First, the graphics card has no meaningful impact on OT operation, in native or VM mode. OT does not utilize GPU processors for its calc’s.

2. An SSD should improve performance vs a mag HD, and an M2 “chip” SSD helps even more, if the PC supports it. However most OT calc’s are RAM based, so this is less important than the processor.

3. Imo, the best chipset to use for multiple VM’s is one intended for a server - a fast Xeon chip with error-correction RAM is ideal (I use this in two of my three machines).

4. The speed of the CPU can be constrained if the Level 2 (or Level 3) cache is slow or inadequately sized. Max out the supporting cache if possible.

Re OT with RT feed: only one instance at a time per license is legal. But you don’t need RT feed for any testing. So, just use that for actual trading.

Hope that helps.

Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing

Top of the page Bottom of the page
SteveLuerman
Posted 9/6/2020 11:36 PM (#9835 - in reply to #9833)
Subject: Using VM Clones to Speed OT Testing



Elite

Posts: 414
100100100100
Location:
USA: CO, Boulder
Salim,
You are limited to 5 OT installs with data connections. And limited to 1 OT using RT data.

Jeff started this thread to explain how to do TESTING with LINKED-CLONE VMs. By using Linked-Clones, you can make lots of copies and use them for testing, not for installing new instances of OT or plugins or connecting to data sources. YOU SHOULD NOT CONNECT YOUR LINKED-CLONES TO THE INTERNET. If you do connect a linked clone to the internet, it will consume an OT license when it accesses data, and you will need to call Nirvana to reset your license count.

Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing

Top of the page Bottom of the page
SalimHira
Posted 9/7/2020 9:49 AM (#9836 - in reply to #9093)
Subject: Using VM Clones to Speed OT Testing



Veteran

Posts: 183
100252525
Location:
USA: MD, Columbia
Thank you Steve, for the clarification, as prior had kept making assumptions and was not clear, but it makes sense when you download 10000+ bars for testing purposes, there is no need to be logged to internet.

Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing

Top of the page Bottom of the page
JeffBrown
Posted 9/10/2020 3:00 PM (#9837 - in reply to #9093)
Subject: Using VM Clones to Speed OT Testing



Friend

Posts: 45
25
Location:
WI: ,
Great discussion on both clones and system configuration. Over the past couple of years, I've tried different configurations for the VM's. I wrote the original post using VirtualBox, but to squeeze out just a little more performance, I migrated to KVM. I've always used a Ubuntu host so it was a natural alternative to VirtualBox, just like HyperV may be an alternative if you run Windows as your host OS -- or use one of the VMware products; love the choices. I still use linked clones for segmenting a problem and running parts of the problem in parallel as described in the original post, but just use a different hypervisor.

For a base OT install in a VM, I personally use 2 CPU and between 2048 MB to 2560 MB per VM. The 2560 MB is what I use when I'm trading live with ATM and a method that uses 16 market states, 11 strategies (no AI), and 2 years of data. If I add VT to the mix, I up my RAM to 3GB for live trading if I'm running both OT and VT simultaneously. I just haven't needed more.

For testing, I keep 2 CPU but increase the RAM to 3072 MB. I do this because I'm commonly running more than 11 strategies, but I've found that using more than 3GB RAM doesn't help mainly because OT is a 32-bit app and can only access 3GB (from one process). On occasion, I will increase this to 4GB if I see the VM using swap, but that isn't too common. If I see swap engage after setting RAM to 4GB, I look more closely at what I'm attempting to test and make adjustments -- the test is likely to create problems for OT and I get bad results no matter how much RAM or CPU I throw at it.

The above is my general allocation when I'm not training ARM strategies. Because of the parallelism that Hans added to OT, to take advantage of a second thread for training, I needed to make adjustments. I discovered, at least on KVM, I needed set the CPU topology. I use 4 Sockets -- defined as 2 Cores with 2 Threads per core. The VM sees this as 1 socket with 4 virtual processors (viewed from Task Manager's Performance CPU view). These settings are likely different depending on your hypervisor. Through trial and error, essentially, I needed to present to Windows something that looked like 4 virtual processors. Once Windows saw 4 processors, then OT would use a second virtual processor for ARM training.

As for RAM... 6GB. That gives each OT thread 3GB to use (since OT spawns a new thread for ARM training, that thread runs on one of the other processors and can access another 3GB of RAM -- 2 OT threads, using up to 3GB per thread for a total of 6GB).

On the topic of Disk, I currently run with a RAID 1 configuration -- not for performance, but for data protection. These are still spinning disks but I have a M2 SSD as cache to improve the overall throughput. I commonly run between 5 and 7 VMs of different types on this host, but I find that disk IO is often my bottleneck when launching a VM and during OT data activities (EOD data updates, OT Maintenance runs, Database Repair). During these periods, multiple VMs compete for IO resources and this slows down OT analysis runs that need data from disk too. My ATM runs I mentioned above, with 2 VM's kicking off their ATM run 23 minutes after market close, they will take approximately 1.5 hours to run. Conducting those same runs on a quiet system -- less than 20 minutes.

So, I love the ideas on SSD. I've been considering building a Ryzen system with PCI-e 4.0 and using a RAID 6 configuration of M2 SSD attached to a PCI-e 4.0 expansion card (they just need to make that card -- can only find PCI-e 3.0). I've also considered introducing a 2TB SSD to house my VM disks without waiting for the expansion card. I may still do that but need a method to protect against SSD failure. I'm just a little trigger shy since I burned through 2 SSD drives in 6 months -- total data loss. I was just lucky enough to have had a solid backup strategy.

Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing

Top of the page Bottom of the page
Jump to page : 1
Now viewing page 1 [50 msgs/pg]
( E-mail a Link | Printer Version | Search Room )

Owner of site: Jim Dean -- Forum content is confidential, and may not be distributed without written permission.