Search | User Listing Vidsite | Calendars | Quotes
Home Page ->  General Discussion -> Operating System & Software -> View ThreadLogon (or Register, or Join TradeTight)

You are logged in as a limited-access Guest.To join TradeTight, first read the info in the Organization & Content room, then click the link above. 

1 Cor 15:58 (NET) ... Always be outstanding in the work of the Lord, knowing that your labor is not in vain in the Lord.


Sticky Using VM Clones to Speed OT Testing
Jump to page : 1
Now viewing page 1 [50 msgs/pg]
Jump to room :
JeffBrown
Posted 5/21/2018 12:43 PM (#9093)
Subject: Using VM Clones to Speed OT Testing



Friend

Posts: 44
25
Location:
WI: ,
How to create a copy (clone) of an OT VM – Testing with ease
Attached PDF has the contents of this thread's presentation

We all have different ideas on what market indicators, strategies, and symbols will make up the most profitable trades. We also have different styles and time frames that we like to trade. OT offers so many different customization options and parameters that it makes our heads spin. Add to it tools like Strategy Wizard that help us iterate through various settings and it can be Herculean! How can we possibly conduct a lifetime of analysis in a shorter period of time?

Cloning can help. I’m sure we’ve all thought about how we could clone ourselves and get more done. While we can’t do that (yet), we can clone a VM – make an identical copy so that we can do twice or three times as much. VirtualBox and VMware support cloning capability. For my example, I’ll use VirtualBox. But before we get into how to clone, let’s first talk about the type of clones.

Full Clone vs Linked Clone
Virtual Machines consume hard disk space. The base Windows VM that we created in another thread as an example (OT-Demo), consumes 16 GB before we added OT to it. Once we add OT, our strategies and plugins, data, and a few other useful pieces of software, our VM could easily grow to 60 GB. Multiply that by the number of VM’s you want to use for testing and suddenly you are out of disk space.

Once you get comfortable using virtualization, you may want to create specialized VM’s for specific tasks such as live trading vs strategy testing. We call this VM sprawl. This also consumes disk. So, the first challenge you are likely to face is running out of disk space.

Creating a Full Clone will not help you conserve disk space, but it does help you save time. For instance, when we built the OT-Demo VM, the installation of Windows probably took 20 minutes. Adding OT and all the plugins could take another hour. Add in updates and reboots and we spent say 2 hours creating that one VM. Now we want to create another one to use for testing or live trading. A full clone will give us an exact copy of an existing VM. So, if we cloned OT-Demo that occupies 16 GB, the clone will take 16 GB. But since OT-Demo already exists, it will take a matter of minutes instead of hours to create. That’s the time saver of a Full Clone – fast create of a new VM based on an existing VM.

A full clone is a copy of a VM. They are not connected in any manner meaning you can move it, delete it, change it, and there is nothing used or shared with the original VM.

But I said our problem is likely to be shortage of disk space, so why would I choose a Full Clone? I’ll cover that in a moment. First, let’s describe a Linked Clone.

A Linked Clone is a method of cloning/copying that links the new VM’s hard disk file to the original VM’s hard disk file (the VM that you used as the base). This method saves valuable disk space. In the example of OT-Demo where its hard disk file is 16 GB, a linked clone to OT-Demo (let’s call it OT-Clone) may only take up 2 MB – yes, 2 MB vs 16 GB. Why the difference and how can I fit a full OS in only 2 MB of space? The answer lies in the use of snapshots.

A linked clone takes advantage of the snapshot capability of the virtualization software. It begins by taking a snapshot of the current state of the base VM (OT-Demo) to preserve that point in time. It then creates a new disk and attaches that to the new VM (OT-Clone). This new disk will hold only the changes made to OT-Clone that are different from what is found in the OT-Demo point in time snapshot. So, since OT-Clone was just created, there is essentially no difference between OT-Demo and OT-Clone. As you use OT-Clone, this will not be true and the 2 MB disk of OT-Clone will grow in size. But you can already see the advantage from both a time to create perspective and an amount of storage consumed perspective.

So, a Full Clone makes an exact copy of an existing VM that becomes essentially a new standalone VM. A Linked Clone creates a new VM but shares a virtual hard disk with the original base VM – they are linked together at the storage level through the use of snapshots.

Now, back to the question of why would I choose a Full Clone since it consumes more storage? The simple answer is for performance and resiliency. Here’s what I mean, and sorry, this is going to get a little technical.

When you make a Linked Clone, a snapshot is made against the main disk of the base VM (OT-Demo). There are actually a minimum of 3 files introduced. The first is a point in time snapshot of the base VM’s disk. A second snapshot is created to capture changes caused by running OT-Demo from this point forward. Finally a third snapshot is created to capture changes caused by running the clone (OT-Clone) from this point forward.

When you run OT-Clone and it reads the virtual disk, the virtualization software needs to first check to see if the disk blocks of the OT-Clone snapshot are different from the point in time snapshot of the original OT-Demo disk. If they are different, the virtualization software returns the data found in the OT-Clone snapshot, otherwise it returns the data from the original OT-Demo disk. It is essentially a cascade of checking first the content of OT-Clone’s disk and if not found there, then check OT-Demo’s original disk. This takes time. OT isn’t terribly disk I/O intensive so you won’t see much of a performance impact (likely imperceptible – we are talking about milliseconds of time).

As for resiliency, we have to consider what might happen if that one base disk (the original OT-Demo disk) becomes unavailable. Maybe we saw it in the directory and it had an old timestamp on it and decided to delete it. Or perhaps the physical disk drive failed and the original file now has a bad set of blocks. Now, let’s run through the cascading logic – check the clone first and the block isn’t found there so let’s move on to the original disk and… BAM! It doesn’t exist or is faulty. What happens to the OT-Clone VM? What happens to the OT-Demo VM? Essentially, they both may not function. You may then need to rebuild both of the VM’s.

On the other hand, if you had created OT-Clone as a Full Clone, the only VM affected would be the OT-Demo VM since it had the bad virtual disk. So, performance and resiliency are the main reasons you’d consider using a Full Clone.

So, I should use a Full Clone for OT testing? No – use a Linked Clone unless you intend to move the new VM to a different computer. File damage is rare. It can happen, but it is rare. The newer your computer, the less likely you are to encounter file problems. You can also take a backup of that original file and restore it if it becomes corrupted. Save space and use a Linked Clone.

How about the amount of maintenance? Is there less maintenance to the OS or OT if I use a Linked Clone? Short answer – No. Once you make the clone, from the view of each VM, you have a running OS. It needs to be patched/updated. OT enhancements become available and you need to install the new version. In both cases, you’ll perform both of those activities in each VM – twice as many VM’s means twice as much maintenance. So, if you notice you aren’t using a VM, delete it. Free up that disk space. You can always create a new clone within minutes.

Alright, enough on Full Clones vs Linked Clones. Let’s get into preparing a VM to create clones for testing.


Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing


Attachments
Attachments UsingVMClonesToSpeedOTTesting.pdf (986KB - 5 downloads)
Top of the page Bottom of the page
JeffBrown
Posted 5/21/2018 12:44 PM (#9094 - in reply to #9093)
Subject: Using VM Clones to Speed OT Testing



Friend

Posts: 44
25
Location:
WI: ,
OT Testing with Multiple Clones

You’ve got that great idea, but it will take time to run the tests. You want to continue trading but you also want to run a few tests. Clones to the rescue.

Step 1: Prepare OT for the testing in the VM you want to clone – the “base VM”
It’s a good idea to install all the strategies, plugins, and load the data you want for your tests before you clone the VM. As discussed in the prior section, this will help minimize the amount of hard disk space.

Step 2: Create a Linked Clone (unless you really need a Full Clone).
To create a Linked Clone, go to the VirtualBox Manger, right click on the VM you want to clone, and select Clone from the popup menu.

(Clone-1.png)




Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing


Attachments
Attachments Clone-1.png (153KB - 1 downloads)
Top of the page Bottom of the page
JeffBrown
Posted 5/21/2018 12:47 PM (#9095 - in reply to #9093)
Subject: Using VM Clones to Speed OT Testing



Friend

Posts: 44
25
Location:
WI: ,
You will then be greeted with the window below:
• Change the Machine Name to something different if you’d like.
• Select Linked Clone
• Check the box to Reinitialize the MAC address. This isn’t required but it helps avoid network trouble if you leave the network setting active. We will be deactivating the network shortly.
• Click Clone.

10 seconds later I have a new VM. That was easy!

Do you want another clone? Repeat the steps above beginning with right clicking on the original VM again. Be sure to give it a unique name. This time, 1 second later I have a second clone.

(Clone-2.png)




Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing


Attachments
Attachments Clone-2.png (141KB - 1 downloads)
Top of the page Bottom of the page
JeffBrown
Posted 5/21/2018 12:48 PM (#9096 - in reply to #9093)
Subject: Using VM Clones to Speed OT Testing



Friend

Posts: 44
25
Location:
WI: ,
When you are done cloning, you will see the original base VM (OT-Demo) and the 2 clones I created. Looking at the Snapshots detail, you can see the cloning process made a snapshot for each of the clones. You don’t have to do anything with these, but it confirms what you read above about Full Clones vs Linked Clones.

(Clone-3.png)




Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing


Attachments
Attachments Clone-3.png (113KB - 1 downloads)
Top of the page Bottom of the page
JeffBrown
Posted 5/21/2018 12:50 PM (#9097 - in reply to #9093)
Subject: Using VM Clones to Speed OT Testing



Friend

Posts: 44
25
Location:
WI: ,
Before we use the clone, we will deactivate the network for each of the clones. You don’t necessarily need to do this and in some cases you may want the network active. But, here’s the consequences.
• When Windows starts, it looks for a network connection. If it finds one, it may decide to conduct maintenance (download the latest Windows Update, update the virus definition…). This consumes network bandwidth and CPU. If it downloads data, it will save it to the virtual disk – consuming your valuable hard disk space. And since you will remove these Clones when you are done with your testing, do you really want Windows to waste time doing maintenance?
• When Windows does maintenance, it also check for Product Key registration. Windows will likely treat this temporary VM as a new installation of Windows and using an existing registration key. It usually doesn’t impact your base VM since you registered it with Microsoft, but why take the chance that you’ll have to talk with Microsoft to clean up its license registry when you delete the clone?
• OT itself is licensed software. Nirvana gives us and ability to run 5 instances. If you add a plugin or get your OT license keys, you are likely to exceed the 5 at some point (maybe not immediately, but if you create and destroy clones with network enabled, you will eventually do this). You then need to make a call or email the Nirvana support team to have your license keys reset. That wastes time – both yours and Nirvana’s.

So, let’s deactivate the network.
• Click on the Clone to select it.
• Click on the Settings (Red box) or you can click on the bold Network in the VM detail section of the VirtualBox Manager.
• Select Network if it isn’t already selected.
• Change the “Attached To” from NAT to “Not Attached”
• Click OK.

(Clone-4.png)




Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing


Attachments
Attachments Clone-4.png (126KB - 1 downloads)
Top of the page Bottom of the page
JeffBrown
Posted 5/21/2018 12:53 PM (#9098 - in reply to #9093)
Subject: Using VM Clones to Speed OT Testing



Friend

Posts: 44
25
Location:
WI: ,
“WAIT,” you say. “I forgot to download all the data I needed… How do I activate the network again?”

If your VM is not running, just reverse the process – change the network back to NAT or Bridged.

Both of these options (NAT or Bridged) will connect you to the internet. NAT stands for Network Address Translation. It will essentially use your host computer’s IP address for communicating with the network. If you select Bridged, then your local network (home network) will assign your VM a unique IP address for your local network. That's just a little more technical jargon for you, both will get you to the internet.

Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing

Top of the page Bottom of the page
JeffBrown
Posted 5/21/2018 12:55 PM (#9099 - in reply to #9093)
Subject: Using VM Clones to Speed OT Testing



Friend

Posts: 44
25
Location:
WI: ,
What if my VM is running… Can I change the Network Setting?

You certainly can! Some settings require you to shut down the VM before making the change. This means shutting down the OS inside the VM. However, the network is not one of those changes that need the OS shut down. Let's activate the network on a running VM:

• Right click the Network icon in the bottom right of the VM window (Red box).
• Select Network Settings.
• You will be greeted with the same dialog box as above. Change the Attached To: to either NAT or Bridged.
• Press OK.
• Your network will now be attached. Remember to detach it once you are done collecting your data.

(Clone-5.png)




Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing


Attachments
Attachments Clone-5.png (174KB - 0 downloads)
Top of the page Bottom of the page
JeffBrown
Posted 5/21/2018 1:02 PM (#9100 - in reply to #9093)
Subject: Using VM Clones to Speed OT Testing



Friend

Posts: 44
25
Location:
WI: ,
Other Helpful Tips

Snapshots are your Friends - learn to love them

OT is a powerful tool, but it can be fragile. Occasionally, it will crash or throw an error. This is okay and you can easily restart OT, but what if it occurs in the middle of a long Strategy Wizard run? Maybe you are 1 week into the run and OT crashes. That’s devastating – a week lost.

Here’s a helpful tip, create a snapshot periodically. If OT crashes, simply restore to the most recent snapshot, start the VM and you are back running. Yes, you did lose all the work that was done from the point of the snapshot to the point of failure, but at least it isn’t all gone.

When I’m running something that I know will take a week or month to run, I introduce automated snapshots – I make my host periodically run a command that will create a snapshot every 6 or 12 hours. Since snapshots take up space, I go through and delete the earliest snapshots every few days to reduce the number back to 1 snapshot for each testing VM. How do I do this? I’ll give you the basic VirtualBox command, but you’ll have to create your own script and google how to set up the schedule on your host computer.

I place the following command in a file I call vmsnap.bat

“C:\Program Files\Oracle\VirtualBox\VBoxManage.exe” snapshot “OT-Clone-1” take Auto --description “Snapshot During Analysis” --live --uniquename Timestamp


• This is one line.
• The dashes are double dash.
• Change “OT-Clone-1” to the name of your VM.
• Quotes should be typed as well – they solve the problem of embedded spaces.
• If you have multiple VM’s you are using for testing, just duplicate the line and change the name of the VM.
• When the bat file is executed, it will then snapshot both of the VM’s.

The same command format works for Linux and OS X. The command basically instructs VirtualBox to take a snapshot to be named Auto and adds a description. It is to take the snapshot against a live (running) VM and include the Timestamp in the name if it is needed to make the name unique.

Now use the Windows Scheduler to run the vmsnap.bat file every so often. You’ll have to google how to do this for your version of Windows.

Happy testing!


Use Cloning to Create a Template

This is perhaps a more advanced topic. A template can be considered a base VM that you want to use to use as the base for other VM's.

For instance, during the create of the VM with just the OS but before you installed OT, you have a pristine copy of a VM with Windows. If you plan to create non-OT instances, this would be a good spot to conduct a full clone of the VM -- maybe call it "WindowsTemplate". You might do the same after you installed OT with your plugins and call it "OT-Template".

This gives you a known VM upon which to spawn other, independent VM's.



Edited by JeffBrown 5/21/2018 1:25 PM


Thread moved by JimDean on 5/21/2018 5:42 PM from Custom TradeTight Routines > Universal Instructions & Tools > Using VM Clones to Speed OT Testing

Top of the page Bottom of the page
Jump to page : 1
Now viewing page 1 [50 msgs/pg]
( E-mail a Link | Printer Version | Search Room )

Owner of site: Jim Dean -- Forum content is confidential, and may not be distributed without written permission.