I was really torn on just how big of a difference, using differencing disks in Virtual PC was giving me on saving diskspace, until I spent some time last night with the following tests to see if it was worthwhile for me.
Let me state, your methods for using Differencing disks may be completely different than mine, and in some scenarios, there is big benefit. These are simply “My” opinions based on how I do “My” virtual machines :).
First let me start with a post from my peer at Mindsharp, Andrew Connell titled HOWTO: Use Virtual PC’s Differencing Disks to your Advantage. (Note, Andrew has already stated he needs to do another update on the article, I’m just basing this post on what’s contained in it)
Andrew did a great job of detailing how he uses Differencing Disks to optimize his development environments. The problem I have with this scenario, is on his second section about defragmenting the second layer base disks.
What I found, is if you defragment the differencing disk (Because the second layer, or final layer for that matter, “IS” a differencing disk), it actually causes more diskspace to be used.
Because the differencing disk is a “Delta” of the layout from the original base disk to the new disk. If the defragmenter sees that it’s best to move around files in the differencing disk, that were core files from the base disk, the size of the differencing disk is expanded to record the change from the base disk itself. Therefore if you optimize your differencing disk, you run the chance of completely negating what you set out to do.
To see the difference in results, I performed the following task with some images I already had. Here was the basic layout:
This was my base disk which I believe the name pretty much denotes what it contained. It was precompacted, and defragmented before it was SYSPrepped.
From this base disk, I built out my development environment as follows
krichie.dom consists of the following machines (All sizes in KB, and the Initial Size is the size of the differencing disk “After” defragmenting):
DC and SQL2005 (I placed the SQL databases themselves on a completely different virtual disk, but I’m not including that size here, as it doesn’t come into the overall equation). This is the initial DC for my krichie.dom forest
Initial Size: 4,322,372
Total With Base: 7,563,134
Member server of krichie.dom with MOSS2007 installed on it, as well as Visual Studio 2005 (Yes I even installed MSDN for 2005, I know there are better ways to handle this, but well…I wanted it to be there in case I was disconnected and could not pull up online. I’m pretty sure that at some point, I will re-configure it to put MSDN on a seperate shared virtual disk, so I can port it around and re-use it…thus reduce the overall size of my virtual disk)
Initial Size: 11,996,053
Total With Base: 15,236,815
Member server of krichie.dom with SPS2003 installed on it, as well as Visual Studio 2003 (And yes, MSDN for 2003 on it)
Initial Size: 7,995,332
Total with Base: 11,236,094
Take the initial size of the three differencing disks + the single size of the base disk, and that equates to: 27,554,519. That’s 27 GIG!. The Initial Size of the differencing disks became so large because of the defragmenting. As well, I noticed that precompacting them also increased their size too, again…it’s all because of the delta from the base image.
When I saw this, I realized that I’m not really saving anything from using differencing disks IF I defragment them. If I didn’t defragment them, then yes, I would be saving tons of space, but the differencing disks would be horribly slow because of fragmentation.
I then took each differencing disk, and merged them with the base into new images, and the result was as follows (Values are the merged sizes):
Therefore, with unique non differencing disks, the total size was 27,675,340. An actual disk space loss of 120,821 (120 MB)
But then, I defraged each disk, precompacted, and compacted and the results were as follows:
Afterwards, my total disk space used by the images was 20,853,835, which in the end, saved me 6,700,684, nearly 7 GIG of diskspace.
In conclusion, if you need to fire up quick environments for whatever reason, then yes…using Differencing disks and a base image like this is a dream. But, in my scenario, the overall disk loss doesn’t help me. Yes, I’m going to hate rebuilding these things all the time because of activation (I don’t have a Volume License key for Windows Server :)), but I’ll just live with that for the time being.
I’m probably going to try Invirtus VM Optimizer next as Andrew notes here. Only $70 for the home office edition. Don’t know that I’ll rush out to get it right now, but perhaps in the future.
I’ll post a follow up to this, after I give it a whirl.
5 Replies to “Oh what a difference, a differencing disk makes…or does it?”
I am sorry, but I am not sure where to pose this question since you have retired your blog.msdn.com account, but this question is about the SP Utility, specifically the repartition aspect of it. In test scenario, I tried to move a 72 GB site (part of a 276 GB content db) to its own content database using this tool (which by the way rocks!). However, I’ve noticed that at times I get errors during the process and am not sure how to resolve them since I will need to run it via an SDD. In this case, I started the repartition using the insdd parameter, and the back up ran to completion, but during the “deleting site…” phase a warning came on and mentioned that “There is no Web named ‘/sites/test1′” Can you shed any light on this and what environmental settings need to be optimal for this utility to function properly? I did read an article from SharePoint advisor written by Chris Fields that gave me some insight regarding some of his personal best practices, but even doing so, I obviously am still doing something wrong. Any advice would be greatly appreciated. And again, I apologize for this question not being in the right forum.
You’re not doing ANYTHING wrong :), and you don’t need to change anything in your environment.
Also, this is a proper way to pose these questions 🙂
The problem is not with SPSiteManager, but with that site in question.
I have a theory that it’s one of two things
1) an Orphaned Site
2) Possibly checked out documents that cause that error message to manifest itself, if the site is > 2GB (Like yours) and there are checked out documents. (Something I discovered a couple of months ago)
The easiest way to prove it’s simply a problem with that site, is to use STSADM to try to delete the site.
i.e. STSADM -o deletesite -url http://blahblah.
You’ll get the same error, and shows that the problem is with the site.
As for correcting it,
1) Let’s make sure the URL is valid (It should be, because it’s the same URL we used to back it up)
2) If that’s correct, and it’s still causing a problem, check out the following Orphaned Site articles if you have not already, and let’s make sure it’s not an orphaned site (See the hotfix location referenced here: http://blogs.msdn.com/krichie/archive/2006/06/30/652453.aspx)
3) If it’s the checked out documents issue I’m thinking it might also be, you’ll need to contact Microsoft and report the problem. I’m not sure what the status of that one was before I left.
Lastly, thanks for the “Rocks” comment on SPSiteManager 🙂
I removed it from the CodePlex site (For reasons I’ll note in a minute) but the version that was in the Utility Suite package (And the whole utility suite package itself) is still on the web components download site at Microsoft (I can’t controll that one), but the reason I’m noting it is, there were a couple of bug fixes I put into SPSiteManager that are not in the utility suite package, and were only in the updated version of it at the CodePlex site.
(For one of them, see http://blogs.msdn.com/krichie/archive/2006/09/11/749393.aspx)
The whole reason the utility suite is dead, is documented here:
Therefore, I won’t be putting any real effort into updating them, because I don’t own them 🙂 The names, etc are the property of Microsoft, even though they are completely unsupported by Microsoft.
Mind you, I will be providing some cool stuff eventually as either part of DeliverPoint, or something else 🙂
First off, thanks for the quick reply!
Before I had known about your response, I went ahead and just tried to restore the .bak file to the right site and new content database since that part of the repartition apparently worked. However, before I was able to restore the site, I had to clear out the orphan data(using the stsadm -databaserepair) that was possibly floating out there due to the incomplete delete. After deleting all the corruption, I tried to restore the site and it still would not let me. I soon figured out that there was still a site entry in the config db site table and once that was removed, I was able to begin the restore process.
But I guess my question is: How do I know whether the site prior to running the repartition command in SPSiteManager is either orphaned or has documents check out? Deleting the site would not possible because eventually I will have to perform this in a production environment (during a scheduled outage of course).
As for the fixes for the SPSiteManager that are no longer available. Is it possible to still get the last copy before you decommissioned since I still would like to use it?
Thanks again for all your help!
The only way to truly know, is to run the -databaserepair operation you’ve run.
I don’t think the orphans were floating around due to the incomplete delete, the initial attempt at deleting it, showed that the problem already existed 🙂
As for the fixes to SPSIteManager, I looked back, and in fact the only “Bug” fix was the one I mentioned in my last comment. Everything else was new functionality.
We’ve had a few visitors to our website from this page. I wanted to let you know about a new product of ours which works with Differencing disks in new and exciting ways. Please stop by and check it out – it’s called Libra.
Libra abstracts “delta” configurations from original base images then compacts, optimizes and catalogs configurations as “packages.” Libra Packages are portable and can be used later by the same user or distributed to any number of users so work can be leveraged. Furthermore, with Libra it is not necessary for the identical, binary replica of a virtual machine to be used downstream by another user.