I was able to spend a little time again this last weekend working on new music material.
Thought I’d link back to my music site here to let you know that for a short period of time, I’m offering up two songs from my upcoming CD (La famille du solénoïde) in their full entirety for a limited time.
Go check out the following postings for more information!
Ok, now back to codin’.
- Keith Richie
Hey SharePoint community – Barracuda Tools wants to hear from you!
As many of you may know, Todd Bleeker – SharePoint MVP, myself and some other really cool and talented developers are in the process of creating and expanding DeliverPoint 2007 – our Enterprise Management tool for SharePoint.
We have some very exciting plans for DeliverPoint and want to include YOUR valuable experiences and input!!! Part of our development efforts include adding new functionality to DeliverPoint to extend what it currently offers and to empower SharePoint administrators and other key SharePoint personnel by giving them the necessary tools to simplify their enterprise.
To help us better understand what some of the most compelling needs within SharePoint Administration are and to make sure we’re not missing any HOT functionality, we’ve created a short online survey.
To express our gratefulness for your participation in the SharePoint Administration and Wishlist survey, we will be holding a weekly drawing to give away free copies of Todd Bleeker’s and Bill English’s newest SharePoint books! Click on the below link to take the survey.
http://www.surveymonkey.com/s.asp?u=678093339266
- Keith Richie
Like my previous post, Purging a SharePoint list of all items and folders, here is another quick peice of code that will allow you to either Import a file system folder (And all files and subfolders) into a SharePoint Document Library, and also export a SharePoint Document Library to the file system.
I had a SPExportWebFiles tool in the old Utility Suite that did the exporting, and an Import feature in SPSiteBuilder. I’ve been waiting to find the time or opportunity to provide both a sample/tool that provided both import and export, and ran into that need today 
So, I thought I’d share.
The code contained in this file: SPIEFolder.txt, contains sample code to do just that, and create the nested folders as necassary to match the target location.
This code should work with either WSS 2.0/SPS2003 or WSS 3.0/MOSS 2007 without change. (I’ve only tested on WSS 3.0/MOSS btw at this time)
(Update 03-06-2007: See this link as this SPIEFolder is now published at http://www.codeplex.com/SPIEFolder )
So, got a folder you want to import into a SharePoint Document Library and you want to replicate the folder heirarchy and contents? Or, want to export a SharePoint Document Library and replicate the folder heirarchy and contents on the file system? This could help!
Hopefully this weekend, I’ll have some time to provide compiled versions, build scripts, and documentation like SPLSBackup for both this and the List Purging code as noted in my previous post. But I doubt very seriously that the document will be as extensive as SPLSBackup , as I don’t see a big need for that .
HTH
Keith Richie
I saw a posting on
microsoft.public.sharepoint.development_and_programming
requesting this, so I thought I would share here as well in case anyone needed this functionality.
The code contained in this file: SPPurgeList.txt, contains sample code to purge a list of all it’s contents and folders (Define WSSV3 for that).
(Update 03-07-2007: See this link as this SPIEFolder is now published at http://www.codeplex.com/SPPurgeList )
If you need to retain the list itself (Maintaining it’s original GUID) without killing the entire list and re-creating, you can use this code to purge the list contents.
It should work on any list or document library. When I get a chance to really run it through the ringer, I may post it on my CodePlex site.
HTH!
- Keith Richie
Since I started my new job, I have not had a chance to take the time and actually relax a bit, and get back to my music, and my music related activities.
Well, fortunately on Sunday (After having to rebuild my main workstation and getting all my audio/video software/hardware re-installed), I was able to take some time and work on the first draft of the latest song for the new CD I’m working on. I spent a bit more time this evening working on it to polish it up for “Initial posting and feedback”
The track is Ouranos the Progenitor (Uranus), and the “Work in progress” CD is currently titled La famille du solénoïde
La famille du solénoïde is a musical tour through our solar system inspired by the book “The Planets” by Dava Sobel. In a sense, it’s a sequel CD to my previous CD The Maestoso Interstellar Suite. (Yep, for those of you paying attention and looking at those sites, I’m using SharePoint as my backend )
Unlike the other tracks currently listed on the CDs page, I’m releasing the current draft in full. The rest of the tracks (Like all the other CD Products I have listed on my music site) are just 2 minute samples of the full tracks. I’ve decided to release Ouranos the Progenitor (Uranus) in full at this time. (Remember, it’s a work in progress, and will not be the final version that shows up on the CD).
That brings the list of drafts up to 8 tracks now, with one more definite track (For Neptune) to be worked up a bit more, then all the “Primary” tracks are complete as follows:
I may do a couple of tracks on the Minor planets, but I have not decided yet. It depends on the overall length of the CD when all the drafts are nearly done.
I would be interested in your feedback of Ouranos the Progenitor (Uranus), or for any of the other sample tracks!
- Keith Richie
With Windows SharePoint Services 2.0 and SharePoint Portal Server 2003, The SharePoint capacity planning guidelines had this to state about Document Libraries and capacity
From the “Capacity Planning for Windows SharePoint Services” (For WSS 2.0) administration guide.
http://www.microsoft.com/resources/documentation/wss/2/all/adminguide/en-us/stsb07.mspx?mfr=true
In the section “About Capacity and Scale Guidelines“
“Documents – Folder - 2,000 – The interfaces for enumerating documents in a folder do not perform well beyond a thousand entries. “
This was not a hard limit, and many customers were not affected by having many thousand more entries per folder. Until…one day When the Database pages lined up just right, the SQL clustered indices statistics were not refreshed, and it was just a very bad day for the SharePoint admin. Then you were in trouble. The only real solution is to reduce the number of items per folder to within capacity planning guidelines. And it solved just about every issue I was ever involved in. That’s why SPSiteManager’s analysis routine checked all capacity planning guidelines. We could easily identify “Where” to go look.
For Windows SharePoint Services 3.0 and Microsoft Office SharePoint Server 2007 (MOSS) the capacity planning guidelines where changed “Slightly”
From the “Plan for software boundaries (Windows SharePoint Services)” Technet article
“Throughput differences between flat document library vs. document library with folders
Throughput for certain operations decreases as the number of items in a folder increases.
The following figure shows the difference in throughput between viewing all items in a document library with and without the effective use of folders, which is critical for scaling. As shown in the graph below, throughput performance degrades as the number of documents increases when flat library storage is used. The quickest drop in throughput occurs when the total number of documents is less than 2,000, from 151 RPS (at 200 documents) to 63 RPS (at 2,000 documents). At 4,000 documents, throughput decreases to about 13 RPS, or an overall throughput decrease of over 90% from an empty library.“
Note: Remember MOSS is built on top of WSSv3 just like SPS 2003 utilized WSSv2 as the platform, so this applies to MOSS libraries as well.
That statement is followed by a table much like the old V2 capacity planning guidelines with the following for Document Libraries
“Document – 5 million per library – You can create very large document libraries by nesting folders, using standard views and site hierarchy. This value may vary depending on how documents and folders are organized, and by the type and size of documents stored.”
I believe it’s worded this way specifically because the “2000″ item statement for folders before was a not a hard limit, and the purpose of the updated guidelines is to stress that this is not a hard limit, and testing showed the performance degradation (And a cool chart!) with the total number of items, not that the minute you add that 2001th item to your folder that you were going to be in some serious pain . The 2000 item limit was more on the “View” side of things. (Go look at the old docs, you’ll see).
The new guidelines have this to state about views:
“Item – 2,000 per view – Testing indicates a reduction in performance beyond two thousand items. Using indexing on a flat folder view can improve performance. – List view”
So there is a bit more separation on the limits for “Views” versus “True Item counts”, but remember, a File in a a Document Library is really just an Item in a list. Its just a reversal when your comparing Document Libraries to Lists (List items have metadata and file attachments, Document Library items are really just list items with metadata, and a blob of data). The point is, that “View” performance cannot be truly compared to overall “Document Library” performance, but your items within a folder are going to directly affect your view performance. Clear as mud?
Again, when SPSiteManager reported these exceptions to capacity planning guidelines, it was done with a “Warning” element, not a “Oh my goodness!! Do something quick!” element But the importance of that call out was there. It’s point was to show you where you may want to start investigating restructuring of your content.
So, back to the question “Can you really have more than 2000 items per folder in SharePoint now?”
The answer is, “You could always have more than 2000 items per folder in SharePoint, even back in WSSv2/SPS2003. It boils down to acceptable performance, even in WSSv3/MOSS2007.“
Well, that’s “My” answer
Remember, the new capacity planning documents state:
“This value may vary depending on how documents and folders are organized, and by the type and size of documents stored.”
The debate will probably never end, as it depends on a wide variety of factors before you “May” be impacted by this, but if you want my opinion on what you should limit your document library folder contents too? Keep them no larger than 2000 items per folder. The same goes for List Items within a list folder too. This is “My” opinion, and not a direct statement from Microsoft. .
I would highly recommend that you subscribe to and read Joel Olesons’ blog at http://blogs.msdn.com/joelo. He’s got TONS of data on this subject linked from his blog.
Ah, I remember the days of sharing emails with “Dances with Goats” back and forth on capacity planning, and the joys there of. Joel, I miss ya buddy, but I’m still watchin’ ya
Till next time!
HTH
- Keith Richie
A big issue that many administrators have with Windows SharePoint Services 2.0 and SharePoint Portal Server 2003 is a very negative impact on performance when performing site collection level backups on production systems. It became such an issue, that the Microsoft SharePoint Product Groups changed the Administrators guide to explicitly reflect a change in support of using site collection level backups.
The Microsoft Knowledge Base Article 889236 was updated as follows (I’m placing the important parts in bold red):
Supported scenarios for using the Stsadm.exe command-line tool to back up and to restore Windows SharePoint Services Web sites and personal sites in SharePoint Portal Server 2003
http://support.microsoft.com/kb/889236
“Important Before you use Stsadm.exe to back up a Windows SharePoint Services site in an active farm, you should lock the site collection. To do this, use the “No Access” lock.“
The administrators guide was updated as well as follows (I’m placing the important parts in bold red):
Backing Up and Restoring Web Sites
http://office.microsoft.com/en-us/winsharepointadmin/HA011608261033.aspx
“Caution Previous versions of this guide recommended using regular backup operations to allow recovery of single document or list items. This usage is no longer recommended. Because the backup operation involves reading a large amount of data, running the backup process frequently can interfere with the performance of the system, including blocking end users’ access to their sites”
The updated Administrators Guide continues on with the following statements :
Site backup and restore affects performance and can cause access errors.
The process of backing up and restoring sites takes up both memory and processing power on your server. In addition, if you have many sites, or a large amount of data in your sites, the backup process can take a long time, and might result in access errors for your users. If you choose to schedule automatic backups in a batch file or script, be sure to run the backup process when server usage is minimal or, optimally, when there are no users accessing data on your sites.
Site backup and restore are not designed to be used when the server is under active load.
If a site is in use when the backup operation is run, the data in that site may continue to change throughout the operation. The resulting backup file may be inconsistent with the actual state of the site and, if you restore this file, the restored site or database will be inconsistent as well. To avoid possible inconsistencies, lock the site collection prior to backing up a site in an active farm, using the “No Access” lock. After the backup is complete, set the lock back to “Not locked”. For information about locking and unlocking a site collection, see the Managing Locks section of the Configuring Site Collection Quotas and Locks topic.
Therefore, before issuing a STSADM -o backup on a site collection, you have to visit the Manage Qoutas and Locks link in Windows SharePoint Services Central Admin via the browser to lock the site before backing it up.
The problem, is that with Windows SharePoint Services 2.0 and SharePoint Portal Server 2003 there is no automated nor STSADM extension to set the sites lock state prior to backing it up. You must use the Windows SharePoint Services Central Administration UI to set the lock state.
This makes it difficult to automate the backup/restore of site collections through script, and requires manual intervention.
Note: In Windows SharePoint Services 3.0 and Microsoft Office SharePoint Server 2007, there are two new operations in STSADM (getsitelock and setsitelock) that negate the need to visit the Central Admin pages. Therefore you can automate the locking from the command line or script.
To help with this, I just posted SPLSBackup for Windows SharePoint Services 2.0 and SharePoint Portal Server 2003 for general use at the following link:
http://www.codeplex.com/splsbackup
The purpose of SPLSBackup is to mimic STSADM site collection level backup/restore but with options for setting/unsetting the sites lock state during the operation, therefore eliminating the need to manually visit the SharePoint Central Admin pages before and after.
SPLSBackup Performs a site collection level backup/restore (just as STSADM -o backup/restore) yet locks the site before backing up.
The package at the CodePlex link also contains the source code for SPLSBackup.
The source and included binary build of the tool is provided as is.
This tool can also be used as a reference to understanding the SharePoint Object Model in greater detail. Feel free to tailor it to suit the needs of your environment.
Hope this helps!
- Keith Richie
Barracuda Tools, owned by Microsoft SharePoint MVP’s Bill English and Todd Bleeker is pleased to announce the launch of their upcoming Beta program for DeliverPoint 2007! DeliverPoint is an enterprise level permissions tool for SharePoint that simplifies permissions related user management within SharePoint and consequently helps organizations save time and money.
Barracuda Tools is currently seeking Beta participants! We are anticipating launching our Beta program within the next couple of weeks. Participants will receive a discount off the regular retail price of DeliverPoint.
If you’re interested in participating in the DeliverPoint 2007 Beta program, please email Bill Kuhn at bill.kuhn@barracudatools.com
I had wondered if the MigrateUser API and STSADM -o migrateuser operation would work with Forms Based Auth accounts in WSSv3, and just validated that it does Whoo Hoo!
For those of you unfamiliar with this addition to WSS, see the previous KB on this addition to WSSv2. It’s what I used from SPUserUtil back in the day to migrate accounts. In fact, there is a call out to it in the overview of the SharePoint Products and Technologies 2003 Software Development Kit. One of my fortunate claims to fame But remember the disclaimer that they even have noted there:
Disclaimer: SharePoint Utility Suite provides a packaged collection of tools and utilities that demonstrate the rich object model that is delivered with the SharePoint Products and Technologies 2003 SDK (which includes documentation for Windows SharePoint Services 2.0 and SharePoint Portal Server 2003). This package includes code and tool examples that SharePoint developers and SharePoint administrators might find useful; however, the samples in the SharePoint Utility Suite are provided as is and are not supported.
Regardless, in WSSv2, it was an API extension to SPGlobalAdmin, but in WSSv3, it’s a method of the SPFarm object and of course it’s still exposed via STSADM.
When using it for a non Windows account, you have to specify the -ignoresidhistory flag for STSADM, or specify false for the enforceSidHistory argment to SPFarm.MigrateUserAccount(), but it works great.
For example:
stsadm -o migrateuser -oldlogin “fbaaspnetsqlmembershipprovider:fbauser2″ -newlogin”fbaaspnetsqlmembershipprovider:fbauser3″ -ignoresidhistory
This will migrate my FBAUser2 account to FBAUser3 for the fbaaspnetsqlmembershipprovider I have setup.
Keep in mind, just as in V2, this does NOT update the Display Name nor the Email address for the account, and is something you’ll have to do post migration.
HTH
- Keith Richie
In my previous post “The SharePoint Utility Suite is Dead” I think I failed to note that another reason it was dead, is that I do not own the right to use the Names of the primary tools in the suite (i.e., SPUserUtil and SPSiteManager). This certainly doesn’t mean that you won’t see something from me as far as helpful tools that provide a “Better” implementation for WSSv3/MOSS2007 that those tools provided. Those tools were designed using ONLY publicly supported SharePoint OM calls, but legally I could not take and use the name SPUserUtil nor SPSiteManager with me after leaving Microsoft.
The other key thing to remember, is that the amount of time I had to focus on those tools was very, very, very, very, very limited. Trust me, I spent a GREAT deal of time thinking about things that could be improved, how to improve them, etc…Only to get to the point where I didn’t have any time to actually implement any of it. Quite honestly, I think I was able to spend MAYBE roughly 40 hours total over the course of the last year on SPSiteManager to bring it from release 2.1 to 2.3. It was very depressing
A couple of notes about the two biggest tools in the utility suite that I personally was responsible for.
SPSiteManager
Someone noted on a blog post somewhere that they “imagined that [someone] would pick up the ball for [WSSv3/MOSS2007]” as far as providing tools that match the SPSiteManager repartition operation. Well, who says I’m not going to do something like that .
In fact, it’s quite possible that you will see a V2 and V3 simple repartition app from me over the next month (or two , I’m a bit busy at the moment). Will it be on the scale of SPSiteManager? No, not in a free tool But who knows what the future holds
That’s one of the whole reasons I decided to take my new job, so that I could focus on generating and putting real time into things such as this See my initial post on this new blog and you’ll see why
Besides, there were tons of updates in MOSS2007 that negate some of the features that SPSiteManager has. Otherwise, I would expect some cool things to come in the next few months
Whether or not they wind up in DeliverPoint as an enterprise ready “Fully Supported” offering or not, is still up in the air, but we’ll see My goal is to do just that.
When I implemented those SPSiteManager features, I did my best to ensure that they were rock solid, but support for it was basically an email to me, and if I had “the time” I would respond.
Fortunately, there wasn’t a lot of “Oh dear, this isn’t working right” emails, and when they came…I was always concerned that I would not be able to find the time to get in and fix it, let alone get an update out for it.
Most of the comments were more on the lines of “Whoa! That’s cool!!!!” So I was thankful not only for your generous comments, but also your encouragement .
Expect to see WSSv3/MOSS2007 similar features from me
SPUserUtil
For the longest time I wanted to completely re-architect SPUserUtil or just bring it’s features into SPSiteManager, but I knew that with the security changes in WSSv3/MOSS2007, a flat recompile and the existing logic would all have to change.
Over the past 2 or 3 weeks, I’ve been focusing on the new interrogation engine for DeliverPoint, and proved my theories as for taking SPUserUtil up to WSSv3/MOSS2007. In other words, if you were to take SPUserUtil as is, change only what was necessary for WSSv3/MOSS2007, SPUserUtil would do some very unexpected things Not only would it do unexpected things, it would not cover the whole gamut of things you need to look at to successfully clone a user in WSSv3/MOSS2007.
While I patiently waited to find some kind of time work on a WSSv3/MOSS2007 version of both SPUserUtil and SPSiteManager, I thought long and hard about how I might implement them. The cool thing, is that those ideas, I’m actually getting the opportunity to implement in DeliverPoint.
For DeliverPoint, there’s still some work to do, but the new engine is already twice as fast, and requires half the memory from my recent performance tests which is really exciting.
In conclusion, I guess what I’m trying to say guys, is just because the “SharePoint Utility Suite is Dead” because “I don’t own it” doesn’t mean that “I am” nor the dreams and desires I had for what it could become. I also believe that I will be able to share more than ever before, for instance, see my previous post on Traversing Sharepoint List Folder Hierarchies.
I hope this clarifies.
- Keith Richie
Recent Posts Recent Posts on indieOpolis (My Music Blog)
