SharePoint 2007 import / export

I have been working to export site content from one SharePoint 2007 server to another using stsadm import / export.
This is my recording of a commonly occurring error that others have seen at least as far back as 2009.
The thorn in the side is that the TEMP and TMP environment variables for the account running the processes (both import and export) need to be located on a drive with sufficient space to contain the entire site being exported / imported.

stsadm –o export

In the export log, the following error will be recorded:

[5/31/2012 12:23:26 PM]: FatalError: There is not enough space on the disk.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.WriteCore(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.FileStream.Write(Byte[] array, Int32 offset, Int32 count)
   at Microsoft.SharePoint.Deployment.ExportDataFileManager.AddFile(Stream fileStream)
   at Microsoft.SharePoint.Deployment.FileSerializer.SaveFile(SerializationInfo info, ExportObjectManager objectManager, ExportDataFileManager fileManager, SPExportSettings settings, SPWeb parentWeb, Boolean isGhosted, String setupPath, String setupPathUser, Byte setupPathVersion, String webRelativeFileUrl, Int32 size, Byte level)
   at Microsoft.SharePoint.Deployment.FileSerializer.GetDataFromDataSet(Object obj, SerializationInfo info, StreamingContext context)
   at Microsoft.SharePoint.Deployment.DeploymentSerializationSurrogate.GetObjectData(Object obj, SerializationInfo info, StreamingContext context)
   at Microsoft.SharePoint.Deployment.XmlFormatter.SerializeObject(Object obj, ISerializationSurrogate surrogate, String elementName, Boolean bNeedEnvelope)
   at Microsoft.SharePoint.Deployment.XmlFormatter.Serialize(Stream serializationStream, Object topLevelObject)
   at Microsoft.SharePoint.Deployment.ObjectSerializer.Serialize(DeploymentObject deployObject, Stream serializationStream)
   at Microsoft.SharePoint.Deployment.SPExport.SerializeObjects()
   at Microsoft.SharePoint.Deployment.SPExport.Run()

The disk being referred to is not the disk that the exported CMP files are to be written to but the disk to which the current user’s TEMP and TMP folders are located.

stsadm-o import

In the import log, the following error will be recorded:

[6/4/2012 11:47:07 AM]: Error: Failure writing to target file [6/4/2012 11:47:07 AM]: Debug:    at Microsoft.SharePoint.Library.SPRequest.ExtractFilesFromCabinet(String bstrTempDirectory, String bstrCabFileLocation)
   at Microsoft.SharePoint.Deployment.ImportDataFileManager.<>c__DisplayClass2.<Uncompress>b__0()
   at Microsoft.SharePoint.SPSecurity.CodeToRunElevatedWrapper(Object state)
   at Microsoft.SharePoint.SPSecurity.<>c__DisplayClass4.<RunWithElevatedPrivileges>b__2()
   at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)
   at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param)
   at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode)
   at Microsoft.SharePoint.Deployment.ImportDataFileManager.Uncompress(SPRequest request)
[6/4/2012 11:47:07 AM]: FatalError: Failed to read package file.
   at Microsoft.SharePoint.Deployment.ImportDataFileManager.Uncompress(SPRequest request)
   at Microsoft.SharePoint.Deployment.SPImport.Run()
*** Inner exception:
Failure writing to target file    at Microsoft.SharePoint.Library.SPRequest.ExtractFilesFromCabinet(String bstrTempDirectory, String bstrCabFileLocation)
   at Microsoft.SharePoint.Deployment.ImportDataFileManager.<>c__DisplayClass2.<Uncompress>b__0()
   at Microsoft.SharePoint.SPSecurity.CodeToRunElevatedWrapper(Object state)
   at Microsoft.SharePoint.SPSecurity.<>c__DisplayClass4.<RunWithElevatedPrivileges>b__2()
   at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)
   at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param)
   at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode)
   at Microsoft.SharePoint.Deployment.ImportDataFileManager.Uncompress(SPRequest request)
[6/4/2012 11:47:57 AM]: Progress: Import Completed.
[6/4/2012 11:47:57 AM]: Finish Time: 6/4/2012 11:47:57 AM.
[6/4/2012 11:47:57 AM]: Completed with 0 warnings.
[6/4/2012 11:47:57 AM]: Completed with 2 errors.

The “Failure writing to target file” is again the location of the TEMP and TMP directories of the currently logged in user. This error is saying that there isn’t enough temporary disk space available.
I think the fact that SharePoint allows you to specify how large you want the CMP files to be leads you to  think that each file will be individually packed / unpacked but this is simply misleading. The entire site is exported to TEMP then written to CMP files or imported from CMP files to TEMP before written to SharePoint.
I could guess that this behaviour continues into SharePoint 2010 but haven’t confirmed this.

SharePoint only allows a WSP to be deployed globally

One more reminder to me…

Description of the Problem

I’ve been experiencing a problem where on installing a WSP to SharePoint, it is only possible to deploy the solution globally.

Some of the other posts surrounding this issue talk about the SafeControl entry. However, in my instance, I’m just deploying a bunch of files - to the Master Page Gallery and Style Library. I don’t even want to deploy a DLL as I have no code.

Solution I found

1. Deploy an assemly to the GAC.
2. In the Package.Template.xml file, add something like the following:

<?xml version="1.0" encoding="utf-8"?>
<Solution xmlns="http://schemas.microsoft.com/sharepoint/">
  <Assemblies>
    <Assembly Location="My.Branding.dll" DeploymentTarget="GlobalAssemblyCache">
      <SafeControls>
        <SafeControl Assembly="$SharePoint.Project.AssemblyFullName$" Namespace="My.Branding" TypeName="*" />
      </SafeControls>
    </Assembly>
  </Assemblies>
</Solution>

After installing this, I can now deploy to any web application. The down side is I’ve now got a DLL polluting in the GAC that I don’t care about.

WCF RIA Services Learning Notes

Why bother with RIA Services?

Because Lightswitch is awesome! Out of the box I can connect to SharePoint lists (no attachment support in v.1) and to SQL Server. For anything else, I’ll need a custom WCF RIA Service. By anything else, I mean (for now) Dynamics CRM.

I plan to write some more on Lightswitch later on, to document my learning as much as anything. One use case I see for Lightswitch is for creating those applications for which one or two people need a different set of screens to more efficiently work with existing data. In many of these cases, there may be insufficient return (or available developer time) to warrant extensive custom development.

My own interest then is to create a stand alone RIA Service, probably wrapping an existing WCF Service (such as those that ship with Dynamics CRM 2011 or SharePoint 2010), stored procedure or other code.

The following notes are mainly for myself – I’ll probably need to find them again some day soon. Neither of the other two people who might read this blog are likely to ever care…

Visual Studio will allow me to “Enable WCF RIA Services” when I create a Silverlight application. My problem with doing this is that like the WCF project templates, it hides so much of what’s going on and also commits me to creating a Silverlight application anyway. It’s not really what I want to do. I guessed the process is worth doing once, so I followed the “Walkthrough: Creating a RIA Services Solution” from MSDN. But any goose can click through a wizard and I’m afraid I can’t say I’m much the wiser for it.

The following is my post-mortem from working through the walkthroughs and how-tos on MSDN for WCF RIA Services.

Walkthrough: Creating a RIA Services Solution

(Source) I found that this walkthrough left me with more questions than answers. Because so much was automatically created and everything was pretty much thrown together, I didn’t know what pieces were what and more importantly which pieces mattered when building a custom RIA Service for use by Lightswitch (or JavaScript).

How-to: Create a Domain Service that uses POCO-defined Entities

(Source) This was a useful walkthrough though it has an error. It was a useful learning experience.

Building the solution according to the steps led to code like the following :

[EnableClientAccess()]
public class SovereignDomainService :DomainService
{
}

public classSovereign
{
    [Key]
    public int UniqueId { get; set; }
    public string Name { get; set; }
    public string House { get; set; }
    public string Dominion { get; set; }
    public int ReignStart { get; set; }
    public int ReignEnd { get; set; }
    public string Sobriquet { get; set; }

    public List<Sovereign> FetchSovereigns()
    {
        List<Sovereign> sovereignList = new List<Sovereign>
        {
            new Sovereign()
            {
                UniqueId=1,
                Name="John",
                House="Plantagenet",
                Dominion="Angevin Empire",
                ReignStart=1167,
                ReignEnd=1216,
                Sobriquet="Lackland"
            },
            new Sovereign()
            {
               // ... Abbreviated for space...
            }
        };
        return sovereignList;
    }

    public IEnumerable<Sovereign> GetSovereigns()
    {
        Sovereign sovereign = new Sovereign();
        return sovereign.FetchSovereigns();
    }

    public IEnumerable<Sovereign> GetSovereignsByReignEnd(int ReignedBefore)
    {
        Sovereign sovereign = new Sovereign();
        return sovereign.FetchSovereigns().Where<Sovereign>(p => p.ReignEnd <= 1500);
    }
}

Generating this, produced an empty code gen file (RIAServicesExample.Web.g.cs) in the client project.

The correct version puts the three methods inside the SovereignDomainService class. The two GetX methods call FetchSovereigns. When built, this produces a considerable amount of generated code. I certainly wouldn’t want to generate this by hand. The Sovereign class gets transformed and marked with the WCF DataContract attribute, also inheriting from System.ServiceModel.DomainServices.Client.Entity, and the properties get transformed and marked with the WCF DataMember attribute.

The SovereignDomainService is transformed into a partial sealed class. This provides the extension mechanism – as the code I’m viewing is generated, I can’t change it as any changes would be smashed next time I build the project. The constructor creates the SVC file.

An interface for the SovereignDomainService is created as an inner interface. It’s fully qualified name in this example is RiaServicesExample.Web.SovereignDomainContext.ISovereignDomainContext. I’ve never seen one before, though I’d say this follows the same rules as an inner class. Namespaces for the FaultContract and OperationContract types are for tempuri.org. I want to find out how to change this.

Walkthrough: Creating a RIA Services Class Library

(Source) I found this walkthrough to be a good next step from the previous how-to. In this walkthrough, I created a Silverlight project (without WCF RIA Services enabled) and a separate WCF RIA Services Class Library project. This project template is found under the Silverlight section. I’m not convinced this is a good decision as WCF RIA Services can also be built for JSON (JavaScript) clients – it’s just a matter of configuration.

Following this walkthrough, I end up with four projects (from the two project templates). The RIA Services Class Library contains the Entity Data Model, while the Silverlight.Web project contains a reference to the RIA Services Class Library Web project.

There are a few manual steps here that aren’t present in cases where you tick “Enable WCF RIA Services” in the New Silverlight Application wizard when you create a new Silverlight project but it’s no big deal. It mainly boils down to adding a few DLL references and modifying your web.config with values that were automatically created in the WCF RIA Services Class Library project – it comes down to a bit of copy/paste.

Summary so far…

Being able to build custom WCF RIA Services are essential for assembling Lightswitch applications that connect to data sources other than SharePoint 2010 lists and SQL Server databases. In cases where you’re not meant to access the database directly (eg. Dynamics CRM), for re-using existing business logic and for accessing other types of data (eg. XML or NOSQL data), you’re going to have to build a custom WCF RIA Service.

I’m not sure whether I have a similar business case for JavaScript clients. I may be better off using JSON / XML directly. I imagine though that if the RIA Service is already built than it looks like I have everything built that I will need, leaving only the necessary configuration to emit JSON data.

NBN mobile towers planning challenges

Here we go again… Mobile towers and health effects, NBN style this time. Interesting that the Libs are really quiet on the topic, considering that their alternative was all wireless. http://www.smh.com.au/it-pro/government-it/nbn-towers-anger-residents-and-andrew-wilkie-20120113-1py5q.html

NBN: To Wireless or Not To Wireless?

I’ve been following the debate on the NBN for several years now. While the debate over whether to go with the optic fibre service currently being planned and deployed by the Labor Government or to scrap it and implement a cheaper(?) wireless based service proposed by the Liberal Party is the hot issue, there’s never been much debate that I’ve found on the reliability of a wireless network.

I had a 3G wireless broadband service at home at the foot of the Dandenong Ranges, 30km as the crow flies from the Melbourne CBD. I got tired of its unreliability and switched to a fixed line service over a year ago.

It must be noted that a proposed national wireless broadband service wouldn’t run (I hope / expect) on 3G so hopefully reliability would be better.

But for me, here’s the clincher – the 3G service claimed a bandwidth of 3.6Gbps. At home, testing against speedtest.net, on a clear day I could get 330kbps. When the weather was bad, I was recording 180kbps or so – sub-broadband speeds. The government says “broadband” begins at 256kbps.

So the most basic wireless fact that every UHF and HF radio operator knows (and the politicians and commentators are forgetting) is that radio performance is heavily affected by the amount of atmospheric moisture. “Next gen” wireless can’t deny physics.

A wireless NBN is a joke.

Mono project IP in trouble?

This article appeared in my RSS reader today.

The global Mono project team has been laid off. Miguel De Icaza and his team have formed a new company (Xamarin) to continue development.

The problem for them is that their non-open source offerings - MonoTouch (Mono for iPhone / iPad) and MonoDroid (Mono for Android) (no mention of MonoMac) are owned by Novell's new owner, Attachmate. Attachmate obviously just wants whatever of Novell's assets it can get.

So there are a few questions.

Firstly, if MonoTouch and MonoDroid were the money making parts of the Mono Project and Xamarin can't own them, how are they going to continue to exist? Can they get sufficient work doing Mono consulting?

Secondly, what intellectual property risks are they running by trying to engineer replacements? These guys built the original products. Are Attachmate going to take legal action claiming ownership over their own personal knowledge? This is a possibility as they (Attachmate) could claim that they went to work for a competitor, building a competing product using intellectual property owned by Novell. How might this play out? How might a court separate the intellectual property owned by a company from the knowledge held by a group of developers (as individuals or collectively)?

Blogging on Innovation and Entrepreneurship

This semester I'm back at uni. I was so grateful for the break - having time to smell the roses and rediscover life. The strangest thing happened - I got married!!!

I'm taking a subject called Innovation and Entrepreneurship in IT. As part of this, I'm keeping a learning stream in a blog at http://mikehansford.wordpress.com.

My current post is called "The right knowledge management strategy is essential to open innovation."

System.Text.StringBuilder is better than System.String for appending for append operations but...

For ages I've thought that by using a StringBuilder I could append to my strings with impunity.

But now I've just come across this post from Karl Seguin. In this post he says:

A StringBuilder is nothing more than a dynamic arrays for strings with some extra buffer space. When it fills up, it too must be copied to a new, larger, memory block.

So this is why System.Text.StringBuilder has several constructors that takes an Int32 with a specified capacity.

Referring to the MSDN library entry for System.Text.StringBuilder, under the Remarks section is this (emphasis mine):

This class represents a string-like object whose value is a mutable sequence of characters. The value is said to be mutable because it can be modified once it has been created by appending, removing, replacing, or inserting characters. For comparison, see the String class.

Most of the methods that modify an instance of this class return a reference to that same instance.

Also under Remarks is a sub-section entitled "Performance Considerations" that explains that whereas a concatenation operation on a System.String object always results in creating a new object, with a StringBuilder, a buffer is maintained and only once this is filled is a new, larger buffer created and the content copied into this.

So... I can't append to StringBuilders with impunity. A better practice that will yield performance benefits is to allocate sufficient space in the constructor. I can't do this with System.String (there's no constructor that allows me to specify a length). System.String objects are immutable.

Iterative development – Agile is the late comer to this party

I’m doing some pre-reading for my Innovation and Entrepreneurship in IT subject as part of my Masters in Information Systems at the moment. Our lecturer pointed us to this guy’s blog. His blog is all about innovation.

A couple of posts stood out tonight.

1. “You are solving the wrong problem” (http://www.azarask.in/blog/post/the-wrong-problem/) recounts the story of Paul MacCready who was the first to successfully build a human powered aircraft. He collected a nice bounty for his trouble, which had stood for 18 years. The key to his success was his ability to design his aircraft so it could be iterated quickly and cheaply.
2. “Iterative design: Towards the perfect paper plane” (http://www.azarask.in/blog/post/iterative_design_isnt_design_by) tells of the author’s iterative design process for building the ultimate paper plane – a goal that he has spent a lifetime perfecting. He relates the problem of “design a better car”. What’s “better” mean? Faster? More fuel efficient? His conclusion is worth quoting here:

“As you design, you aren’t just moving towards a solution, you are learning about what problem your solution is trying to solve. So, don’t stop iterating your design. And more importantly, don’t stop iterating your problem.”

The importance of having technical expertise inside the business from the start of a SharePoint implementation

I’ve been working in the SharePoint space now for nearly 3 years. I was fortunate to be recruited into a SharePoint developer role by my current employer from a support and ASP.NET background. It’s something for which I’m very grateful.

When I joined the company, we had an existing SharePoint 2007 installation that had been entirely specified and built using consultants. There was no internal knowledge of the platform. In the three years since joining, I’ve developed a very broad knowledge of the platform, spanning development, support and administration. My success in this role is augmented by my Masters degree studies, which focus on the systemic view of information technology within organisations. My failures are more straightforward; a result of my lack of knowledge of the platform.

I’ve come to realise several things that I think are important for a company to succeed in its SharePoint implementation, which I want to discuss here.

Briefly, these things are:

• that while consultancies tend to specialise in implementation of systems, they rarely see the “long tail” of the implementation.
• that it is easy to overlook the weight of a SharePoint installation.

The “long tail”

Consultancies tend to specialise in the implementation of systems – whether green-field projects or augmenting existing systems. However, they rarely see the “long tail” of the implementation. By this I mean they don’t have to live on a daily basis with what they’ve built, with everything that entails – for example user support, dealing with the quirks of the platform and implementation or having to find workarounds for the things that don’t quite work as expected.

There are two problems here:

1. it may take several years for problems to emerge. In a current example, we recently discovered there was a logical error in some code that was written for the initial deployment. While one of the original consultants still exists with the company, the people who wrote that piece of code have since left the company. The consultant has moved on thinking they’ve done a great job and have no idea of the downstream problem. A learning opportunity has been missed.
2. some (most? many?) businesses will just work around the errors  but never tell the consultancy. The worst case for the consultant being that all the while those who deal with the headaches are busy telling their mates about the crap product and crap consultancy (I quite like our consultants, when we can get them – good consultants tend to be in demand). How often could this problem actually come to light if the internal IT department had the expertise to be able to highlight issues and actively discussed them among itself and with its users?

Installation weight

Though I wasn’t with my current employer at the time, I’m pretty sure that the installation weight of SharePoint was not appreciated.

So what do I mean by “installation weight”?

Firstly, we have a live installation. Our farm consists of a single Web Front End (WFE) and a single SQL Server – the bare minimum for a SharePoint installation. We have considered over and over putting up a second WFE, purely for failover reasons – we certainly don’t need it for load reasons. We also only index content overnight, so our index server is part of the single WFE. The live installation is the obvious first contributor to installation weight that everyone thinks of, though it’s easy to overlook the complexity within this.

Having a single farm is fine, as long as you are content not to make changes… So, on top of our live installation, we have a mirror of this environment for testing purposes. Once again, this is one WFE and one SQL Server. So straight away we’ve expanded from 2 to 4 servers. All of these must be actively managed in order to maintain the security of the whole corporate environment as well as the stability of the servers themselves.

Now, as we want to develop coded solutions for our installation, we now also need a dev VM running on my local PC (I’m the sole developer). This adds one more server into the installation weight. More servers, more admin overhead, more power, more storage. This VM also absorbs one more Visual Studio license, in addition to the one installed on my host PC.

Maintaining just one environment

This in itself can be hard to appreciate. Much of this stems from the blurry line between admin, configuration and programming. The use of Powershell in SharePoint 2010 brings the potential for more blurring as both admins and devs are expected to use it. (I think it’s a good idea by the way.)

So just to throw a few things out there, we need to do Windows updates, SQL Server updates, SharePoint updates, SQL Server maintenance (TIP: be careful about messing with the permissions to databases! – it hurts!), deployment processes for code, deployment processes for SharePoint Designer stuff, security, security, security (god help you if you get it wrong and need to retrofit it! – TIP: buy a tool to help), etc., etc., etc… By the way, it might be one thing to build out a coded solution, but if you’re the only developer in the company, how is your solution going to be modified if you’re not there? Maybe we need to be looking at a customisable, purchased solution that plugs into SharePoint that can be configured by less technical staff?

Maintaining a SharePoint environment can be a maze of roles and responsibilities. Depending on how your IT department is structured, you may need someone for Windows OS maintenance, someone for SQL Server maintenance and a third person for SharePoint Server maintenance. Again, the demarcation of responsibilities is awkward, bordering on imaginary as a SharePoint or SQL Server admin may or may not have access to parts of the filesystem, the SharePoint admin requires very liberal SQL Server rights in order to apply SharePoint Cumulative Updates, etc. etc.

I’m sure I’ve missed something, but this gives some idea of what is required to keep a SharePoint farm alive.

So why bother at all then?

SharePoint is not the most lightweight platform going around but when the platform is fully utilised, its benefits can be tremendous.

My feeling is that one of the primary errors made when SharePoint is evaluated is that it is considered alongside other CMSes like Django or Drupal. I wonder whether the idea that “it’s just ASP.NET” is significant in the decision to go with SharePoint.

SharePoint is not another CMS, nor is it “just ASP.NET.” SharePoint belongs to an entirely different category of product, which I call Enterprise Portal software. Members of this category are few and include SAP’s EP (Enterprise Portal) and possibly IBM WebSphere. I’m sure there’s a comparable product from Oracle (perhaps what was called BEA WebLogic).

Owning a SharePoint installation

It’s quite reasonable to expect that if you hire a consultant that specialises in a particular product, that you’re going to get a solution based on that particular product.

The question is, once you’ve got a solution based on that product, how are you going to maintain it and grow it into the foreseeable future?

If you outsource the whole implementation to a consultant, without having an internal expert, what happens is this:

1. The design and installation start to grow up.
2. Some time elapses…
3. A “SharePoint guy” is hired. If this guy is like me, they’re not a SharePoint guy at the start…
4. Some more time elapses…
5. The SharePoint guy’s knowledge about the platform grows
6. The SharePoint guy’s knowledge of the installation grows

It’s important to realise that there’s a lag between the installation’s maturity and the SharePoint guy’s knowledge development. All this time though, the platform is being used and is either stagnating (the money for hiring consultants is gone) – a condition that is fatal for the implementation or continuing to evolve (there is ongoing money for consultants). The poor SharePoint guy’s running to play catch up.

A better solution is for the SharePoint guy to come on board while the platform is still immature. In doing this, his knowledge grows alongside the implementation. His knowledge may still not be that of a Microsoft MVP but he’ll be evolving with the platform. In doing so, this person is better equipped to provide advice, revise existing solutions, deliver new solutions and support the existing platform.

I hate business rules!

Actually, I don’t really hate business rules, they’re one of the bits that makes programming creative, useful and challenging. But I do hate it how they’re spread all over the place.

I’ve been working with a consultant to rev our codebase for our CRM installation lately and today was deployment day. As usual, we thought we had everything together. Now, I’ve been finding myself asking, “What’s causing this to run?” or it’s usual variation, “Why is this happening?” Often, these are attached to the comment “That’s odd.”

I suppose the cause here is that business rules are spread all over the place. Some of them are configured through the UI, some (many) are in code, some are in stored procedures, some are in config files lying around the place. There’s no single way of finding out what causes something to execute. Even in our code, business rules are spread all through the code (mixed in with the infrastructure code, guard clauses, error handling code, etc.)

There’s got to be a better way…

Why bother with strategy?

The following is a response I posted to a blog article titled "Four Reasons Why Productive People Hate Strategic Planning." The article riled me a bit and I couldn't help but post.

-----------------------------------------------------------------

Without a strategy, that is, a goal, how do you know you're not just wasting your time with make-work? The purpose of a strategy is to define the end goals and even some of the intermediate goals. It is not intended to lay out the minute detail of how to get there. Without strategy, you're just blundering about in the dark.

With a strategy, you should be able to answer questions like: "How do I know if I'm on track?", and "How will I know when I get there?" Like reading a street directory, you should be able to say things like "If I come across this intersection, I know I've gone too far but I can fix it by making the next two left turns."

Secondly, a strategy meeting must make decisions. That's the implication of the use of the word "plan." I'm betting that you've suffered too many strategic plans that make no decisions or make tactical, implementation decisions, or those decisions just haven't been visible to you.

Let's take a military example. A general defines a strategic battle plan, outlying broad but specific objectives. This gets broken down and along the line, a regimental commander is instructed to advance on a specific port and hold it to facilitate further landings. This gets further broken up and passed down the line, ending up with platoon commanders who meet the enemy and need to decide how to attack them. Ultimately, a section commander (a Corporal) is tasked with sneaking around onto an enemy's flank and attacking individual soldiers. To the corporals, the strategy is something that may be known but their concern is the soldiers immediately to their front. When you're pounding the ground, it's hard to see the far objective. I guess that's why generals sit up high in their ivory towers - so they can see the objective. The platoon leaders and section commanders have different objectives - take this enemy position and don't get hurt trying.

That's what strategy is about. By the way, I'm a pleb. Without strategy to guide me (or at least it guides my boss, who guides me), I'm just wasting my time and that stinks.

Photography - my new obsession

I got to the end of 2009 and realised my life revolved around computers - work, study, (ahem) free time.

So I needed a new hobby.

• Cycling - fun but not always portable.
• Keep fish - they were bound to die early, I don't think my landlord was really going to go for it, I'd forget to feed them or clean the tank enough, and I move home a bit too regularly.
• Get a pet - I rent OK...
• Photography - portable, can take ages or hardly any time at all, I drowned two Olympus SLRs 10 years ago but I'm smarter now, right?

So, one tax return later, new camera gear. And you know what, it doesn't cost as much to make bad photographs anymore. Digital is really great. So I thought I'd give a few impressions of my gear and my re-introduction to photography.

I bought a Canon 450D and Canon 17 - 85 mm kit lens just after Christmas from Ted's Camera.

I checked out about four shops in Melbourne and thought about buying online too. When I spoke to sales reps in the shops in the city, I was pushed towards Nikon gear - D5000 or D90. Having a look at the shop windows, I could have sworn there was a Nikon promotion on.

Second Hand Lenses

They looked pretty good and the market in second hand lenses looked strong (using eBay as the guide). One strength was that Nikon has kept the same mount since the 1970s so in theory, any lens would be compatible. Then I found an article about Nikon lens compatibility. On the D5000, only fairly recent lenses were going to be compatible. Canon, on the other hand had switched mounts when they introduced the EOS range in the mid-1980s and compatibility goes all the way back to then. So re-evaluating eBay, Canon came out on top. Probably not a major issue but I didn't want to cut myself out of part of the market - having a large range of second hand lenses to choose from enhances my ability to play around with gear (and spend less money doing so).

Design and Ergonomics

Secondly, when I held them, the 450D seemed to fit the hand better. The size of the grip and the positioning of the LCD screen off to the left helped with the fit. For someone with big hands, neither the D90 or D5000 seemed to be as comfortable. One nice feature of the D90 was the second dial for making adjustments to aperture, shutter speed, etc. Not that it's bad on the 450D, actually it's quite nice but maybe there are times when it's easier to reach the second dial.

I don't have to dig through menus very often to change settings. Even when I do need to access the menus, I usually only dig through one level (at most two) to get to what I want. Most of the settings I use regularly, I can add to a custom menu, which sadly is a bit too short. :(  (Am I being picky?)

Kit Lens

Yes, OK I could have spent less than I did and bought one of the cheaper kits. However, I'd read nothing good about the basic 18 - 55mm lenses that shipped with them. Plastic mount, lousy manual focus ring, lousy image quality. I also thought the twin kit was just going to be too much too soon (not having shot in 10 years).

The 17 - 85mm looked much better. A ring type USM motor, giving full time manual focus, a decent manual focus ring, metal mount and a bit more reach. Before I drowned my Olympus gear, I ended up settling with a 50mm f/1.8 and a 135mm f/2.8. The 17 - 85mm on the small sensor gives me the equivalent of about 24mm to 135mm so my old range was covered.

The only downside to this nice chunk of glass is that at 17mm it's not too hot. Zoom in a little and most of the problems go away.

All in all, it's a nice single lens kit (reportedly better than the Canon 18 - 200 kit option) and the IS makes up for some of the problems associated with the smaller maximum aperture (f/4 at 17mm, f/5.6 at 85mm).

File Formats

For an experiment, I tried shooting late into twilight recently in RAW + JPEG. Clearly, the RAW processing built into the camera is far superior to the JPEG processing. Areas of the JPEG that were burnt out or too dark, resolved detail in the RAW file. I wonder if a better camera would resolve JPEG files better.

My obsession

Well, this is quite a distraction from computing. I'm seeing the world differently and am looking for opportunities - often while driving my car. Seeing opportunities can be a challenge but I'm learning, again.

Cheers

Liberal Party leadership spill

The Age reports that Tony Abbott (are you kidding?!) is the new leader of the federal Liberal Party. He one by a single vote, which apparently was informal. God help him if he ever refers to a mandate to lead the party.

http://www.theage.com.au/national/abbott-wins-liberal-leadership--by-one-vote-20091201-k1va.html?autostart=1

Experimenting with IronPython

I’ve been playing around with IronPython for a little while now. Until recently, not really doing much more than spelunking around the .NET Framework with the IronPython console. It’s been great for exploring unknown parts of the framework as I can largely throw away the fluff associated with instantiating objects and bothering with types. Dynamic typing is just beautiful.

I’ve been engaged in a really sticky project – one of those projects that I wish I figured out some completely different way of doing – and by and large have been reasonably happy using C# and VB.NET to build it.

When I get completely jaded with Visual Studio and C# / VB, I’m discovering that SharpDevelop and IronPython make not only a refreshing change but really help with making progress when I’m building out a sticky piece of code. Previously, I’d open a fresh project in C# in Visual Studio and go from there. It’s OK, but switching to a dynamic language that is C-like has been great. SharpDevelop even has a console with (hallelujah!) Intelli-Sense! :-)

Once I figure out what I need to do, it’s not a big deal to translate the IronPython into C#. Now all I need to do is convince the boss that there’s no need to do the translation – IronPython is great just as it is (except for not supporting LINQ very well)!

Cheers!

Review of ACS Victoria Branch Forum - Cloud Computing

I attended the Australian Computer Society (ACS) Victorian Branch (of which I am a member) forum this week. Paul Cooper from SMS Management and Technology presented on the topic of Cloud Computing.

He spoke about how cloud computing (of which there are many, incompatible brands) can bring IT costs closer to zero and how we can gain significant processing enhancements for little green cost. He indicated that cloud based infrastructures and applications can mean that a company can purchase capacity through operational expenditure rather than capital expenditure. He made the simplistic challenge early on to try to find a CFO who would rather spend Cap Ex on ICT instead of Op Ex. This would surely be a difficult challenge to meet.

The short summary of my opinion on the talk is that I didn't buy it.

Costs and benefits of reduced infrastructure cost

In arguing for reducing cost (for the company) closer to zero, he did not consider what costs this in itself may have.

Firstly, I think that it becomes too easy to simply buy more capacity. It may become more expedient to do this than to perform systems maintenance (eg. optimising databases, refactoring software, eliminating files and folders from the  backup schedule...) which would otherwise reduce the need for processing power and storage. On a smaller scale, this is already the case - while we can reduce the resource footprint of an application, the cost of developer / DBA time to do this is often greater than the cost of purchasing more powerful hardware.

My second view on this is that we are likely to get something similar to what is happening to music and video - that for many people, it no longer has much financial value. It's value is purely emotional / aesthetic. Some evidence of this is in the prevalence of music file sharing and the rate at which ring tones are purchased. Ring tones themselves are also only several bars of a song, not even a whole one. Bringing this back to IT, what happens in a business when IT no longer has significant financial value? The risk is that while reducing IT costs in infrastructure and transactional assets closer to zero, emphasis could shift away from IT as a focus for innovation within the business. I think that one of the drivers of IT innovation presently is that IT is not cheap and attempting to innovate with it (particularly in the strategic IT Portfolio asset class) is seen as a way of deriving value from the investment.

Out of sight, out of mind?

In implementing cloud infrastructure, we end up with a situation similar to plugging a 240v car into a power point at night. That is, you're not running a green car / IT environment; you're just outsourcing the environmental cost. No longer does the business pay for it, the cloud provider does.

Staying with the power generation analogy, what happens now when instead of paying the green cost of our power in the La Trobe Valley, we outsource that to another country? We have plenty of examples here. Ok Tedi mine in Papua New Guinea comes to mind. Could the effort to save the Franklin River have been applied as successfully? We're not short of examples where if we remove something from sight, we don't care about it anymore.

Paul talked about siting green data centres in places where green power can be harnessed; particularly geo-thermal power in Iceland or New Zealand. Several questions arise: 1. Do we need to destroy natural environments to construct power plants to harness this power source? 2. Is the expense going to be such that companies will prefer to use a dirty data centre in a developing or third world country on cost grounds?

Utility Computing

Paul presented cloud computing as a utility service, subscribing to the views of Nicholas Carr. I have already indicated that Paul dwelt on the cost savings that could be gained in reducing infrastructure capital cost, reducing direct power costs to the business and indirectly reducing transactional costs through gaining more processing power for the dollar / watt of power.

This model has parallels to Carr's idea that the sole aim of technology should be to drive cost as close to zero as possible. Carr's argument is that implementing information technology should be about as complex as "implementing" mains power. The assumption is that everyone uses technology in the same way. Which they don't.

He also envisaged a world where assembling enterprise applications was about as difficult as plugging components together (Lego style). There is evidence of this occurring already, though mainly in an the research stage (I think Microsoft Oslo is an example).

Strategic Benefits for a Company

Tele-working

He proposed that cloud computing enables tele-working. No it doesn't. One member of the audience suggested that with VPN, anyone can tele-work without cloud infrastructure. Paul responded by saying that the cost of this is an extra computer to function as the gateway. This in itself is not necessary. Often, the VPN gateway can be operated on another device. They can be embedded into a router or built into a server. Portal software such as SharePoint or WebSphere also allow a company to expose a significant portion of its data over the Internet to authenticated users. Furthermore, applications like Lotus Notes support a disconnected operating mode where not only email but whole databases can be taken off-line for remote, occasionally connected work.

Tele-work is not a sound business case for implementing a cloud based infrastructure.

"Overload" Processing

This is where real strategic benefit can be gained, though I think Paul did not emphasise this advantage sufficiently. Paul talked about how a company can set up a hybrid cloud, hosting some / many of your applications on internal infrastructure. He was interested in maintaining close to 100% utilisation and bring online externally hosted cloud resources as needed.

This sounds fair, though I think I'd rather have some spare capacity on my internal infrastructure as the normal load ebbs and flows; particularly processing load but also temporary and to a lesser extent permanent storage.

The advantage I see is for a company requiring significant processing resources intermittently. Perhaps an IT consultancy that periodically performs data cleansing / transformation activities on significant quantities of data on projects for clients could take advantage of cloud infrastructure to shorten the cycle times for these operations. They may only need these substantial resources for 6 to 10 hours per week. It would be uneconomical for this consultancy to have this kind of processing power in house, idle for most of the week. The alternative for this company may be to scale down the processing power and just have to make do with the longer cycle times. The risk here is that the live data evolves beyond the state of the data in the set being used for the transformation.

Summary

Ultimately, what do I think about cloud computing? Where is its value? What are the risks? What are the costs?

Cloud computing is being marketed as a cost reducing measure. While this is so, the case for cloud computing is thin. A cost can only be cut once and once everyone else does it, there's no longer any advantage. A company must then either maintain their expenditure on cloud computing, just to "keep up with the Jones's" or must find a way of innovating their way forward with it.

There are environmental risks as I have discussed and legislative risks which I have not discussed.

Availability of the services that exist to date is high, so this is not an issue. Availability will be mostly affected by the reliability of the business's Internet gateway. A DOS attack on a company router that carries hosted ERP or CRM (or Google Apps) traffic will shut down the business until service can be resumed. At least everyone will have time to catch up on filing.

The success of cloud computing will depend on how it can be used to create sustainable competitive advantage for a company that chooses to innovate with it. In order to be more than just a passing fad, argument needs to evolve from the current cost cutting and Green IT issues and demonstrate how it can strategically enhance a company.

CSIRO pursues WiFi royalties | Australian IT

Good news for CSIRO, the Australian Government science and research body. For years, several technology companies (including Dell, Microsoft, Intel and HP) have tried to invalidate patents CSIRO has in the USA concerning wireless LAN (802.1x) standards.

These companies have argued (unsuccessfully) that government patent belongs in the public domain.

Surely, a loss for CSIRO would have had every single government in the world wondering what value there is for government research.

What will happen now? Surely the group of companies involved will appeal the decision. There’s so much money at stake for them. Meanwhile, CSIRO is targeting the next group, likely to include smartphone manufacturers such as Nokia, Motorola and Samsung.

CSIRO pursues WiFi royalties Australian IT

Use VB 9 XML Literals to parse an XML tree of unknown depth

I’m in the process of building an editor interface for a Flex program we’ve written at my work. It’s my first real project using LINQ. I’ve mainly been using C# but decided to take advantage of XML Literals in VB 9.

The structure of the Flex program is held in an XML file. I’ve been needing to read this XML file in such that the whole navigation structure can be held in a tree control. This is different to the Flex program itself which only needs to display part of the navigation tree at once.

To do this, I needed to transform the tree. As I saw it, I had two options: XSLT (yuck) or LINQ.

The XML doc is structured like the example below.

<sections>   
  <section>
    <id>section1</id>
    <label>This is section 1</label>
    <layout>navStructure</layout>
    <properties>
      <tree>
        <!-- Nested collection of node elements -->
        <node label="Blah">
          <node label="Blah 1" url="somedoc.pdf"/>
          <node label="Blah 2" url="somedoc2.pdf"/>
        </node>
      </tree>
    </properties>
  </section>
  <!-- More section elements -->
</sections>

There are several <section> elements with this layout. The <node> elements also have variable nesting levels.

The hard part is in writing a LINQ query that will find all the <node> elements without knowing the nesting depth beforehand.

This first section of VB is in the calling Sub. Note in the second expression hole that calls out to the BuildChild function.

The BuildChild function takes an XElement as a paramater and writes a <node> element with the label and url attributes. An expression hole is then opened which recurively calls back into the BuildChild function on the collection of XElements contained in the current XElement.

Dim doc As XDocument = XDocument.Load("content.xml")
Dim tree = _

    <tree>
    <%= From n In doc.<module>.<sections>.<section> _
        Where n.<layout>.Value = "navStructure" _
        Select <node label=<%= n.<label>.Value %> url="">
                <%= From m In n.<properties>.<tree>.<node> _
                    Select BuildChild(m) _
                %>
               </node> _
     %>
     </tree>

Private Function BuildChild(ByVal element As XElement) _
        As XElement
    Dim retElement As XElement

    retElement = <node label=<%= element.@label %> _
                       url=<%= element.@url %>>
                     <%= From e In _
                            element.Elements                        
                         Select BuildChild(e) %>
                 </node>

    Return retElement
End Function

This is how I’ve solved my problem for now. My understanding is that LINQ is more closely related to functional programming in its feel and I suspect that it can provide a better solution. Recursion is difficult to read and debug and it would be great to get rid of it if possible.

Cheers
Mike

Exploring open source - Part 2

Firstly, a little about my journey with open source software. I'm by no means an "expert" (def: an unknown drip under pressure) but neither am I a total ignoramus.

Intro to Linux - 2001

I came across Linux during my Diploma in IT in 2001 / 2002 where we were exposed to Red Hat 5 or 6 or thereabouts. Here is where I was introduced to the concept of open source software. It didn't last long as I figured that as we lived in a Windows world, I had to keep up with Windows and prioritised my efforts here. Getting a job during the tech-wreck was work enough.

The idea of open source in the Windows world certainly wasn't obvious to me then, so I can't say what the state of it was back then.

Asterisk Implementation

My first real use of open source was when my boss in 2005 set me the task of developing the company knowledge of the VOIP PBX Asterisk. Four years after playing with Linux, I was still able to remember enough to get going quickly. (hooray!!!) I actually managed to get a working installation off the ground, complete with an exim4 service to handle voicemail.

Current Open Source Usage

Since then, in the Windows world, I've been playing with open source software mainly in the software development space but also a little in the support space. Principally, I built a company intranet in DotNetNuke (DNN), a wonderful platform for hosting content management systems, based on VB.NET. Otherwise, my main use for open source has been for development support tools like Subversion and TortoiseSVN (of course), MbUnit, NUnit, NMock2 (yeah so what if Rhino Mocks is "better" - I like the expressiveness of NMock2 and you gotta start somewhere), SharpDevelop and am now dipping into IronPython (the Python implementation for the .NET Framework). I also configured and deployed a helpdesk / support tool in PHP and MySQL (of questionable quality). I'm sure there are others that elude me for now. Some of these tools are either ports from Linux / UNIX apps while others are ports from (or inspired by) Java tools (possibly open source).

It's interesting to note that the purpose of all of these tools have been to make the life of IT staff easier or more productive. No open source software that I have been in contact with has been for the purpose of enhancing end user productivity. DNN is kind of the exception, although its reason for existence was to remove me from the process of posting new content to the intranet. I learned that the IT staff should not be an essential part of the business process. Implementing DNN meant that users could post content themselves with little intervention from IT.

Open Office.org

On the open source front, I have dipped into Open Office several times since version 1.0 (possibly a little earlier) and every time have also dipped out. For a while I used it while studying part time in 2007 while holding down a full time job in preference to my other poor choice - Microsoft Office 97. I continually found that menu choices were hiding somewhere unexpected and getting help was difficult. At least as difficult if not more so than Microsoft Office. The final straw was when editing a table was more difficult than it should have been. I ditched it in favour of Office. At least the thing worked according to my expectation.

My most recent exploration with Open Office was last week when I investigated version 3.1. I opened a moderately complex Word doc, a document from Microsoft “Deployment and upgrade for Office SharePoint Server 2007” (link). It contains an automatically generated table of content, hyperlinks hidden behind display text and some pictures. There might be some tables and bulleted lists. When I opened it, some of these hyperlinks had been inexplicably changed from Arial to Symbol font.

Apologies to the Open Office folks, but this is a deal breaker. If I'm in a position of specifying a new office suite to a company, I am not going to recommend deploying a product that does this. It just has to do this right.

Summing Up

If you get me on a bad day, I'll be a Microsoft bigot and will curse open source to the ends of the earth. It's wrong for me to do this and once I calm down, I'll tell you that I'll consider it if:

• it works reliably
• I'm looking for software that will do a task better
• I don't hate the world on the day you ask. ;)

Till next time
Cheers
Mike

Exploring open source

I’ve spent several days recently in the company of an avid open source implementer and advocate. Until recently, he featured on an Australian radio show as the resident open source (ie. Linux) geek.

Anyway, I’m very firmly in the Microsoft camp so we ended up in an argument over the worth of open source software.

The best thing that came of it was I decided that I really should know about the state of play in the open source world and know what I think (and be able to support it) about open vs. proprietary (or closed source) software. As an designer and implementer of solutions, I feel that I really need to know if I am to make a sound business decision whether to employ it or to avoid it in a given situation.

My pre-condition to this discussion is that open source or proprietary software bigotry has no place in IS design and implementation. It only serves to remove tools from our toolkits.

What I hope to cover in what I expect will become a series of posts will include:

• I want to know what I think about the various licences that exist (eg. GPL, LGPL, MIT, MSPL, etc)
• How can open source contribute to a business in terms of
  • cost savings
  • competitive advantage
• How can a company make a living out of open source development
• Is open source really more secure because anyone can read the code?

At the end of all this, my expectation is that I will have a better understanding of open source and have some opinions which I can support with evidence of where and when I’d consider employing it.