Iterative development – Agile is the late comer to this party

I’m doing some pre-reading for my Innovation and Entrepreneurship in IT subject as part of my Masters in Information Systems at the moment. Our lecturer pointed us to this guy’s blog. His blog is all about innovation.

A couple of posts stood out tonight.

1. “You are solving the wrong problem” (http://www.azarask.in/blog/post/the-wrong-problem/) recounts the story of Paul MacCready who was the first to successfully build a human powered aircraft. He collected a nice bounty for his trouble, which had stood for 18 years. The key to his success was his ability to design his aircraft so it could be iterated quickly and cheaply.
2. “Iterative design: Towards the perfect paper plane” (http://www.azarask.in/blog/post/iterative_design_isnt_design_by) tells of the author’s iterative design process for building the ultimate paper plane – a goal that he has spent a lifetime perfecting. He relates the problem of “design a better car”. What’s “better” mean? Faster? More fuel efficient? His conclusion is worth quoting here:

“As you design, you aren’t just moving towards a solution, you are learning about what problem your solution is trying to solve. So, don’t stop iterating your design. And more importantly, don’t stop iterating your problem.”

The importance of having technical expertise inside the business from the start of a SharePoint implementation

I’ve been working in the SharePoint space now for nearly 3 years. I was fortunate to be recruited into a SharePoint developer role by my current employer from a support and ASP.NET background. It’s something for which I’m very grateful.

When I joined the company, we had an existing SharePoint 2007 installation that had been entirely specified and built using consultants. There was no internal knowledge of the platform. In the three years since joining, I’ve developed a very broad knowledge of the platform, spanning development, support and administration. My success in this role is augmented by my Masters degree studies, which focus on the systemic view of information technology within organisations. My failures are more straightforward; a result of my lack of knowledge of the platform.

I’ve come to realise several things that I think are important for a company to succeed in its SharePoint implementation, which I want to discuss here.

Briefly, these things are:

• that while consultancies tend to specialise in implementation of systems, they rarely see the “long tail” of the implementation.
• that it is easy to overlook the weight of a SharePoint installation.

The “long tail”

Consultancies tend to specialise in the implementation of systems – whether green-field projects or augmenting existing systems. However, they rarely see the “long tail” of the implementation. By this I mean they don’t have to live on a daily basis with what they’ve built, with everything that entails – for example user support, dealing with the quirks of the platform and implementation or having to find workarounds for the things that don’t quite work as expected.

There are two problems here:

1. it may take several years for problems to emerge. In a current example, we recently discovered there was a logical error in some code that was written for the initial deployment. While one of the original consultants still exists with the company, the people who wrote that piece of code have since left the company. The consultant has moved on thinking they’ve done a great job and have no idea of the downstream problem. A learning opportunity has been missed.
2. some (most? many?) businesses will just work around the errors  but never tell the consultancy. The worst case for the consultant being that all the while those who deal with the headaches are busy telling their mates about the crap product and crap consultancy (I quite like our consultants, when we can get them – good consultants tend to be in demand). How often could this problem actually come to light if the internal IT department had the expertise to be able to highlight issues and actively discussed them among itself and with its users?

Installation weight

Though I wasn’t with my current employer at the time, I’m pretty sure that the installation weight of SharePoint was not appreciated.

So what do I mean by “installation weight”?

Firstly, we have a live installation. Our farm consists of a single Web Front End (WFE) and a single SQL Server – the bare minimum for a SharePoint installation. We have considered over and over putting up a second WFE, purely for failover reasons – we certainly don’t need it for load reasons. We also only index content overnight, so our index server is part of the single WFE. The live installation is the obvious first contributor to installation weight that everyone thinks of, though it’s easy to overlook the complexity within this.

Having a single farm is fine, as long as you are content not to make changes… So, on top of our live installation, we have a mirror of this environment for testing purposes. Once again, this is one WFE and one SQL Server. So straight away we’ve expanded from 2 to 4 servers. All of these must be actively managed in order to maintain the security of the whole corporate environment as well as the stability of the servers themselves.

Now, as we want to develop coded solutions for our installation, we now also need a dev VM running on my local PC (I’m the sole developer). This adds one more server into the installation weight. More servers, more admin overhead, more power, more storage. This VM also absorbs one more Visual Studio license, in addition to the one installed on my host PC.

Maintaining just one environment

This in itself can be hard to appreciate. Much of this stems from the blurry line between admin, configuration and programming. The use of Powershell in SharePoint 2010 brings the potential for more blurring as both admins and devs are expected to use it. (I think it’s a good idea by the way.)

So just to throw a few things out there, we need to do Windows updates, SQL Server updates, SharePoint updates, SQL Server maintenance (TIP: be careful about messing with the permissions to databases! – it hurts!), deployment processes for code, deployment processes for SharePoint Designer stuff, security, security, security (god help you if you get it wrong and need to retrofit it! – TIP: buy a tool to help), etc., etc., etc… By the way, it might be one thing to build out a coded solution, but if you’re the only developer in the company, how is your solution going to be modified if you’re not there? Maybe we need to be looking at a customisable, purchased solution that plugs into SharePoint that can be configured by less technical staff?

Maintaining a SharePoint environment can be a maze of roles and responsibilities. Depending on how your IT department is structured, you may need someone for Windows OS maintenance, someone for SQL Server maintenance and a third person for SharePoint Server maintenance. Again, the demarcation of responsibilities is awkward, bordering on imaginary as a SharePoint or SQL Server admin may or may not have access to parts of the filesystem, the SharePoint admin requires very liberal SQL Server rights in order to apply SharePoint Cumulative Updates, etc. etc.

I’m sure I’ve missed something, but this gives some idea of what is required to keep a SharePoint farm alive.

So why bother at all then?

SharePoint is not the most lightweight platform going around but when the platform is fully utilised, its benefits can be tremendous.

My feeling is that one of the primary errors made when SharePoint is evaluated is that it is considered alongside other CMSes like Django or Drupal. I wonder whether the idea that “it’s just ASP.NET” is significant in the decision to go with SharePoint.

SharePoint is not another CMS, nor is it “just ASP.NET.” SharePoint belongs to an entirely different category of product, which I call Enterprise Portal software. Members of this category are few and include SAP’s EP (Enterprise Portal) and possibly IBM WebSphere. I’m sure there’s a comparable product from Oracle (perhaps what was called BEA WebLogic).

Owning a SharePoint installation

It’s quite reasonable to expect that if you hire a consultant that specialises in a particular product, that you’re going to get a solution based on that particular product.

The question is, once you’ve got a solution based on that product, how are you going to maintain it and grow it into the foreseeable future?

If you outsource the whole implementation to a consultant, without having an internal expert, what happens is this:

1. The design and installation start to grow up.
2. Some time elapses…
3. A “SharePoint guy” is hired. If this guy is like me, they’re not a SharePoint guy at the start…
4. Some more time elapses…
5. The SharePoint guy’s knowledge about the platform grows
6. The SharePoint guy’s knowledge of the installation grows

It’s important to realise that there’s a lag between the installation’s maturity and the SharePoint guy’s knowledge development. All this time though, the platform is being used and is either stagnating (the money for hiring consultants is gone) – a condition that is fatal for the implementation or continuing to evolve (there is ongoing money for consultants). The poor SharePoint guy’s running to play catch up.

A better solution is for the SharePoint guy to come on board while the platform is still immature. In doing this, his knowledge grows alongside the implementation. His knowledge may still not be that of a Microsoft MVP but he’ll be evolving with the platform. In doing so, this person is better equipped to provide advice, revise existing solutions, deliver new solutions and support the existing platform.

I hate business rules!

Actually, I don’t really hate business rules, they’re one of the bits that makes programming creative, useful and challenging. But I do hate it how they’re spread all over the place.

I’ve been working with a consultant to rev our codebase for our CRM installation lately and today was deployment day. As usual, we thought we had everything together. Now, I’ve been finding myself asking, “What’s causing this to run?” or it’s usual variation, “Why is this happening?” Often, these are attached to the comment “That’s odd.”

I suppose the cause here is that business rules are spread all over the place. Some of them are configured through the UI, some (many) are in code, some are in stored procedures, some are in config files lying around the place. There’s no single way of finding out what causes something to execute. Even in our code, business rules are spread all through the code (mixed in with the infrastructure code, guard clauses, error handling code, etc.)

There’s got to be a better way…