Iterative development – Agile is the late comer to this party

I’m doing some pre-reading for my Innovation and Entrepreneurship in IT subject as part of my Masters in Information Systems at the moment. Our lecturer pointed us to this guy’s blog. His blog is all about innovation.

A couple of posts stood out tonight.

1. “You are solving the wrong problem” (http://www.azarask.in/blog/post/the-wrong-problem/) recounts the story of Paul MacCready who was the first to successfully build a human powered aircraft. He collected a nice bounty for his trouble, which had stood for 18 years. The key to his success was his ability to design his aircraft so it could be iterated quickly and cheaply.
2. “Iterative design: Towards the perfect paper plane” (http://www.azarask.in/blog/post/iterative_design_isnt_design_by) tells of the author’s iterative design process for building the ultimate paper plane – a goal that he has spent a lifetime perfecting. He relates the problem of “design a better car”. What’s “better” mean? Faster? More fuel efficient? His conclusion is worth quoting here:

“As you design, you aren’t just moving towards a solution, you are learning about what problem your solution is trying to solve. So, don’t stop iterating your design. And more importantly, don’t stop iterating your problem.”

The importance of having technical expertise inside the business from the start of a SharePoint implementation

I’ve been working in the SharePoint space now for nearly 3 years. I was fortunate to be recruited into a SharePoint developer role by my current employer from a support and ASP.NET background. It’s something for which I’m very grateful.

When I joined the company, we had an existing SharePoint 2007 installation that had been entirely specified and built using consultants. There was no internal knowledge of the platform. In the three years since joining, I’ve developed a very broad knowledge of the platform, spanning development, support and administration. My success in this role is augmented by my Masters degree studies, which focus on the systemic view of information technology within organisations. My failures are more straightforward; a result of my lack of knowledge of the platform.

I’ve come to realise several things that I think are important for a company to succeed in its SharePoint implementation, which I want to discuss here.

Briefly, these things are:

• that while consultancies tend to specialise in implementation of systems, they rarely see the “long tail” of the implementation.
• that it is easy to overlook the weight of a SharePoint installation.

The “long tail”

Consultancies tend to specialise in the implementation of systems – whether green-field projects or augmenting existing systems. However, they rarely see the “long tail” of the implementation. By this I mean they don’t have to live on a daily basis with what they’ve built, with everything that entails – for example user support, dealing with the quirks of the platform and implementation or having to find workarounds for the things that don’t quite work as expected.

There are two problems here:

1. it may take several years for problems to emerge. In a current example, we recently discovered there was a logical error in some code that was written for the initial deployment. While one of the original consultants still exists with the company, the people who wrote that piece of code have since left the company. The consultant has moved on thinking they’ve done a great job and have no idea of the downstream problem. A learning opportunity has been missed.
2. some (most? many?) businesses will just work around the errors  but never tell the consultancy. The worst case for the consultant being that all the while those who deal with the headaches are busy telling their mates about the crap product and crap consultancy (I quite like our consultants, when we can get them – good consultants tend to be in demand). How often could this problem actually come to light if the internal IT department had the expertise to be able to highlight issues and actively discussed them among itself and with its users?

Installation weight

Though I wasn’t with my current employer at the time, I’m pretty sure that the installation weight of SharePoint was not appreciated.

So what do I mean by “installation weight”?

Firstly, we have a live installation. Our farm consists of a single Web Front End (WFE) and a single SQL Server – the bare minimum for a SharePoint installation. We have considered over and over putting up a second WFE, purely for failover reasons – we certainly don’t need it for load reasons. We also only index content overnight, so our index server is part of the single WFE. The live installation is the obvious first contributor to installation weight that everyone thinks of, though it’s easy to overlook the complexity within this.

Having a single farm is fine, as long as you are content not to make changes… So, on top of our live installation, we have a mirror of this environment for testing purposes. Once again, this is one WFE and one SQL Server. So straight away we’ve expanded from 2 to 4 servers. All of these must be actively managed in order to maintain the security of the whole corporate environment as well as the stability of the servers themselves.

Now, as we want to develop coded solutions for our installation, we now also need a dev VM running on my local PC (I’m the sole developer). This adds one more server into the installation weight. More servers, more admin overhead, more power, more storage. This VM also absorbs one more Visual Studio license, in addition to the one installed on my host PC.

Maintaining just one environment

This in itself can be hard to appreciate. Much of this stems from the blurry line between admin, configuration and programming. The use of Powershell in SharePoint 2010 brings the potential for more blurring as both admins and devs are expected to use it. (I think it’s a good idea by the way.)

So just to throw a few things out there, we need to do Windows updates, SQL Server updates, SharePoint updates, SQL Server maintenance (TIP: be careful about messing with the permissions to databases! – it hurts!), deployment processes for code, deployment processes for SharePoint Designer stuff, security, security, security (god help you if you get it wrong and need to retrofit it! – TIP: buy a tool to help), etc., etc., etc… By the way, it might be one thing to build out a coded solution, but if you’re the only developer in the company, how is your solution going to be modified if you’re not there? Maybe we need to be looking at a customisable, purchased solution that plugs into SharePoint that can be configured by less technical staff?

Maintaining a SharePoint environment can be a maze of roles and responsibilities. Depending on how your IT department is structured, you may need someone for Windows OS maintenance, someone for SQL Server maintenance and a third person for SharePoint Server maintenance. Again, the demarcation of responsibilities is awkward, bordering on imaginary as a SharePoint or SQL Server admin may or may not have access to parts of the filesystem, the SharePoint admin requires very liberal SQL Server rights in order to apply SharePoint Cumulative Updates, etc. etc.

I’m sure I’ve missed something, but this gives some idea of what is required to keep a SharePoint farm alive.

So why bother at all then?

SharePoint is not the most lightweight platform going around but when the platform is fully utilised, its benefits can be tremendous.

My feeling is that one of the primary errors made when SharePoint is evaluated is that it is considered alongside other CMSes like Django or Drupal. I wonder whether the idea that “it’s just ASP.NET” is significant in the decision to go with SharePoint.

SharePoint is not another CMS, nor is it “just ASP.NET.” SharePoint belongs to an entirely different category of product, which I call Enterprise Portal software. Members of this category are few and include SAP’s EP (Enterprise Portal) and possibly IBM WebSphere. I’m sure there’s a comparable product from Oracle (perhaps what was called BEA WebLogic).

Owning a SharePoint installation

It’s quite reasonable to expect that if you hire a consultant that specialises in a particular product, that you’re going to get a solution based on that particular product.

The question is, once you’ve got a solution based on that product, how are you going to maintain it and grow it into the foreseeable future?

If you outsource the whole implementation to a consultant, without having an internal expert, what happens is this:

1. The design and installation start to grow up.
2. Some time elapses…
3. A “SharePoint guy” is hired. If this guy is like me, they’re not a SharePoint guy at the start…
4. Some more time elapses…
5. The SharePoint guy’s knowledge about the platform grows
6. The SharePoint guy’s knowledge of the installation grows

It’s important to realise that there’s a lag between the installation’s maturity and the SharePoint guy’s knowledge development. All this time though, the platform is being used and is either stagnating (the money for hiring consultants is gone) – a condition that is fatal for the implementation or continuing to evolve (there is ongoing money for consultants). The poor SharePoint guy’s running to play catch up.

A better solution is for the SharePoint guy to come on board while the platform is still immature. In doing this, his knowledge grows alongside the implementation. His knowledge may still not be that of a Microsoft MVP but he’ll be evolving with the platform. In doing so, this person is better equipped to provide advice, revise existing solutions, deliver new solutions and support the existing platform.

I hate business rules!

Actually, I don’t really hate business rules, they’re one of the bits that makes programming creative, useful and challenging. But I do hate it how they’re spread all over the place.

I’ve been working with a consultant to rev our codebase for our CRM installation lately and today was deployment day. As usual, we thought we had everything together. Now, I’ve been finding myself asking, “What’s causing this to run?” or it’s usual variation, “Why is this happening?” Often, these are attached to the comment “That’s odd.”

I suppose the cause here is that business rules are spread all over the place. Some of them are configured through the UI, some (many) are in code, some are in stored procedures, some are in config files lying around the place. There’s no single way of finding out what causes something to execute. Even in our code, business rules are spread all through the code (mixed in with the infrastructure code, guard clauses, error handling code, etc.)

There’s got to be a better way…

Why bother with strategy?

The following is a response I posted to a blog article titled "Four Reasons Why Productive People Hate Strategic Planning." The article riled me a bit and I couldn't help but post.

-----------------------------------------------------------------

Without a strategy, that is, a goal, how do you know you're not just wasting your time with make-work? The purpose of a strategy is to define the end goals and even some of the intermediate goals. It is not intended to lay out the minute detail of how to get there. Without strategy, you're just blundering about in the dark.

With a strategy, you should be able to answer questions like: "How do I know if I'm on track?", and "How will I know when I get there?" Like reading a street directory, you should be able to say things like "If I come across this intersection, I know I've gone too far but I can fix it by making the next two left turns."

Secondly, a strategy meeting must make decisions. That's the implication of the use of the word "plan." I'm betting that you've suffered too many strategic plans that make no decisions or make tactical, implementation decisions, or those decisions just haven't been visible to you.

Let's take a military example. A general defines a strategic battle plan, outlying broad but specific objectives. This gets broken down and along the line, a regimental commander is instructed to advance on a specific port and hold it to facilitate further landings. This gets further broken up and passed down the line, ending up with platoon commanders who meet the enemy and need to decide how to attack them. Ultimately, a section commander (a Corporal) is tasked with sneaking around onto an enemy's flank and attacking individual soldiers. To the corporals, the strategy is something that may be known but their concern is the soldiers immediately to their front. When you're pounding the ground, it's hard to see the far objective. I guess that's why generals sit up high in their ivory towers - so they can see the objective. The platoon leaders and section commanders have different objectives - take this enemy position and don't get hurt trying.

That's what strategy is about. By the way, I'm a pleb. Without strategy to guide me (or at least it guides my boss, who guides me), I'm just wasting my time and that stinks.

Photography - my new obsession

I got to the end of 2009 and realised my life revolved around computers - work, study, (ahem) free time.

So I needed a new hobby.

• Cycling - fun but not always portable.
• Keep fish - they were bound to die early, I don't think my landlord was really going to go for it, I'd forget to feed them or clean the tank enough, and I move home a bit too regularly.
• Get a pet - I rent OK...
• Photography - portable, can take ages or hardly any time at all, I drowned two Olympus SLRs 10 years ago but I'm smarter now, right?

So, one tax return later, new camera gear. And you know what, it doesn't cost as much to make bad photographs anymore. Digital is really great. So I thought I'd give a few impressions of my gear and my re-introduction to photography.

I bought a Canon 450D and Canon 17 - 85 mm kit lens just after Christmas from Ted's Camera.

I checked out about four shops in Melbourne and thought about buying online too. When I spoke to sales reps in the shops in the city, I was pushed towards Nikon gear - D5000 or D90. Having a look at the shop windows, I could have sworn there was a Nikon promotion on.

Second Hand Lenses

They looked pretty good and the market in second hand lenses looked strong (using eBay as the guide). One strength was that Nikon has kept the same mount since the 1970s so in theory, any lens would be compatible. Then I found an article about Nikon lens compatibility. On the D5000, only fairly recent lenses were going to be compatible. Canon, on the other hand had switched mounts when they introduced the EOS range in the mid-1980s and compatibility goes all the way back to then. So re-evaluating eBay, Canon came out on top. Probably not a major issue but I didn't want to cut myself out of part of the market - having a large range of second hand lenses to choose from enhances my ability to play around with gear (and spend less money doing so).

Design and Ergonomics

Secondly, when I held them, the 450D seemed to fit the hand better. The size of the grip and the positioning of the LCD screen off to the left helped with the fit. For someone with big hands, neither the D90 or D5000 seemed to be as comfortable. One nice feature of the D90 was the second dial for making adjustments to aperture, shutter speed, etc. Not that it's bad on the 450D, actually it's quite nice but maybe there are times when it's easier to reach the second dial.

I don't have to dig through menus very often to change settings. Even when I do need to access the menus, I usually only dig through one level (at most two) to get to what I want. Most of the settings I use regularly, I can add to a custom menu, which sadly is a bit too short. :(  (Am I being picky?)

Kit Lens

Yes, OK I could have spent less than I did and bought one of the cheaper kits. However, I'd read nothing good about the basic 18 - 55mm lenses that shipped with them. Plastic mount, lousy manual focus ring, lousy image quality. I also thought the twin kit was just going to be too much too soon (not having shot in 10 years).

The 17 - 85mm looked much better. A ring type USM motor, giving full time manual focus, a decent manual focus ring, metal mount and a bit more reach. Before I drowned my Olympus gear, I ended up settling with a 50mm f/1.8 and a 135mm f/2.8. The 17 - 85mm on the small sensor gives me the equivalent of about 24mm to 135mm so my old range was covered.

The only downside to this nice chunk of glass is that at 17mm it's not too hot. Zoom in a little and most of the problems go away.

All in all, it's a nice single lens kit (reportedly better than the Canon 18 - 200 kit option) and the IS makes up for some of the problems associated with the smaller maximum aperture (f/4 at 17mm, f/5.6 at 85mm).

File Formats

For an experiment, I tried shooting late into twilight recently in RAW + JPEG. Clearly, the RAW processing built into the camera is far superior to the JPEG processing. Areas of the JPEG that were burnt out or too dark, resolved detail in the RAW file. I wonder if a better camera would resolve JPEG files better.

My obsession

Well, this is quite a distraction from computing. I'm seeing the world differently and am looking for opportunities - often while driving my car. Seeing opportunities can be a challenge but I'm learning, again.

Cheers

Liberal Party leadership spill

The Age reports that Tony Abbott (are you kidding?!) is the new leader of the federal Liberal Party. He one by a single vote, which apparently was informal. God help him if he ever refers to a mandate to lead the party.

http://www.theage.com.au/national/abbott-wins-liberal-leadership--by-one-vote-20091201-k1va.html?autostart=1

Experimenting with IronPython

I’ve been playing around with IronPython for a little while now. Until recently, not really doing much more than spelunking around the .NET Framework with the IronPython console. It’s been great for exploring unknown parts of the framework as I can largely throw away the fluff associated with instantiating objects and bothering with types. Dynamic typing is just beautiful.

I’ve been engaged in a really sticky project – one of those projects that I wish I figured out some completely different way of doing – and by and large have been reasonably happy using C# and VB.NET to build it.

When I get completely jaded with Visual Studio and C# / VB, I’m discovering that SharpDevelop and IronPython make not only a refreshing change but really help with making progress when I’m building out a sticky piece of code. Previously, I’d open a fresh project in C# in Visual Studio and go from there. It’s OK, but switching to a dynamic language that is C-like has been great. SharpDevelop even has a console with (hallelujah!) Intelli-Sense! :-)

Once I figure out what I need to do, it’s not a big deal to translate the IronPython into C#. Now all I need to do is convince the boss that there’s no need to do the translation – IronPython is great just as it is (except for not supporting LINQ very well)!

Cheers!