Andy's Mind

Someone has sent me a link to a quite emotional but interesting article by Tim Bray on why the world of enterprise systems delivers so many failed projects and sucky software while the world of web startups excels at producing great software fast. Tim makes some very valid points about technology, culture and approach to running projects. It is true that huge upfront specs, fixed bid contracts and overall waterfall approach are indeed culprits behind most failed IT projects, and that agile, XP and other key trends of recent years can help.

However, I don’t think they can really cure the problem, because we are facing a deeper issue here: the overall overcomplexity in our civilization.

Main drivers of this overcomplexity are bloated states and economy dominated by corporations. Both states and corporations have IT systems today – and the complexity of those IT systems has to reflect the complexity of organisms and processes they try to cover.

The IT system for a national health care system or a state run compulsory social security “insurance” is a very good example. It must be a complex mess because what it is trying to model and run is a complex, overbloated mess – in most cases a constantly changing mess. And it can’t be launched early because it is useless unless it covers the whole scope of what it is supposed to do: because most of what it covers is regulations and laws you can’t deliver a system that meets half of the regulations or 10% – it can’t be used. By the very nature of the domain the system has to be launched as a finished whole.

Plus, on top of all that, comes the scale. If you can imagine a completely privatized health care no system will ever cover all citizens – each doctor, hospital, insurer etc. will cover just its clients, a subset of the population. A system like NHS has to handle all of the UK’s population by design.

Same problem with corporations, especially those that have been around for long (by long I mean decades, not years): scale and mentality. You just can’t manage 75 thousand people easily, especially if they are spread around the globe, in a simple and agile way.

Just think of all accounting requirements global corporations have to handle with their IT systems – but this is just the tip of the iceberg. Whole world economy floats in a sea of legislation – legislative diarrhea of the last decades produced a legal swamp which is a nightmare to understand let alone model a system to comply with it. For a global corporation multiply that by all the countries it is in and stick some international regulations on top of this. This is something corporate systems have to cope with.

What is also important – much of that overcomplexity is computer driven: it would not have been possible if not for the existence of IT systems and computers that run them.

Take VAT tax – it is so complex I always wonder what idiots gave the Nobel prize to the moron who invented it (well, I used to wonder about that when Nobel prize had any credibility). Clearly, implementing it is completely impossible without computers & systems everywhere.

Same about the legal diarrhea I mentioned – I think it can be largely attributed to Microsoft Word. Ever wondered why the EU Constitution (now disguised as “Lisbon Treaty”) has hundreds of pages while the US Constitution is simple and elegant? Well, they couldn’t have possibly written a couple hundred page document with a quill pen which forced them to produce something concise.

But going back to the key issue of whether the corporate IT systems can be better: they can, but a deeper shift in thinking is needed. Instead of creating huge, complex systems corporate IT should rather be a cloud of simple, small systems built and maintained to provide just one simple service (exactly what web startups are doing – each of them provides simple a service, together they create a complex ecosystem). However, this shift would have to occur on the organizational level too – large organizations with complex rules should be replaced with small, focused entities with simple rules for interaction between them.

But to get there we would need a world-wide “agile adoption” reaching well beyond IT. But that means a huge political change, that is nowhere on the horizon. Unless, of course, one other enabler of our civilization’s overcomplexity fades: cheap, abundant energy.

As agile software developers providing outsourcing services we are frequently asked to sign different NDAs and MNDAs. After over two years and dozens of NDAs I noticed a certain pattern which I will call “The Law of NDAs”.

It goes like this:

“The originality and value of the idea protected by an NDA is inversely proportional to said NDA’s length, penalties involved and insistence on signing it.”

In other words, on average, the more harsh and menacing the NDA is the less original and innovative the idea supposedly protected by it turns out to be.

Interestingly, not only average NDAs and ideas fall under this law, but also do extreme cases. For example, I remember one guy who had a 7 (seven) page long NDA to protect his revolutionary idea that turned out to be yet another social network (which, as far as I know, didn’t in the end see the light of day). Conversely, we had a group of high-profile European entrepreneurs who shared with us their truly revolutionary idea (related to multimedia) without even asking us to sign anything.

One may wonder why it is so, but for now I’m satisfied with having observed the pattern. Has anyone else noticed it?

David Harvey wrote a piece on his blog entitled “The Scrum picture is wrong” where he says that the well known, canonical even, picture of Scrum loops errs by focusing too much on the product (deliverable, “potentially shippable product increment”) and forgetting that continuous improvement of the team is another important outcome of each sprint that should be shown with another loop.
[Go and read David’s piece now if you haven’t yet]

As much as I agree with David’s insight I don’t think his version of the Scrum loop should be used to “sell” Scrum outside of our community. I very much value all the focus on teams and their improvement, but it helps to understand that this is something that is important (and should be!) only for us, sitting deep in the software development community. Clients ultimately pay for products, not for our methodologies, our teams improving or our overall well being and happiness.

Let’s take the case of outsourced development, which I know firsthand. Our clients pay us for the product they get from us, but once the project is done they are gone and I don’t think my team getting better on their project is something that even crosses their minds. And why should it, anyway? Unless they would want us to extend their product or start another project with us there is no benefit there for them. What really counts is whether the product we delivered will allow them to meet their business goals, their commitments – and their bottom lines. So when I work to convince them to forget fixed bids and go with Scrum I don’t waste the attention they give me on telling them that thanks to Scrum my team will get better.

But also if you have your own, in-house teams, building your product then in the long term it is that product that counts more than the teams. Why? Well, because not only your clients don’t pay for your teams, but for the product – you also don’t really own your teams. People change jobs (one call from Google’s recruiters to your best people can make your life very hard, believe me), get sick, even die – that’s inevitable. And (as I have learned the hard way) pampering your best people won’t prevent them from quitting – so is probably not worth it. In the long run – measured in years – it counts how much value your clients get from your product, not how perfect your team has become. If your team is getting better with every sprint but your product doesn’t sell you will hit the bottom hard sooner or later. Yes, your company may be then one of those legendary places that excelled technologically but are no longer around (Commodore, SGI etc.) which may be fine for the employees – they will just change jobs – but is not fine for owners and/or investors who will have to cover the loses.

So, team getting better is a tool, a mean, to get a better product, not the other way around (unless you run a monastery – there indeed work is just a tool for monks to perfect themselves spiritually).

Just to make sure no one misunderstands me: I’m not saying we should forget about team improvement, not care about our workers’ well being, career development etc. Yes, we should do all that and more, help everyone excel in what they do, make sure as much as we can people like what they do, heck, do it with passion and care, using fully their potential. All that is important and valuable. But we should not forget that in the end it is the client paying for a product who makes it all happen. And all our methodologies, frameworks and diagrams, all our “management science”, all techniques and research is aimed at improving efficiency – and that means, sorry to be blunt, getting better products faster and cheaper.

So I think the current Scrum picture is quite right in its focus on the product. And the message is spot on. It should be still used to “sell” Scrum to managers, clients and businessmen. David’s diagram is insightful and fine, but let’s keep it within our world of process freaks, agile activists and scrum preachers.

Departing from my usual topics I want to share an observation on the big picture that we are being fed by politicians, mainstream media and most of the Internet.

This big picture is – in short – that we humans are damaging the planet Earth, causing catastrophic global warming driven by CO2 emissions due to burning fossil fuels (mainly in cars), as well as having kids, eating meat etc. We are also facing a huge energy crisis, economic crisis, swine flu pandemic and terrorists. Doom is just around the corner, there is no hope unless we submit to a whole portfolio of totalitarian policies proposed by the very same politicians in the very same media outlets.

Apart from whether all of this picture is true (most is not) what is interesting here is a complete lack of any positive program for our civilization.

It seems the only thing our elites can come up with is downsizing humanity by all means possible from contraceptives to choking off industrial production with this whole "carbon tax" scheme. But this is not a positive program, it is at best an idea on how to prolong status quo for those left, especially those on top of the chain. Let’s suppose we will cut human population by 90% (as some "ecologists" – or rather antihuman madmen – are suggesting) – what next? Where is the path upwards, not downwards?

What is amazing is that people in general are not noticing this lack of any long term prospective. They seem so scared of (mostly fake) dangers they see in the media each day they don’t see our leaders have no idea what to do. Or, worse, they have an idea, but it doesn’t include most of us.

All of this is extremely short sighted. Notice that in all of those doom scenarios Earth is being presented as a limited, closed environment – something akin to Eco Spheres. Global warming models, for example, generally don’t factor in the influence of the Sun, space radiation, Earth orbit cycles & changes and so on. So do those who talk about energy and resources scarcity.

The fact is, however, that Earth is just a part of the unimaginably huge and complex system called the Universe. This is our world, not just this tiny rock. And universe is full of energy and matter. Even our own solar system is full of energy and matter way beyond anything we humans may need for centuries. Just the solar energy descending on Earth in the visible spectrum every day is way beyond all energy needs we have or may have in foreseeable future.

Also, as it has been shown, the limit of Earth’s atmosphere is as much a hard limit for life as the water surface is for fish. Fish can’t walk on the land, we can’t get to space without lots of energy and technology – but it doesn’t mean the world just ends there (like they thought flat Earth ends in Middle Ages). Even the limit of Earth’s atmosphere is just an arbitrary line drawn by humans to simplify things – in fact it is a changing continuum between the dense atmosphere on the ground and the vacuum of space above it (and even that vacuum is not completely empty). And that space influences what happens here on Earth in a powerful way – much more so than anything we humans can come up with.

How all that relates to the current picture of doom and gloom? Well – my question is: if we know we’ll run out of resources at some point why we are not trying to get to the resources beyond what is available on Earth? And since when did we forget that for any sane human being fellow humans should come first, before whales and bats? Why instead of trying to kill off humans (or prevent them from being born) – which is what our leaders seem to be busy doing – we are not trying hard to make sure that a) everyone is fed and b) we can have a future in space?

It’s not a problem of technology or money. The technology is there – we have heavy lift rockets, most are just not being produced anymore (like the famous Russian Energia). We even have nuclear technology that could give even better lift and could propel us further than just the Low Earth Orbit, but it never was really used. In fact it seems that when it comes to space technology we are not advancing, not stagnating, but actually falling back.

The money is also there – just the bailouts for the “too big to fail” would have funded NASA for years. This is not mere rhetoric. NASA’s budget for FY 2008 is approximately $17 billion or 0.6% of US Federal budget. The bailouts did cost between $4 to $8.5 trillion according to different sources. That is between 235 and 500 years of NASA’s funding. Or 29 to 63 Apollo programs (cost of whole Apollo program is estimated at $136 billion of 2005 US dollars). And some say the total cost of bailouts etc. is $23 trillion – you can calculate yourself how many trips to Moon, Mars and elsewhere would that buy – even at NASA, known for its wastefulness and reckless spending on bureaucracy etc.

Those numbers show how mad this is, how we risk our collective future by massive misallocation of resources. Can you imagine how much technology and knowledge we could have obtained if just a fraction of those heaps of money wasted on Wall Street would have been allocated to space exploration? How many good, real jobs would have been created – jobs that actually create something, not “service” jobs that mainly mean people flipping burgers and waiting tables?

So something just doesn’t add up here. Either all our leaders just never look up into the sky at night and can’t use their brains for anything other than campaigning – or there is a barrier there we are not being told of. In any case instead of pursuing the only positive path we are being told the best thing we can do is planet-wide civilization suicide.

Which is why I’m sick when I see attempt by mass media and major corporations to create an illusion of grass-root support for the Copenhagen meeting and tax on breathing they want to impose there on the whole world (like this one). What’s really sickening is that this scam seems to work, that people do believe in this whole heap of lies they are being told without questioning them, without thinking. And without realizing there is a positive path – we just don’t follow it, we don’t even consider it.