Zembling the best of the social web

I’ve been playing with Zembly and I love it. Zembly is a simple and social way to a-zemble (get it?) the best widgets and mash-ups. Zembly partners with other great services that I love (like Dapper) to enable you to extract information from your favourite websites and use them to build widgets, mash-ups and apps for the iPhone, Facebook and meebo amongst others. It’s quick and it’s easy and because Zembly comes with a whole lot of infrastructure baked-in, you don’t need to think about hosting, or scaling the architecture, because it is all in the cloud and taken care of.

Zembly is proving a breakthrough by really thinking through usability for developers: how do you make it really simple for developers to socially build social apps, and remove all the things that typically block them from doing that?

As you code your widgets (yes, you do need to hack a little to make the apps), you can clone other widgets that Zemblers have made, and tailor them to your own purposes. Zembly is part of Sun, and as you’d expect, there is some great scalable in-the-cloud infrastructure behind it. It has competition in the likes of Sproutbuilder (not to be confused with Sproutcore, which Apple used as part of the MobileMe rollout).

You can find an interesting interview with Zembly’s CTO, Todd Fast. He mentions the exponentially growing number of Facebook apps, going from 24,000 to 30,000 and above. He says that Zembly is looking at building niche applications for interesting special-interest groups — long-tail applications — by bringing the threshold for building these applications down. He also says that most social platforms, even the open ones, are very hetereogenous and fragmented. Zembly’s answer is to try to build small re-usable pieces, that you can play, standing on the shoulders of giants. He also explains why he feels that most developers don’t want to think too hard about how to scale their applications, whether it be in the form of databases with millions of rows or allowing tens of thousands of users or the analytics that measure the success of the applications. The latter functionality allows developers to measure social statistics: what kinds of users are installing the application and how, alongside measures of how virally the application is being installed.

All pretty cool and a lot of fun to play with. Zembly is currently in private beta, but if you put your name down, I think you’ll find you don’t have to wait long.

Small is beautiful

I have written a good few blog posts that have died, unfinished and unpublished because the nebulous idea I was trying to express was just too complex. Small is beautiful in blogging.

One of the key characteristics of a feature which we used in rolling out our agile methodology was that it had to be small. This could be defined by technology or the business:
Either by the development estimate — a few story points of under a couple of weeks of pure development
Or by a way of describing the business feature — for instance a one sentence description in what the user might see differently
Business users quickly get the idea, and technology loves it.

This has a number of great advantages. First up, you can fit a few features in an iteration and still keep iterations short. Next, if some features do take longer than expected or your velocity system has not kicked in yet it is easy to drop a feature or two from an iteration, keep to your plan and still deliver business value. Lastly and perhaps most importantly, small features are exponentially easier to define requirements for, develop and test for. As you add a little more complexity to a feature, the time it takes to go through the development lifecycle takes exponentially longer sine each piece can interact with the rest of the feature.

This principle — which is known elsewhere as “Keep It Simple” — can applied to a variety of other domains with great success. Simple works. It is also a great definition of agility. Simple means small and simple means only doing what you need and no more. Without simplicity, testing is hard and it’s hard to deliver the constant stream of business value needed to build the groundrock of an agile rollout: trust.

Understanding before documentation

One of the most important principles I use in all the agile and lean software development or cross functional teams I run is ‘understanding first’. This means that teams must first seek to understand and only then write that up. My default form of this is to take two very different groups — perhaps IT and the business or product management and producers or finance and a budget owner — and get them to each explain their viewpoint to the other. They talk until the other group can explain their viewpoint to their own satisfaction. Then they sit together and write it up.

The longer that the groups work together the more rapport that they build. Talking is many times quicker than writing and this is many times quicker again than waiting for an email response. Once understanding is reached the last crinkles can be ironed out by writing together.

Often the business will ask IT to do the write up so that they can review at their leisure. This is a mistake. It confuses the time spent as support with the total time needed. A little feedback in the first write up can be the difference that makes the difference so that only one iteration is needed.

Large releases mean less useful features in production

I remember at an old employer that we would parcel a number of features into large releases. At the time we thought it was more efficient to do this. We did not have a modular architecture, so everything seemed to be interconnected (often in illogical ways). For this reason, we needed to do a huge amount of analysis to check that we weren’t going to wreck something in the release. Similarly, the regression testing was huge. This had a nasty impact on our lead times, i.e. it took a long time before we could turn an idea into software working in production.

What we did not realise though was that these large releases hit us in yet another place, causing more pain and misery. The long list of features that was waiting on other features all had some kind of documentation, be it a line in an Excel sheet, a fully dressed change request document, or more analysis documentation. This is sometimes called software ‘work in progress’ or WIP. This derives from the name given to inventory in a factory that is neither raw materials nor finished product, but something in between. All work in progress perishes over time. Simply put, it gets out of date. Either business requirements change, or technical systems change, or both. So, someone needs to invest time to get the WIP uptodate. This time is time that is not used to get software working in production.

So, even though we thought these large releases were helping us to increase efficiencies, they were actually slowing us down. Per business-valued working feature delivered to production, we were investing more time than we would have done with small releases.

Large releases can be tricky to resolve, since they often have two or three root causes. Firstly, refactoring to get towards a modular architecture can help. Secondly, regular builds — i.e. more frequent than once every six weeks — will push you in the right way. This is often supported by regular regression tests and some simple configuration management best practices. Lastly, learning to define features as small increments that add value to clients is also vital to reducing release sizes.

In order to achieve this triumvirate of best practices, there is often a tendency to reduce value-added work, e.g. focus on refactoring to the exclusion of client-valued features. Avoid this like the plague — it pushes you into a vicious circle, where the business trusts IT less and less, and as a result, IT trusts the business less and less. Trust is the platform on which agile change is built.

Technorati Tags: agile, lean

Dealing with unexpectancies

Nuno writes that “Why shouldn’t we try finding out find a model to deal with unexpectancies instead?
The thing is that if you by some cosmic accident find too much unexpectancies along the way your iteration plan will become a baby monster.”

Well, we do assume change will happen and most times it does. We also try to build some slack into each iteration, so that things can go wrong (because they always do) — that’s what velocity is for. But in the end: you’re right. There is that worst case scenario where everything falls apart and all the high-level estimates are wrong and we have to update all the planning just like it was a classicly managed project — i.e. update the project plan and re-do all the resource planning. From my experience this worse-case rarely actually happens.

There are some underlying assumptions:

• Features are independent, negotiable, valuable to users, estimable, small and testable
• We can easily move features from one iteration to another
• We can use the same resources for each iteration

We know that in reality these assumptions are either wrong and at best partially true. However, we can turn that around and make that a roadmap for our business and IT, e.g.

• Spread the knowledge between BA and developer resources so that it becomes easier to use the same resources for a number of different iterations (hitting different systems)
• Make the technical architecture more modular (there are some pretty standard ways of approaching this for legacy systems)
• Teach the business how to take thin enough client-valued strips of features across systems so that they are small enough for a few to fit into an iteration

Technorati Tags: agile, development, lean, software

Six Month Review

Mr Anderson has some impressive results in his Six Month Review:

• An average of a release to production every 9.5 business days

• Zero escaped defects this year

I am getting on to being about six months into my new role and I would love to be showing results like this. I guess we are on different journeys…

He also points out that he does not have that many good old fashioned agile processes:

• No test driven development
• No pair programming
• No continuous integration
• No burn down charts

His conclusion is spot on: there are many way of being agile. You need to take a close look at what is not working where you work and find ways of improving. But one size does not fit all. The process of discovering what works and what is stopping you can be consistent. David suggests four ‘golden rules’: focus on quality, reduce work-in-progress, balance demand against capacity, and prioritize. These are great starting places to start.

At World Directories, I have been starting from a philosophy of Thinking Processes: identify the systems of cause and effect that determine how the organisation works (here: how the Online & Search business works with IT, Operations and our partners); identify the few variables that determine the performance and work out how to improve things in a way. This has led to a number of agile best practices that we are now introducing across the different countries, including: iterative time-boxed development, simplifying the metrics we use, effective prioritisation and focusing on quality.

Technorati Tags: agile, development, project management, software, tdd, testing, toc

Less formal, more rigorous

In conversation with our head of audit, I was explaining how our agile methodology is actually a lot more rigorous than old-school project management. We are also less formal. The two are hard to distinguish at first. Let me give a couple of examples.

We are more rigorous. If we start to move our timeboxes around, our whole paradigm falls apart. If we unexpectedly discover that a feature is going to take too long to test, we push it to the next iteration instead of moving the iteration end date out. If we don’t do this, we need to re-write the release plan, re-estimate tasks, re-do resource plans and (where I work at least) replan priorities for other projects, as well as generating the same set of documentation for these other projects. This is all waste work: work that does not bring us closer to client-valued features deployed and working in production. It’s also typically the thin end of the wedge: pretty soon our teams spend more time on documentation that on getting value-adding software working flawlessly in production.

We are less formal because all documents that we use are supportive not directive. I don’t demand people use a certain Excel template for a release plan. Thinking about certain attributes for each feature and for each iteration add value in our experience, and in teams that are starting up I strongly recommend (and yes, even enforce) this. But I don’t really care how our teams record this information. I have been down the path of only checking that everyone is using the same Word template or Excel sheet and mistaking that for real value — and I won’t be going down that path again.

Technorati Tags: agile, development, lean, processimprovement, project management, software

What is the breakfast of champion agile development teams?

Feedback is the breakfast of champions, or so the saying goes. The more I peer into our methodology for getting more projects done faster and more successfully, the more I think that great early feedback is the key to its success. In software development methodologies, feedback is called testing.

The phrase “working client-valued software features” contains two types of test: that the software works and that it adds value to a client. Client wishes change over time as does the software landscape that the new business features are released into. The world is moving on as we develop software. The more frequently each test is repeated the more quicker the project can react to these differences. Unit tests and functional tests help us discover what is required to get the features working. End-to-end tests and frequent releases are required to discover what is required to get the features adding value. The later a mistake is found, the more work that needs to be re-done in order to correct it

Each type of test requires work, so there will be a trade-off between testing and development. A key component of agile is flipping the script so that we bake tests into the development lifecycle. This makes tests so easy to execute that it becomes inexcusable not to test. Feedback comes baked-in every day.

You can see this from the Top Ten Agile Best Practices. Of the ten, four are ways of introducing feedback earlier:

• Code regression testing
• Continuous integration
• Test-driven development
• Database regression testing

How do you get earlier feedback in your projects?

Technorati Tags: agile, development, feedback, project management, software, tdd, testing

Compassion as a core programming virtue

James Dobson says “Compassion is a central theme of the work my company does.” Maybe he’s right and we should add compassion to those three fundamental virtues of a programmer. I’m going to quote the full text from the Netropolis, in turn quoting Randal Schwarz:

• “LAZINESS: The quality that makes you go to great effort to reduce overall energy expenditure. It makes you write labor-saving programs that other people will find useful, and document what you wrote so you don’t have to answer so many questions about it. Hence, the first great virtue of a programmer.
• IMPATIENCE: The anger you feel when the computer is being lazy. This makes you write programs that don’t just react to your needs, but actually anticipate them. Or at least that pretend to. Hence, the second great virtue of a programmer.
• HUBRIS: Excessive pride, the sort of thing Zeus zaps you for. Also the quality that makes you write (and maintain) programs that other people won’t want to say bad things about. Hence, the third great virtue of a programmer.”

In this context we could define:

• COMPASSION: What you feel when you want your team to work better together, and not let your fellow software developers suffer the pain of poor quality software development

It’s the Emotional Intelligence in the programmer virtues. When you start thinking of the way you work with others to deliver software, the other virtues kick in. Given that we are lazy, we want to deliver software features as quickly as possible with the minimum amount of work. We are impatient to deliver great software and move on to the next business feature without all that re-work in between. We have hubris about how we work together as a team: not just our software development process, or the tools that enable it, but our culture too.

Technorati Tags: agile, development, eq, processimprovement, compassion, software

Theory and practice

James Dobson has a good post on compassion in agile transformations. Here was my comment.

Maybe it’s the humanist in me (and I mean w-a-y down inside and hard to find in me) that makes me agree that much of religion is not about high-faluting theory, but simple practice. Preaching agile to the masses can be fun, but we often forget that what people are doing right now works for them:

• Waterfall works for the management dogmas in place right now (e.g. cost accounting)
• Process-oriented project management (e.g. Prince 2) is a lot more effective than doing no management.
• Focusing on local maxima like developer utilisation, or lines-of-code (excuse me while I clean my mouth out with soap) works for cost accounting driven management
• Writing detailed plans 3 or 6 or 12 months in advance can be a lot more useful than not thinking about the future at all

I think to change a company you need to both change some of these cultural axioms and some of the behaviours, such as moving from measuring cost to measuring throughput and cost. When you do that, you need to give everyone in the organisation the props to do the job and to continually help them in doing it. For example, showing them how to simplify down to stories, story points and velocity (software development throughput), and showing them how much more enjoyable coding is when we do that.

It’s hard graft and it’s satisfying graft.

Technorati Tags: agile, development, project management, processimprovement, software