Archive for the ‘Testing’ Category

What is the breakfast of champion agile development teams?

March 11, 2007

Feedback is the breakfast of champions, or so the saying goes. The more I peer into our methodology for getting more projects done faster and more successfully, the more I think that great early feedback is the key to its success. In software development methodologies, feedback is called testing.

The phrase “working client-valued software features” contains two types of test: that the software works and that it adds value to a client. Client wishes change over time as does the software landscape that the new business features are released into. The world is moving on as we develop software. The more frequently each test is repeated the more quicker the project can react to these differences. Unit tests and functional tests help us discover what is required to get the features working. End-to-end tests and frequent releases are required to discover what is required to get the features adding value. The later a mistake is found, the more work that needs to be re-done in order to correct it

Each type of test requires work, so there will be a trade-off between testing and development. A key component of agile is flipping the script so that we bake tests into the development lifecycle. This makes tests so easy to execute that it becomes inexcusable not to test. Feedback comes baked-in every day.

You can see this from the Top Ten Agile Best Practices. Of the ten, four are ways of introducing feedback earlier:

  • Code regression testing
  • Continuous integration
  • Test-driven development
  • Database regression testing

How do you get earlier feedback in your projects?

Technorati Tags: , , , , , ,


Mac unit testing: how to get started

February 24, 2007

I wanted to write a post on Mac unit testing. I have been making my first attempts to do some test driven development, and have found a bunch of great resources. Some are very specific to the environment I’m using (coding Objective C and Cocoa on Mac OS 10.4 with Xcode 2.3) and some are general unit testing and other testing resources. I’m curious how these resources apply to both older and new versions (e.g. do these all work the same on Leopard or Xcode 3.0?), but I’m also commenting on how some older resources work in the new versions. For instance, you’ll find a good number of testing resources that refer to a time when OCUnit was not bundled with Xcode.

Unit testing on CocoaDev is a great place to start. It notes that OCUnit is now shipping with Mac OS 10.4 and Xcode 2.1. From a number of online reviews UnitKit seemed like the main competitor for OCUnit (and definitely has a better name). However, with the bundling of OCUnit, its creator Duncan Davidson decided to extend it further only by sending in patches to OCUnit. So once, you’ve decided on OCUnit, how do you use it?

There are one or two good introductions to OCUnit and one great one. Note that if you are using Xcode 2.3, the steps to ensure OCUnit is working in your project are basically the same (add a new target, add in the framework and create a new group for the test cases), except that your test target will be type “Cocoa > Unit Test Bundle” not “”Cocoa > Test Framework“”. Similarly, when you are creating a test case, the file that you add will be of will be of type “Cocoa > Objective-C test case class” not “ ”Cocoa > Objective-C SenTestCase subclass“. A lot of the links on this page you’ll find on the references and further reading section of Sen:te’s OCUnit page. Little known fact: Sen:te comes from a Go-term roughly meaning getting one step ahead. (and more obviously, adding the initials of the term to do the word itself gives you a bunch of functions with ”teST“ in them.)

All of these intros take a (perfectly well coded and working) app that exists already and add some tests on top. What would have made these intros better is to have taken a real test driven development approach: write the tests first and the code then the app around them.

A great general introduction to xUnit best practices is xUnit Patterns. It seems pretty exhaustive and I’ve found a few hours of good browsing there. It’s pretty early days for me but I’m enjoying reading a section (or two … or three) and trying out the principles in Xcode and OCUnit.

Technorati Tags: , , , , ,

To unit test or not to unit test?

February 24, 2007

Unit testing is just obviously the right thing to do, right? Not everyone agrees. Since at least one who does not actually cuts an awful lot of great code, he is well worth listening to. After all, it’s all about the code. I don’t actually write programs for a living (any more), so hats off to those who do. The inimitable Wil Shipley writes that Unit Testing ‘teh suck’ (between you and me, I’m still not really sure what this means above ‘sucks bigtime’). In the ensuing flamewar which can be read in the comments, there are some excellent points, e.g.:

• Unit testing gives you massive ROI. The earlier you test in your development lifecycle, the less you build on buggy code and the less you rework you need to do.
• Unit tests are less necessary for genius coders. If you are truly brilliant and write flawless code, unit tests are an overhead. The reality is that most of us are not this, even when we think we are.
• For a single app that is written once from scratch, which does not re-use code and which will not be re-used, and which will never have later patches or versions, unit testing does not add much value. For everything else, every time you write code without a unit test, you are building up a debt. That debt will need to be paid off with regression testing. At some point in the not so far future, those payments are going to be crippling you so much that it’s almost impossible to code. NOTE: this does not mean you have to unit test it, but you will have it test it. Wi’s alternative is that you a) try your very best to break your app and b) get it into a beta program with real users.
• It’s easier to code unit tests for some types of app that it is others. GUI code can be very tough to add unit tests for, unless you use a model-view-controller model, or more generally, ensure that UI is a very light wrapper. The best test of the usability and usefulness of a GUI is some tough, try to crash it in any way you can, beta testing. There are GUI testers out there, but YMMV.
• Just doing unit testing is not enough, you have to do it right. Ken Anderson wrote “I find that a number of programmers who think following the unit test mantra is the ”Right Thing“ are mislead [sic] and write freakin dumb unit tests. It is not worthwhile to write a unit test for something that will be absolutely obvious when running the application (does this button actually call this method when I click it – kind of like, if I stick this pencil in my ear, will it hurt?). Unit tests should be used for more complex tests that are difficult to verify easily, like, does the result of this complex calculation match the known good quantities I’ve calculated outside the application environment.”

Bill Bumgarner wrote a response to Wil, which shows how, for infrastructure code, unit testing has gone from being fad through a hype into a day-in-day-out technique of many great teams. He explains how Apple’s Core Data could not and would not have been finished without a lot more time or a lot more coders than they had.

Technorati Tags: , ,

Reducing waste by introducing testing earlier

January 17, 2007

I was browsing Implementing Lean Software Development: From Concept to Cash by Mary Poppendieck & Tom Poppendieck, and found some great information about reducing waste. We have just started examining how we can improve our test processes as a way of introducing Test Driven Development. We have found two or three practices that remove most of our pain. Re-reading the book showed me that we have re-created some of the best practices that the Poppendiecks wrote about. Perhaps surprisingly, the practices are not about testing, but about development.
We have found that most elements of testing simply cannot wait until the end of the project or iteration. They need to be integrate much earlier. For instance, one legacy architecture is tightly coupled, not modular and with an amazing mountain of crippling ‘testing debt’, i.e. we have not built unit, functional or regression tests for the code. This means that more often that not, testing is the bottleneck around which we should plan, not development. In turn, this means that building a release plan around development estimates does not make sense: the critical path for our software lifecycle comes from testing. Therefore, we need our test manager involved in release planning.

Similarly, the earlier we can generate good tests of the software the faster we highlight and can fix bugs. The longer we leave these to fester throughout the development life-cycle the more damage they cause. Our ideal is making the user acceptance testing a verification rather than flushing out new bugs. We approximate RTY, the manufacturing measure of first time right by measuring how many features work without bugs in UAT. The goal is to drive this to 100%.

“According to Shigeo Shingo, there are two kinds of inspection: inspection after defects occur and inspection to prevent defects [See Shigeo Shingo, Study of ‘Toyota’ Production System, Productivity Press, 1981, Chapter 2.3]. If you really want quality, you don’t inspect after the fact, you control conditions so as not to allow defects in the first place. If this is not possible, then you inspect the product after each small step, so that defects are caught immediately after they occur. When a defect is found, you stop-the-line, find its cause, and fix it immediately.”

“The job of tests, and the people that develop and runs tests, is to prevent defects, not to find them. A quality assurance organization should champion processes that build quality into the code from the start rather than test quality in later. This is not to say that verification is unnecessary. Final verification is a good idea. It’s just that finding defects should be the exception, not the rule, during verification. If verification routinely triggers test-and-fix cycles, then the development process is defective.”

Technorati Tags: , , , , , ,