The Art of Unit Testing 98
FrazzledDad writes "'We let the tests we wrote do more harm than good.' That snippet from the preface of Roy Osherove's The Art of Unit Testing with Examples in .NET (AOUT hereafter) is the wrap up of a frank description of a failed project Osherove was part of. The goal of AOUT is teaching you great approaches to unit testing so you won't run into similar failures on your own projects." Keep reading for the rest of FrazzledDad's review.
AOUT is a well-written, concise book walking readers through many different aspects of unit testing. Osherove's book has something for all readers, regardless of their experience with unit testing. While the book's primary focus is .NET, the concepts apply to many different platforms, and Osherove also covers a few Java tools as well.
| The Art of Unit Testing with Examples in .NET | |
| author | Roy Osherove |
| pages | 296 |
| publisher | Manning |
| rating | 9/10 |
| reviewer | FrazzledDad |
| ISBN | 1933988274 |
| summary | Soup-to-nuts unit testing with examples in .NET |
Osherove has a long history of advocating testing in the .NET space. He's blogged about it extensively, speaks at many international conferences, and leads a large number of Agile and testing classes. He's also the chief architect at TypeMock, an isolation framework that's a tool you may make use of in your testing efforts – and he's very up front about his involvement with that tool when discussing isolation techniques in the book. He does a very good job of not pushing his specific tool and also covers several others, leaving me feeling there wasn't any bias toward his product whatsoever.
AOUT does a number of different things really, really well. First off, it focuses solely on unit testing. Early on Osherove lays out the differences between unit and integration tests, but he quickly moves past that and stays with unit tests the rest of the book. Secondly, Osherove avoids pushing any particular methodology (Test Driven Development, Behavior Driven Development, etc.) and just stays on critical concepts around unit testing.
I particularly appreciated that latter point. While I'm a proponent of *DD, it was nice to read through the book without having to filter out any particular dogma biases. I think that mindset makes this book much more approachable and useful to a broader audience – dive in to unit testing and learn the fundamentals before moving on to the next step.
I also enjoyed that Osherove carries one example project through the entire book. He takes readers through a journey as he builds a log analyzer and uses that application to drive discussion of specific testing techniques. There are other examples used in the book, but they're all specific to certain situations; the brunt of his discussion remains on the one project which helps keep readers focused in the concepts Osherove's laying out.
The book's first two chapters are the obligatory introduction to unit testing frameworks and concepts. Osherove quickly moves through discussions of "good" unit tests, offers up a few paragraphs on TDD, and lays out a few bits around unit test frameworks in general. After that he's straight in to his "Core Techniques" section where he discusses stubs, mocks, and isolation frameworks. The third part, "The Test Code" covers hierarchies and pillars of good testing. The book finishes with "Design and Process" which hits on getting testing solidly integrated into your organization, plus has a great section on trying to deal with testing legacy systems. There are a couple handy appendices covering design issues and tooling.
Osherove uses his "Core Techniques" section to clearly lay out the differences between stubs and mocks, plus he covers using isolation frameworks such as Rhino.Mocks or TypeMock to assist with implementing these concepts. I enjoyed reading this section because too many folks confuse the concepts of stubbing and mocking. They're not interchangeable, and Osherove does a great job emphasizing where you should use stubs and mocks to deal with dependencies and interactions, respectively.
The walkthrough of splitting out a dependency and using a stub is a perfect example of why this book's so valuable: Osherove clearly steps through pulling the dependency out to an interface, then shows you different methods of using a stub for testing via injection by constructors, properties, or method parameters. He's also very clear about the drawbacks of each approach, something I find critical in any design-related discussion – let me know what things might cause me grief later on!
While the discussion on mocking, stubbing, and isolation was informative and well-written, I got the most out of chapters 6 ("Test hierarchies and organization") and 7 ("The pillars of good tests"). The hierarchy discussion in particular caused me to re-think how I've been organizing an evolving suite of Selenium-based UI tests. I was already making use of DRY and refactoring out common functionality into factory and helper methods; however, Osherove's discussion led to me re-evaluating the overall structure, resulting in some careful use of base class and inheritance. His concrete examples of building out a usable test API for your environment also changed how I was handling namespaces and general naming.
If you're in an organization that's new to testing, or if you're trying to deal with getting testing around legacy software, then the last two chapters of the book are must-read sections. Changing cultures inside organizations is never easy, and Osherove shows a number of different tools you can use when trying to drive the adoption of testing in your organizations. My own experience has shown you'll need to use combinations of many of these including finding champions, getting management buy off, and most importantly learning how to deal with the folks who become roadblocks.
The Art of Unit Testing does a lot of things really well. I didn't feel the book did anything poorly, and I happily include it in my list of top software engineering/craftsmanship books I've read. All software developers, regardless of their experience with unit testing, stand to learn something from it.
You can purchase The Art of Unit Testing with Examples in .NET from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
Tiring to read (Score:5, Interesting)
I read this book recently and found it tiring. Much of it reads like a blog, and like many books, the author randomly switches stances. He'll refer to the reader as "the reader", "you", "we", and in the third person. This is the kind of book where it's hard to keep a clear idea of what the author is talking about, because he doesn't have a clear idea of what he's trying to communicate.
When I think of tiring books like this, I cannot avoid always remembering Steve McConnel's Code Complete (first edition; I haven't looked at the second edition yet). Reading that book is like having your autonomy assaulted, because the author constantly tries to get you to accept the things he's claiming, via whatever means necessary, rather than presenting them along with rational arguments, and letting you decide when to apply them. I'm not saying Osherove's book is that bad, just that it has that same unenjoyable aspect that makes it a chore to read and get useful information from.
I recently also read Kent Beck's Test-Driven Development and highly recommend it, if you simply want to learn about unit testing and test-driven development. It's concise and enjoyable to read. Unfortunately it doesn't cover as many details, and I don't have any good alternatives to books like Osherove's (and I've read many at my local large university library).
Read it; Loved it. (Score:3, Interesting)
Related: Working Effectively with Legacy Code (Score:3, Interesting)
I've read several unit testing books recently, and another I found somewhat useful is Michael Feathers' Working Effectively with Legacy Code [slashdot.org]. It has all sorts of techniques for testing legacy code, i.e. code that wasn't designed for testability, and which you want to make as few modifications to. So he gets into techniques like putting a local header file to replace the normal one for some class used by the code, so that you can write a replacement class (mock) that behaves in a way that better exercises the code. Unfortunately Feathers' book is also somewhat tiring to read, due to a verbose writing style and rough editing, but I don't know anything better.
Re:It's not art, it's basic engineering (Score:2, Interesting)
Singletons are pretty easy to test as long as you don't use the *antipattern* of the class that enforces its own, uh, singletonicity. If you have .getInstance() methods, you have that antipattern. Yes, it's in the GoF book but frankly the GoF is just wrong on that point. It's a lifecycle pattern, and lifecycles like that should be taken care of by the context, like a class factory or a DI container. If you have a DI container, testing a singleton is an absolute snap and in fact *easier* than non-singletons.
A decent mock framework can handle even the bad old singleton pattern though. And part and parcel of a mock framework is their ability to "rewind".
Unit testing is not a silver bullet (Score:3, Interesting)
I work on distributed real-time software (financial industry) and can tell you that unit tests for components I write are either
1. trivial to write, therefore useless
2. impossible to write, therefore useless
I find full logging and reliable time synchronization both easier to implement and more useful in tracking bugs and / or design errors in environment I deal with than unit testing.
tl;dr - check return values, catch exceptions and dump them in your logs (and use state machines so you know where exactly you were, and so on...)
Re:Error coding... (Score:3, Interesting)
Error handing and reporting is very complex. This is mostly due to the complexities involved with different parts of a system interacting with each other.
Windows GetErrorInfo call will return NULL on the second call, despite no other COM calls made in-between. This took a while to understand what was happening.
Do you check that an IErrorInfo object is valid for the method you just called by using ISupportsErrorInfo? Do you check to see if this is a C#/.NET System.Exception (_Exception) and then use ToString to get back the nice stack trace?
Does the Windows API call return a HRESULT, an NT error code, a registry API error code, a BOOL with a corresponding GetLastError call to get the details or something else? Are you checking errno on all C API calls? Or the appropriate error checking call for the API you are working with? Do you realise that some Windows APIs have different return code behaviour on Win9x and NT+ (e.g. the GDI calls)?
Do you guard all your COM calls written in C++ to ensure that a C++ exception does not leak outside the COM boundary? Do you report said exception as a HRESULT and IErrorInfo that lets you track down the problem?
Do you ensure that an exception is not thrown from outside a destructor? A thread function? A Windows/GTK/Qt/... event handler? Across language boundaries?
How do you present an error to the user? Are you showing the exception message (e.g. displaying E_UNEXPECTED as "Catastrophic failure")?
It is not always possible to write perfect code. Developers will forget to check for access permissions to a file before writing to it, or just ignore any errors (did you know that std::ofstream will not report an error if the file it is writing to is read-only (at least with the Microsoft C++ implementation and Windows, without any fancy std::ios flags?)).
Have you ever dealt with infinite recursions that involve 3 or more functions? Do your COM/DBus/... calls check for/handle network failures? Do you report these in a friendly way to the user? Do you try to recover/reconnect in this case?
Knowing exactly what a system call does is also impossible, unless you have access to the source code for that particular configuration. The MSDN documentation is not reliable for Windows APIs, as it leaves a lot of the important stuff out. The POSIX documentation only covers the important/most common error cases.
Re:Does he back up anything he says (Score:5, Interesting)
Does your speaker have anything concrete with citations to back his assertions up, or is he happily dismissing one of the few genuine advances in software engineering in the last decade?
we found that the code developed using a test-driven development practice showed, during functional verification and regression tests, approximately 40% fewer defects than a baseline prior product developed in a more traditional fashion. The productivity of the team was not impacted by the additional focus on producing automated test cases. This test suite will aid in future enhancements and maintenance of this code.
-- http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.129.7992&rep=rep1&type=pdf [psu.edu]
A Spring 2003 experiment examines the claims that test-driven development or test-first programming improves software quality and programmer confidence. The results indicate support for these claims
-- http://portal.acm.org/citation.cfm?id=949421 [acm.org]
Experimental results, subject to external validity concerns, tend to indicate that TDD programmers produce higher quality code because they passed 18% more functional black-box test cases.
-- http://qwer.org/tdd-study [qwer.org]
We observed a significant increase in quality of the code (greater than two times) for projects developed using TDD compared to similar projects developed in the same organization in a non-TDD fashion.
-- http://portal.acm.org/citation.cfm?id=1159733.1159787 [acm.org]
My apologies for the rough and ready citations, I only picked the ones I could find on the first fucking page of Google search results.
Re:Does he back up anything he says (Score:3, Interesting)
I have never seen any scientific studies on it, but I use Unit Testing as a tool to help me code and debug better and it works a LOT better than anything I tried prior to that. And when I break some of my old code, I know exactly what's breaking with just a glance.
Also, I have occasionally be charged with massive changes to an existing system, and Unit Testing is the only thing I know of that lets me guarantee the code functions exactly the same before and after for existing uses.
tl;dr - I don't need a scientific study to tell me a tool is working well for me.
As is Beck's book... (Score:1, Interesting)
Interesting that you should mention Kent Beck's book, as I too have read it recently and found it to be the shittiest pile of steaming turd at I've ever seen put into book form. It was *SO* slow going, so condescending, and it was so sorely lacking in the way of cohesive rational arguments that if I hadn't been sold TDD through other means, I would have abandoned the concept altogether. It might be okay as an introduction to a complete novice programmer, but if you've had any experience in the industry at all, I'd recommend avoiding it unless you either want something to put you to sleep, or you're a sucker for punishment.