The Best Estimating Technique

When you think of estimating and planning techniques, what method do you think of first?  Do you use a parametric technique, Monte Carlo simulation, or something else?  I was over at the Implementing Scrum website and found an awesome post, asking this same question.  If you think people get crazy whether or not you should utilize an Agile approach or a traditional project management approach on a project, wait until you talk about how to estimate work.  In the traditional project management world, I've seen people use SLOC and PERT to arrive at an estimate.  In the Agile world, I've used story points and have seen others use gummy bears and t-shirt sizes. In the end, I really don't care what estimation technique is used and I'm pretty sure the customer won't either.  All anyone should be worried about is if the estimates are accurate and if you get the work done.  Am I right or am I wrong?  I would love to read your input.

Thank you to Mike Vizdos for letting me use his cartoon and for the inspiration for this post.

Expert Judgment and Passing the Sniff Test

Looking for two very informative posts on estimating?  Check out one by Josh Nankivel of pmStudent and Glen Alleman of Herding Cats.  Both are discussing estimating techniques that work for them. I wanted to take a moment to add my two cents. Though I certainly believe estimating should be more science than art, I look at estimates from a different perspective. As a disclosure, I'm not the one doing the estimating on this project, therefore I'm not going to say I agree or disagree with any one technique.  Depending on your situation, one estimating technique may provide more accurate results than the other.

What I would like to add, from my perspective, is the need for expert judgment. If you are an expert in a given estimating technique and it gives you the results you and your customer(s) need, does that not validate it as an acceptable estimating choice?

If the estimating technique does not produce the desired results, wouldn't it fail the metaphorical sniff test?

Recently, I questioned a vendor's estimate based on a different technique.  I used a parametric estimate to see if the vendor's estimate would pass or fail my sniff test.

What exactly is a parametric estimate?

An estimating technique that uses a statistical relationship between historical data and other variables to calculate an estimate for activity parameters, such as scope, cost, budget, and duration.  Source: PMBOK Page 439

So, why did the vendor's estimate not pass my sniff test?  As part of a standard estimating practice, software vendors should include time for fixing bugs. Upon review of a recent status report, I noticed the vendor reporting half as many bugs were discovered in a current build than had been estimated. When asked about this, the vendor was very excited to confirm that they indeed found half as many defects in the code they originally estimated and predicted a cost savings of several hundred thousand dollars to the project.  Going into the current build, I knew what the standard deviation was and considered the possible variance.  This fell way below that.

So, why were they discovering so few bugs?  At first glance, I would predict two possible reasons.  [1] Quality through development improved.  [2] Quality through testing worsened.  Either way, you get the same initial result of fewer defects identified.

We'll know the true answer once initial user acceptance testing begins.  If there were no baselines to compare the actuals to, I might not have given it a second thought.

Graphic source via Flickr: pump