Outdated Success Criteria

Outdated Success Criteria

I know this is going to probably get me some “hate” comments.  It seems like if I write about anything but a zombie, that’s what happens. But I do like to write about topics that make people stop and think. Think of this post a bridge between a historical project management and futuristic project management.  Let’s think about success in both an objective and subjective way.

I’m seeing more and more topics about the measurement of success.  Geoff Mattie just wrote a post over at the PMI Voices site, titled Can Agile Conquer the Physics of the Triple Constraint?

Geoff refers to Triple Constraint and states

The “iron triangle” as some refer to it, defines three pillars: cost, scope and time. It asserts that you have to prioritize the three with an understanding that trying to have all of them at the same time compromises quality.

I applaud Geoff in his zealousness and hope this works for him and hit customers.  Being his blog post is on the PMI website, I want to point out the the iron triangle is not in the PMBOK.  Rather, on page 6, it states

Managing a project typically includes… balancing the competing project constraints including, but not limited to Scope, Quality, Schedule, Budget, Resources, and Risk.

I remember a few years back, when taking the PMP exam, I had a question about typical project constraints.  The answer was not limited to 3 or even 4 “pillars”.  So, where am I going with this?

graphic by Jessica Clarke

I’m curious why people continue to measure the success of a project, merely on the basis of an iron triangle.  I think this concept is outdated and perhaps created by a project manager to help an executive understand project management at a 100,000 foot view.  I am also curious why many continue to use the Chaos report, (which leverages triple constraint) as the de facto report of industry success or failure.  I am not debating that it has historical significance.  But, I am questioning if it should be the way of measuring project success.

Jeff Sutherland has a blog post about the happiness metric. In his post, he mentions Tony Hsieh of Zappos.  I recently read the book Delivering Happiness by the Zappos CEO.  Again, what’s my point?  Perhaps the Chaos report should introduce happiness or customer satisfaction at part of its success criteria.

Too subjective you think?  I think not!

I recently saw a presentation by Sanjiv Augustine as part of the VersionOne AgileLive Webinar Series

One of the concepts presented in Sanjiv’s presentation was a NPS (Net Promoter Score) metric.  Think of it as a customer satisfaction or “happiness” metric.

NPS is based on the fundamental perspective that every company’s customers can be divided into three categories: Detractors, Passives, and Promoters. By asking one simple question — How likely are you to recommend [Company X] to a colleague or friend? — you can track these groups and get a clear measure of company performance through its customers’ eyes.

So, what is the Zappos NPS?  In a YouTube video of Tony Hsieh at the NPS Conference  (1-26-09), Tony said Zappos offered random email surveys that resulted in an 83% NPS and phone surveys resulted in a 90% NPS.  Though they lose money on some of their customers, they are an overwhelming success.

Do you believe the Standish Group Chaos Report should include NPS to define success? Are the original classifications outdated?


14 Replies to “Outdated Success Criteria”

  1. Hi Derek – just saw this post. As you may know, we use NP here at Steelray and are getting better and better results! But we don’t use it as project metric – merely as a happiness-of-customer-barometer. So, adding NP as an “official” metric of a project – I’ve just never thought about it. As I did, I kept thinking of reasons it shouldn’t be. One is that if you delivered something to your client that wasn’t included in the project requirements, i.e. radically changing the scope simply to meet “featuritis” from the customer – then this metric isn’t really valid. Let’s say after doing so you get a very high NP score, so the customer is happy. But is that what the project was really all about? If the client is happy but you actually lost money on the project – which happens so often – then all this metric tells you is that you did not handle stakeholders and requirements like you should have! But let’s say you handled the project well, and that you accomplished the initial objective – yes, this score is a good metric.

    1. Laura, I think you make a very valid point. I agree, I don’t think the NP is the end-all be-all of metrics. All I’m proposing is that customer satisfaction should not be ignored. As I stated in my response to Jeremy, based on the criteria defined in the Chaos report, a project can be listed as a success or failure and ignore customer satisfaction or happiness. My proposal is it should at least be considered.

      I’m not expecting everyone to agree with me. But I’m very excited about the conversation the post has generated.

      1. I hear ya. But customer satisfaction is really important, and an NP score is a good metric for that. I’m just saying customer satisfaction can mean a lot of things, and I urge project managers to accomplish the objectives, not engage in featuritis! good post though Derek – good point.

  2. Jeremy, I certainly don’t believe quality should be thrown to the dogs. If a vendor delivers poor quality code, and they still meet the schedule, budget, and scope, the Chaos report would state the project was a success. I don’t see how that is acceptable, unless the customer believes poor quality code is acceptable. At the end of the day, we’re here to satisfy the customer. If the customer’s highest priority is quality and the vendor delivers on that (making them happy), at the expense of some scope, I don’t think it should be counted against them. If the customer is happy, I see it as a successful project.

    1. I don’t believe a lot of customers are aware of, or qualified to understand Technical Debt or what they are agreeing too. That Agile allows or even encourages teams to skirt around it with the shield/caveat of “oh we did what you asked” isn’t taking the kind of responsibility I like to see.

      Maybe it’s a personal thing, but I don’t like a process that has no concept of quality or big picture and focuses on mainly speed and scope, though I’m a fan of them as well, but not exclusively.

      1. If I understand what your saying, I want to confirm that I don’t believe we should be out to JUST please the customer(s). But we shouldn’t forget who we’re working for. Several years ago, I tried this “yes,yes,yes” approach. I wound up with a customer who would ask me to make 4 right turns, rather than continue going straight. I learned my lesson.

      2. +1. There is undoubtedly a tension between the ‘managing uncertainty’ underpinning of agile and the danger of building up technical debt by assuming that enterpise level quality can be created by successive tactical change.

  3. Sridhar, I like where you’re going with this. What I’m getting from your comment, if I read it correctly, is use NPS but balance it with traditional success parameters. I’m good with that. I’m not proposing we lose our minds trying to satisfy the customer. I just don’t want to omit them when deciding if the project is a success or failure.

    Today, I reviewed contract award fee comments from my client (customer) about the vendor. Though the vendor is saying they are on budget and on time, the customer is very unhappy. Long story short, if NPS was used, it would be very low. Ask the vendor if the project is a success and they will say yes. Ask the customer and they will say no. At this time, NPS is not being leveraged on the program. I think it should at least be considered.

    1. I agree with you completely. However, I think in your case, the client and the vendor are not agreeing on the traditional parameters itself and hence the NPS would definitely be low. One of the reasons could be that the project success parameters were changed during execution due to some reason (schedule was extended, too many change requests leading to increased cost, bugs caught in acceptance tests exceeded initial targets etc). The vendor might have re-negotiated these parameters and the client had no option but to accept them in the interest of the project.

      Such a situation will result in process success parameters being met, but NPS being low. In short, the success parameters were not negotiated to the client satisfaction -> the parameters are not correct.

      All I am saying is that NPS reflects the overall “feeling” of the client rather than measurable parameters (In fact, one NPS score was less because the project team from Sweden, including the PM, could not speak fluent English, but that’s an extreme example). One way to mitigate is to have a provision for additional qualitative inputs in case NPS is scored low by the client – then the vendor will have more information on why it could not satisfy the client. Otherwise they will be left blaming each other.

      What do you think? Are there any other scenarios that could play out differently?

      1. I like your proposal. “Have a provision for additional qualitative inputs in case NPS is scored low by the client”. Perhaps the NPS could be leveraged more like a canary in a coal mine.

        The client and vendor certainly do have issues that need to be worked out. Both have to be willing participants though. I can tell you, yes, the project success parameters were definitely changed during execution phase.

  4. Yes, i think the triangle should mature into an a hexagon with quality,customer satisfactions /NPS and the Project ROI (Which by the way is different from cost) being included.

  5. Hello Derek,

    Thanks for the interesting post and interesting POV.
    All my project manager’s life I have been thinking that “Iron triangle” is usable only for planning a Release (project) and not for measuring project’s success 🙂

    We definitely can’t measure success only by meeting budget and scope (or time). Often, projects with “success” on such criteria are not successful as products, i.e. miss the market, users and customers (said by Poppendieck and others).

    For success measurement NPS, Kano analysis and other techniques should be used.

    1. Tim, thanks for adding your perspective. I sometimes feel the Iron Triangle was written on the back of a napkin at a cocktail party and at some point became dogma for some. I still believe the Standish Group should expand the criteria for measuring success. Then again, am I giving them too big of a megaphone?

      I haven’t used Kano analysis before. I learn something new every day. Thanks!

Leave a Reply

Your email address will not be published. Required fields are marked *