What the Hell Does 7 Mean?
I have been using GetSatisfaction for a while now… love the service, if you haven’t signed up I strongly advise you join up and get in on the conversation. That being said, there are some elements of their UI that are confusing and I recently contributed to a suggestion to improve something that I personally had a hard time finding/using. The staff were quick to respond and even asked me for feedback on a change to address my point. I was very impressed.
One thing I still find confusing while using the site is the use of the 10 point Net Promoter Score (NPS) system.
Wikipedia: The Net Promoter score is a management tool that can be used to gauge the loyalty of a firm’s customer relationships. It serves as an alternative to traditional customer satisfaction research.
Now this gauge is fairly commonly used to by companies to determine how happy/loyal their customers are (How likely is it that you would recommend us to a friend or colleague?), but taking a step back and looking at it from a intuitive UI experience it is confusing both for the person voting as well as the person analyzing the data. Excluding the numbers 0, 5 and 10 … the interpretation for all the rest of the numbers are completely up in the air. I can’t actually think of 10 values for things for a poll like this, so I surely don’t know how I could distinguish voting for an item.
Breaking it Down
So to prove a point, lets break it down:
- 0 = Not a chance I would recommend this to anyone, I hate it
- 1 = I wouldn’t recommend this product at all. It is garbage. i would only use it if I HAD TO due to price or lack of competition.
- 2 = This product performs bare bone functions and is an unenjoyable experience
- 3 = This product is tough to use and unituitive but does perform.
- 4 = This product works ok, although if there are other products/services that perform this function at this price point, I would recommend taking a look at them.
- 5 = I am completely indifferent to this product/service
- 6 = This product is satisfactory, it does what it says but could definately be improved.
- 7 = This product is good, couple things I would change, but overall does what is says.
- 8 = This product is really good, worth the money
- 9 = This product is great, I would advise everyone getting it.
- 10 = I love this product, I tell everyone about it, I am getting it for people for gifts.
Now as you can see I defined each of the 10 points for what they might mean to me… if you read through that, you can see it is quite an effort to evaluate and choose. Bundle that with the fact that I am sure each indivdual would define those 10 things slightly differently makes it not only a burden to the user to answer, but from a reporting standpoint sort of worthless (if each number really means everything).
Intuitive Questioning
So lets address the problem and come up with a solution. A company wants to find out what a user thinks of their product. We want this to take, at most, 3 seconds for a user to read, evaluate and answer. Ok first things first, get rid of the numbers and use descriptive words:
- No
- Probably Not
- Indifferent
- Probably Would
- Yes
That covers it. Now I understand from a reporting standpoint you might want to find out a little more info… why don’t they like the product? what can we do to improve it? Thats fine, but lets not burden the user with those questions right off the bat especially since the combination of asking them about price, features, support, reputation would even turn those 10 choices to 30 or 40. So ask the original question with your 5 options and then have a tailored secondary question with multiple checkboxes for answers that are relative to their answer. “Yes”, no real need to ask those people anything… they love your product. For the people that probably would, ask the user “what they liked about the product”, these are your strong points… people that like product are already going to be focused on these, don’t ask them what they DIDN’T like, no real need to make them think about that. Indifferent, ask those people what you need to improve to get them to like the product. Probably Not and No, ask those users to tell you what they didn’t like about the product as they will be focused on the negative.
This allows you to get the most valuable information from the users that are thinking about your product/service in that frame of mind that they are in, without burdening every user with 30 or 40 questions. The goal is to make your users WANT to answer the question and not feel like they are being put through a huge process. Your average user can answer the question and check off a couple of options in 3 or 4 seconds.
So now your users are happy to give you feedback as well as your staff get valuable/useful information that is precise. This is really just an example of realizing bad UI and redefining, “what do I expect the user to do” and “how can I get them to do it without having to think”.
For more information about this topic, I highly suggest you check out Steve Krug’s Don’t Make Me Think.”Usability design is one of the most important–yet often least attractive–tasks for a Web developer. In Don’t Make Me Think, author Steve Krug lightens up the subject with good humor and excellent, to-the-point examples.”
September 27th, 2008 at 11:39 am
Hey there! First off, thanks for using the site, and glad you’re getting value out of it. Want to share two thoughts I have about your post, the first tactical, and the second more conceptual. Really interested to have this discussion with you!
1) Though I get where you’re coming from regarding intuitiveness, the issue for us, and the reason we decided to go with NetPromoter, is one of *standardization*. NetPromoter has taken off in the business world in recent years, and has become a meaningful and more importantly *credible* measure by which businesses can gauge customer loyalty both for themselves and in a reliable and standardized way against other businesses in their category. Standards, as anyone who’s ever coded a Web site can tell you, are really freaking important if you want to achieve something larger than just a bunch of ratings collected on one site in one place on the Web.
So, yes, we could roll our own approach in the manner you suggest above, but we would lose access to the larger ecosystem of value that participating in NetPromoter gives us and the companies who use us. Not denying that there might be (and often are) significant trade-offs that come with using a standardized approach — one size fits all means something rarely fits perfectly — but, simply put, there’s still a huge gain in using something that other people also use that more than makes up for the lack of tailoring. I consider this a critical piece of the argument that’s missing from the conversation above.
2) The other thing, and I wonder if Steve (who I consider a good acquaintance, as we’ve met many times through the other company I started) might not actually agree with me on this: I *want* people to have to think before they answer this question. I don’t want it to be immediate and intuitive. I would *like it* if they paused for a moment to consider what those numbers mean to them, and relatively where they end up on the scale, as you’ve done here in this post. Certainly there are times, when developing a Web interface, where you want it to be easy, mindless, effortless, intuitive — but this is one where I’d prefer a little bit of thought. “Don’t Make Me Think!” isn’t a blanket suggestion for the entire Web — it’s about making certain I don’t have to put thought into things I shouldn’t have to waste time thinking about so that I can instead focus that thought in those areas where it matters. Like, say, my thoughts about whether or not I’d actually recommend a product I use to a friend.
Frankly, too much of what passes for reviews or ratings on the Web lacks coherence, and falls into the “1-5 stars” or” Definitely like/Sort of like” category. But here, with the very important question “How likely are you to recommend this to a friend or colleague?” the approach gains significance — it gains context that then makes the measurement of those numbers against each other have real value. It’s quantifiable, measurable, comparable, meaningful, and scalable. We like it a whole lot for all those reasons.
All that said, it’s unlikely we’re going to remove NetPromoter as the fundamental base on which we consider and rank companies in our system; it’s just got too much value to us both within and outside the confines of our own site. But, I’d be *really* interested to hear if you could think of ways we could improve the intuitiveness/usability of the NetPromoter system, instead of replacing it outright. We’re on our second rev of it since we introduced it (we removed the “I’m a customer” button in favor of just assuming you’re a customer in those cases where you rank something, as in the screenshot above) but I still don’t think we’ve quite nailed it. Thoughts? Suggestions? Ideas? Your (and everybody’s) help much appreciated.
September 27th, 2008 at 12:03 pm
@monstro – Thanks for the reply. First off I want to preface this with my article wasn’t aimed at GetSatisfaction’s use of NPS… but NPS in general. Of course GetSatsifaction is a very visible example of NPS usage and a lot of people are adapting it so you are obviously then one of the champions of NPS in that sense.
I do agree with wanting to make people think, and I understand where you are coming from, but I think the difference between what you are saying and what I was referring to is you want them to think “do I want to recommend this product to someone else” and my reference to not thinking is “what is the difference between 6 and 7?” I think asking more direct questions like “would you recommend this product” with a simple yes or no answer causes people to think even on some level, but making the process of answering a puzzle into itself where you aren’t even thinking about what you would choose but what are my choices is a problem.
As far as analyzing the data without a definition of what those numbers mean relative to the product it is like asking a person “do you like vanilla ice cream” and then giving 10 customers a different vanilla ice cream, one soft serve, one with real vanilla, one low fat, etc … and then thinking that the vote is real comparative data.
As far as being a standard I do agree with that, I understand and do support the use of standards, even flawed standards for standards sake, as I said my disagreement is more with the NPS system in general and not your specific use of them. I do think that if you are really concerned about providing real quality feedback to your customers (companies) that the NPS doesn’t really provide them anything in terms of a single ranking for so many factors (price, features, support, customer service). As a user of get satisfaction from a company sense (I have 3 projects currently using getsatisfaction for support) I would love to be given the option of using NPS (maybe the default choice) and a more question based option. I of course understand the draw backs of the more clicks the less likely people are going to do it, but in my case NPS doesn’t tell me anything… did they like my product? Was the price point acceptable? Do they want it supported on more platforms? Not sure how this would effect the ranking in your system, but it would be a nice option.
Anyway, keep up the excellent work! Love the product and will continue to use it!