Is there any research on numbered vs. star voting systems?

Question

Has anyone done any research on user acceptance of the following voting systems for different target audiences?

Or

I'm not interested which is more accurate or how the votes will be used for ranking. What I'm interested is from a user perspective, which is more intuitive - based on the demographic of that user.

Obviously, as developers, we all understand the StackOverflow style voting system, but I'm curious as to whether this only makes sense because of the way we [as developers] think. Does the Amazon style star system make more sense to the voter on sites targetted a more basic users?

Has anyone done any research on this and if so what was the outcome? Does anyone have any links to research results?

Pentapody

Well what are you looking for? A system to accurately rate content based on accuracy? Or a system to rate based on popular opinion? Do not use a public voting system if you want the former. Please explain a bit more what you want the results of this voting to be. – Modla

I agree with silky that this question is too ambiguous. There are so many different scenarios for voting. Once people vote then you have to decide how to rank, and that leads to situations such as the Netflicks contest, where you try to decide how to use the voting for ranking. If that is your interest there has been research on that concept. – Fear

I don't have any facts to supply unfortunately, but I suspect that when given the option to rate something anonymously people will pick one of the two extremes. On Youtube for example people will typically one-star something or 5-star it. On Amazon however, where a vote is neither anonymous nor isolated (i.e. the voter includes his/her reasons for the vote) there are far more 2, 3 and 4 star votes. – Egret

Another thing to consider is culture based behaviour. Westerners will typically vote more extreme than Asians. When given the option to vote from 1 (not at all) to 10 (I fully agree), asian polls tend to be far more crowded in the 4,5,6 domain. The StackOverflow voting system does not suffer from this. – Egret

On the negative side for the StackOverflow voting system is that it's very obvious to developers [which are the target audience of this site] what the voting system is about, but when you're targetting an audience that may be far less computer literate, the star system is immediately obvious as we had stars right back as far as kindergarten. – Pentapody

@Ben, if you keep track of the number of downvotes and upvotes (instead of just a single integer which gets incremented/decremented), you can display the vote result as a percentage. – Egret

Yeah, I've got that. What I'm more curious about is how usable either option is for the person voting and are there particular target audiences that take to one option rather than the other. – Pentapody

Answer 1

The two "voting systems" you mentioned seem to me to serve two different purposes: voting and rating. Voting up a question means something different than rating a product 5 stars. Voting up/down seems to make more sense on a site like SO or Digg, whereas rating something with stars is probably better applied to a product or a song.

Having said that, I would think stars are more readily understood by more people.

Indiscernible

Answer 2

Gaming the Vote by William Poundstone is a great book I read recently. He's explored virtually every voting system that's been tried and analyzed them. Everything from political elections to web site voting systems is covered. (hotornot is discussed at length). I highly recommend it.

Vachill

Did he say anything about systems similar to stackoverflow's system? – Baumann

@Tchalvak: The book predates the launch of stackoverflow (by a little) so it isn't mentioned explicitly. As for "similar" systems, I think stackoverflow presents enough new innovations (eg. upvote rep != -downvote rep, wiki style features affecting voting, answer sorting rules) that it would be very interesting to see Poundstone re-visit the topic including a discussion of SO's voting system. – Vachill

Answer 3

I want to point out that if you have dealt enough with the iPhone app store, you will notice that a lot of users vote opposite (1 star instead of 5 stars) of convention with a clearly glowing review. At first I thought this was an abnormal occurrence but I've seen it hundreds of times now.

So the "up tick" vs "down tick" style makes a lot of sense in light of that trend.

Urger

Would Digg's thumbs-up/thumbs-down thing be more intuitive than the up-tick/down-tick? – Pentapody

For an app store... I think I'd rather see stars. For example, what about if an app is just mediocre? 2-3 stars. There are a lot of apps that have such a rating. What if it is great except for one or two features? 4 stars. Again, a lot of apps with a 4-star rating. – Indiscernible

jnylen - the problem is if people mistake 1 star for perfect, then a vote of two perfects, one mistaken and the other correct, averages 2.5, which according to your summary is just mediocre even though it's perfect. Ben - Yes I think so - although in a internationalized world we need to be aware that hand gestures are not universal. Thumbs up doesn't mean 'good' everywhere. – Urger