Appendix A
Instructions
Oral Instructions (Read Aloud Before the Experiment, in German)
Welcome to the experiment and thank you very much for your participation. I will briefly read you some general explanations about the experiment. Please do not click on “Start experiment” until the end of these instructions. The participants of the experiment are all here in this room and are all taking part in the same experiment. The experiment aims at gaining insights on human behavior. The experiment lasts about 90 min and on average you will receive between 7 and 15 €, depending on your behavior, but at least 2 €. You will play anonymously and can’t coordinate with each other. The disbursement of payoffs will also be carried out anonymously. No other participant will see how much you receive and the experimenters will not find this out either. During the experiment you may have to wait for the other participants. This may take a few minutes. Please remain patient during this time. When everyone has finished the experiment, you will be asked to go outside one after the other. There you will receive your payment. All instructions and explanations can be found on the following screen pages. Please read all the information carefully before leaving a screen by mouse click. Once you leave a screen, you will not be able to access it again. Never use the back function of the browser and do not surf on other websites during the experiment. We will log the accessed pages during the experiment. Never close the browser! A violation of these rules will exclude you from any payoff. Please remain calmly seated at your workplace. Please refrain from any conversations. If you have any questions, please raise your hand. We will then come to you. Now click on “Start experiment”.
Instructions—Welcome to the Experiment!
Thank you very much for your participation! At the beginning of the experiment the general laboratory procedures will be explained. These will be read aloud by the experimenter. Please click on “Start experiment” as soon as you are asked to do so. Never use the back-function of the browser and do not surf on other websites during the experiment. We log the accessed pages during the experiment. Never close the browser! A violation of these rules will exclude you from any payoff (Fig. A.1).
Instructions—General Information
In this experiment you have the task to write product reviews in an online shop and to evaluate the reviews of 4 other reviewers. You always interact with the same 4 reviewers over 4 rounds throughout the experiment. Each reviewer has identical tasks and receives the same instructions. At the beginning of the experiment you will receive € 6.4 as initial endowment. In each round you will receive earnings or deductions to your payoff account depending on your behavior. The remaining payoff account at the end of the experiment will be paid to you. In the experiment characters will be converted into Euro. 25 characters = 10 Eurocents (Fig. A.2).
Instructions—Phase 1: Writing the Review
Each round consists of three phases, which are explained to you one after the other.
Review—Rating—Round result
At the beginning of each round you will receive a product to which you should write a short review. The review should describe the product and make it easier for other users to make a purchase decision.
You will also be asked to give the product an overall grade (school grade 1–6) and indicate how much you are willing to pay to buy the product. This information will not be shared with other reviewers (Fig. A.3 and Table A.1).
Table A.1 Instructions—Phase 1: Writing the review Instructions—Phase 2: Rating Reviews of Others
After writing your own review you will be asked to evaluate the reviews of the 4 other participants regarding their helpfulness.
Helpfulness Assessment
Helpfulness is the usefulness of the review for a possible purchase decision. You can rate helpfulness on a scale of 5 from unhelpful (1 star) to very helpful (5 stars).
Please note that all reviews must be rated (at least 1 star = not helpful at all).
ONLY IN TREATMENT BT
Award for the review with the best helpfulness rating.
The review that gets the best average helpfulness rating from the other 4 reviewers gets a price of € 5 in addition to its normal payoff.
If several reviews have the same average rating, the price will be split.
ONLY IN TREATMENT FWT
The rating has no effect on the payoffs (Fig. A.4).
Instructions—Phase 3: Result of the Round
At the end of each round, you will be informed about your total payoff for that round and the average helpfulness rating for your own review.
ONLY IN TREATMENT 1 You will also receive € 1 for evaluating the product (overall impression, willingness to pay) and rating the other reviews.
Your Payoff at the End of the Round
You have written a review with X characters. These characters will be deducted from your account balance. The other 4 reviewers wrote reviews with a total of Y characters.
Since you also benefit from the reviews as a user, you will receive half of the characters credited as a payout.
ONLY IN TREATMENT FWT You will additionally receive 1 € credited for evaluating the product and rating the other reviews.
ONLY FOR THE BEST REVIEW IN TREATMENT BT You have the best helpfulness rating and get an additional payoff of € 5.
Purchase of Test Products
In the first phase you indicated how much you are willing to pay for the product. If this willingness to pay is higher than the purchase price of the product (purchase on the Internet), you will receive the product at purchase price and the purchase price will be deducted from your payoff account. Please note that you will receive the product at purchase price. Important: The indication of the willingness to pay is binding.
Example
If you are willing to pay a high price, you can enter it as willingness to pay, but you will then receive the product at the cheaper purchase price.
Payment and Questionnaire
At the end you will find an anonymous questionnaire about your experiences with online review systems.
When you leave the laboratory, you will receive your payment and, if applicable, the products you have purchased (in their original packaging) (Fig. A.5).
Decision Screens
All decision screens show a bar with (1) the round number, (2) the stage in the current round (review, rating, result of the round), (3) the total payoff (excluding the current round) and (4) the conversion rate of characters into Euros (25 characters = 10 Eurocents) (Fig. A.6).
Decision Screen 1—Writing Review
On the first screen, participants can write their review. The actual text on screen is marked by “”. The remaining text is explanation (Fig. A.7).
-
1.
General instructions for writing the review: “The product is now distributed. It is collected after the round. Please write a review about the product. You have max 400 character available. Each character that you write will be deducted from your payoff account. By writing the review, you create utility for the other reviewers. They each get half of your characters as payoff.”
-
2.
In the text field, subjects could write their review and were informed about the remaining characters.
-
3.
In the info field, subjects could see (in live) how writing affected their payoff. The text states: “Your payoff account (after deducting the characters from your review)”.
-
4.
In the info field, subjects could see (in live) how writing affected the others’ payoff. The text states: “Euros that will be given to each of the 4 other reviewers.”.
-
5.
This message informs subjects about the fact that “The following inputs are not shown to the other reviewers”. (meaning 6 and 7).
-
6.
Subject have to rate the product. “Please rate your overall impression of the product with a school grade (1–6)”. In Germany, 1 is the best and 6 the worst grade. You pass with at least a 4.
-
7.
Subjects have to express their willingness to pay for the product. “How many € are you ready to pay to buy the product? (willingness to pay)”.
-
8.
The message at the bottom informs subjects about the binding nature of the input in 7. “If your willingness to pay is above the buying price of the product, you get the product at the end of the experiment for the buying price. This input is binding.”
-
9.
“Submit inputs”
Decision Screen 2—Rating Reviews
Then subjects rate the reviews of the 4 other reviewers (Fig. A.8).
-
1.
The instruction state “Please rate the helpfulness of the 4 other reviews with a star rating. (1 = not helpful at all, 2 = little helpful, 3 = average helpful, 4 = helpful, 5 = very helpful). The reviewer with the highest average rating in your group of 5 gets additional 5 €. With a tie, the award will be split”. (for the BT treatment). In the FT, the last two sentences were changed to “The rating does not influence your payoff.” The € 5 bill was only shown in the BT treatment.
-
2.
Each review is headed by stating “Participant A has written the following review:”
-
3.
The review was reproduced. The number of characters was added in brackets “(This review has 347 characters)”.
-
4.
“Please rate the helpfulness of the review”.
Decision Screen 3—Feedback
The third screen provides feedback (Fig. A.9).
-
1.
In the BT treatment, subjects were informed if they had the highest rating. “You have the highest helpfulness rating and get € 5.” If not they were informed as well. In the FWT treatment, this part was omitted.
-
2.
Subjects were informed about the average helpfulness rating they obtainted. “Your review got an average helpfulness rating of 1.25 by the 4 other reviewers”.
-
3.
Feedback on payoffs was provided: “In this round, you received the following payoff. You wrote a review with 208 characters. These characters were deducted from your payoff account. The other reviewers wrote reviews with 359 characters in total. As a user, you profit from these reviews and you get half of all characters as payoff. You had the highest helpfulness rating and got an additional payoff of € 5.” The last sentence was changed accordingly in treatment FT to “You will additionally receive 1 € for evaluating the product and rating the other reviews.”
-
4.
Feedback on total payoff. “Overall, in this round your payoff has increased by € 4.886”.
-
5.
Button to start the next round with warning “Please click only on the button once you read all instructions”.
Questionnaire After Experiment
Thank you for participating in the experiment! The following questionnaire will collect your experiences with online product reviews in the “real world”. Please fill in the following questionnaire carefully. Answering the questions takes about 10–15 min. The data collected cannot be personally attributed to you. This questionnaire has 29 questions.
Experience as Reviewer—Part I
The following questions ask about your experience when writing online product reviews.
-
1.
How many product reviews have you written on Amazon?
-
None
-
Up to 10
-
Up to 20
-
Up to 50
-
More
-
2.
Have you already written reviews for other e‑shops than Amazon? Question only asked if product review written (question 1).
-
3.
When was the last time you wrote a review? Question only asked if product review written (question 1).
-
In the last week
-
In the last month
-
In the last year
-
Over a year ago
Motivation Reviews in General
-
4.
What motivates you to write product reviews? Question only asked if product review written (question 1). Please select the appropriate answer for each item: (1 = absolutely true, 2 = true, 3 = hardly true, 4 = not true)
-
In return, I’m receiving monetary incentives such as money or coupons.
-
I will receive free test products in return.
-
My experience helps other customers with the assessment of product quality.
-
I enjoy it to write reviews.
-
I like to be in contact with other reviewers and readers.
-
A reputation as a good reviewer is important for me.
-
I want to support other customers to buy the right products.
-
I like to exchange myself with people who have similar interests.
-
I hope that others reviewers and readers will give me advice on problems with products.
Experience as Reviewer—Part II
-
5.
What was (or is) your best reviewer rank on Amazon? Question only asked if product review written (question 1).
-
Best 100
-
Best 1000
-
Best 10,000
-
Best 50,000
-
Best 100,000
-
Higher than 100,000
-
I do not know
-
6.
Do you participate in the Amazon Vine program? Question only asked if product review written (question 1).
-
7.
Have you already received “Not helpful” votes for your reviews? Question only asked if product review written (question 1).
-
Never
-
Few
-
Some
-
A lot
-
I do not know
-
8.
Were you able to understand this evaluation? Question only asked if product review written (question 1) and if not helpful votes received (question 7).
-
All
-
Almost all
-
Some
-
Few
-
None
-
9.
What are the reasons for you to write a positive product review? Question only asked if product review written (question 1). Please select the appropriate answer for each item: (1 = absolutely true, 2 = true, 3 = hardly true, 4 = not true)
-
I want to help other people with my positive experience.
-
I want to give other people the opportunity to buy the right product.
-
So I can express the joy of a good purchase.
-
I like to tell other people about a successful purchase.
-
So I can show other people that I have bought cleverly.
-
I would like to recommend the company.
-
I like to support good companies.
-
10.
What are the reasons for you to write a negative product review? Question only asked if product review written (question 1). Please select the appropriate answer for each item: (1 = absolutely true, 2 = true, 3 = hardly true, 4 = not true)
-
This is how I better process the frustration over a bad buy.
-
So my anger over a bad buy is reduced faster.
-
I want to warn other customers about bad products.
-
I want to spare other customers a bad product experience.
-
I want to pay back the manufacturer of the bad product.
-
It is less time-consuming than complaining to the manufacturer by phone or e‑mail.
-
I believe that the manufacturer will solve the problems of their product faster if I discuss them publicly.
-
The provider of the review platform will forward my complaint to the right place at the manufacturer.
-
11.
How often do you shop on Amazon?
-
More than 30 products a year
-
Between 30 and 10 products a year
-
Less than 10 products a year
-
Never
-
12.
How often do you shop on the Internet?
-
More than 30 products a year
-
Between 30 and 10 products a year
-
Less than 10 products a year
-
Never
-
13.
How often do you read product reviews on Amazon before you make a purchase of a product?
-
Always
-
Often
-
Sometimes
-
Rarely
-
Never
-
14.
How often do you read product reviews on other online platforms before you make a purchase of a product? Question only asked if customer never read on Amazon (question 13).
-
Always
-
Often
-
Sometimes
-
Rarely
-
Never
-
15.
I read first those reviews which … Question only asked if review are read (question 13). Please select the appropriate answer for each item: (1 = always, 2 = often, 3 = sometimes, 4 = rarely, 5 = never)
-
is displayed first automatically
-
was rated as most helpful by other customers
-
is the most recent
-
evaluate the product best
-
evaluate the product worst
-
16.
Before I decide to purchase a product, I read all the available product reviews (on Amazon).
-
Always
-
Often
-
Sometimes
-
Rarely
-
Never
Reading Motivation Customers
-
17.
I read the product reviews of other customers … Question only asked if reviews read (question 13). Please select the appropriate answer for each item: (1 = absolutely true, 2 = true, 3 = hardly true, 4 = not true)
-
because they’re helping to make the right purchase decision.
-
because it saves a lot of time if I want to inform me about a product before buying it.
-
in order to find advice and solutions for problems.
-
because I feel better, when I read that other people have the same problem with a product.
-
to benefit from the experiences of others, before I buy a product.
-
because it is the fastest way to get information about a product.
-
because I get to know about recent trends.
-
to find confirmation, that I have bought the right product.
-
to compare my product evaluation with that of other people’s.
-
because I like to share the experiences with others reviewer.
-
because I am rewared for reading and rating (e.g. vouchers, free test products).
-
to find the right answers when I have problems with the product.
-
because I like to be part of the review community.
-
because I am interested in new products.
-
to find out if I am the only one with a certain opinion about a product.
-
18.
How often have you rated the reviews of other customers? Question only asked if reviews read (question 13).
-
Never
-
Up to 10 times
-
Up to 20 times
-
Up to 100 times
-
More often
Customer View Helpfulness
-
19.
I find a customer review particularly helpful if it … Question only asked if reviews read (question 13). Please select the appropriate answer for each item: (1 = absolutely true, 2 = true, 3 = hardly true, 4 = not true)
-
Discusses the disadvantages of the product.
-
Discusses the advantages of the product.
-
Is easy to read.
-
Contains much expert information.
-
Discusses both advantages and disadvantages of the product.
-
Is very short.
-
Extensively discusses the experiences of the reviewer with the product.
Customer Opinion Formation
-
20.
I find reviews that are already rated as “helpful” by many other customers. Question only asked if reviews read (question 13).
-
Always
-
Often
-
Sometimes
-
Rarely
-
Never
-
21.
Have you ever given up buying a product because there have been no or very few customer reviews?
-
22.
Have you ever bought a product, although you had found mostly bad reviews for it before the purchase?
-
23.
If a product in which I am interested is rated negatively in a review, I do not make a purchase.
-
Always
-
Often
-
Sometimes
-
Rarely
-
Never
-
24.
If a product in which I am interested is rated positively in a review, I buy the product or at least seriously consider a purchase.
-
Always
-
Often
-
Sometimes
-
Rarely
-
Never
Demographic Information
-
25.
Are you male or female?
-
26.
Are you enrolled as a student?
-
27.
Which subject do you study? Question asked only if enrolled as student (question 26).
-
Teaching
-
Law
-
IT
-
Economics
-
Business Administration
-
Media and Communication
-
International Cultural and Business Studies
-
European Studies
-
Governance
-
Internet Computing
-
Mobile and Embedded Systems
-
Linguistics and Text Sciences
-
Other
-
28.
In which subject area did you achieve your highest level of education? Question asked only if not enrolled as student (question 26). Options are the same as in question 26.
-
29.
How old are you?
Appendix B
Additional Results
What are the factors that drive the assignment of helpfulness ratings? To answer this question, we performed an OLS regression for each treatment, with the helpfulness rating as the dependent variable (see Table B.1).Footnote 28 The independent variables fall into four categories.
The first category refers to review quality and includes only review length, our proxy for quality. In FWT, review length is positively and highly correlated with helpfulness. Helpfulness increases by 0.843 units for 100 additional characters; in BT, the increase is only 0.157 (and significantly lower than 0.843, two-sided t-test, p < 0.001).
Table B.1 OLS regressions with helpfulness ratings assigned to other as dependent variable The second category contains all variables which reflect strategic considerations. Due to strategic considerations, the length of a participant’s own review could be positively correlated with the helpfulness rating. Consider a participant who wrote a short review while her group members wrote long reviews. In order to win the reward, she has to obtain the highest helpfulness rating in her group. If she expects that longer reviews receive higher helpfulness ratings, she expects the other group members’ helpfulness ratings to be higher than the helpfulness rating she receives. In such a situation, the only thing she can do to maximize her chances of winning the reward is to assign lower ratings to others. In fact, we expect an inverse relationship between the length of the review written by a participant and the incentive to downvote others. The shorter the own review, the greater incentive to downvote other participants. Contrary to these considerations, we found the length of a participant’s own review to have no effect on the helpfulness rating she assigns.
The helpfulness rating received in the previous round could have a positive effect. Such a pattern would be consistent with indirect reciprocity, where participants who have received a high rating in one round (a form of approval) are more likely to assign a high rating in the next round. Our data does not support this pattern. In both treatments, the previous round’s helpfulness rating has no effect.
The last strategic factor is the event of winning the bonus. In BT, winning the bonus has no significant effect on helpfulness ratings.
The third category of variables pertains to product evaluation. Differences between participants in their perception of a review’s helpfulness could be driven by differences in product evaluation. Participants may consider reviews which voice different opinions to their own on the product less helpful in a purchase decision. We control for this effect with the variables “product score” and “WTP” (both standardized as z-scores). To account for the possibility that only large differences may matter and the direction of the deviation may not, we include squared differences in scores and WTP. Only in FWT, the difference in scores influences the assignment of helpfulness ratings in the expected direction. In BT, different product perceptions does not influence the helpfulness ratings.
The fourth category contains a set of control variables for gender and field of study. None of the control variables had a significant effect. In both regressions in Table B.2 we control for group and round effects.Footnote 29
Summarizing these results, we can say that the assignment of helpfulness ratings is driven by review length. All other factors had only small or non-significant effects. The helpfulness ratings are proxies for review length, which are not biased by strategic considerations or differences in product evaluations. This implies that the differences in correlations between treatments cannot be explained by differences stemming from strategic considerations or differences in product evaluations. Comparing the values for R2, we see that in FWT compared to BT, the dependent variables explain a much larger share of the variance in helpfulness ratings. This is in line with the presence of strategic downvoting.
As a robustness check for our main results in Table 2 we also provide the results from a fixed effect regression, which are well in line with the OLS results.
Table B.2 Fixed effects regressions with average helpfulness rating as dependent variable As a supplement to Table B.1, we regress review length on rounds. In Table B.1, due to the lags, the first round is excluded and we report dummies for round 3 and 4 there. The results from Table B.1 are confirmed by this additional analysis. Only in treatment FWT, the coefficient for the round is negative and significant, indicating that review length decreases by about 30 characters per round.Footnote 30 The regressions support the results found in Fig. 5.
Table B.3 Review writing behavior over time with review length as the dependent variable