When I was leading the product design team at AWeber, it was standard practice to offer an incentive to users (usually an Amazon gift card) to prompt them to fill out a survey or sign up for an interview.
We wanted to know, however, if those incentives were adding value. Could we still get customers to respond, even without the offer of a gift? 🤔
To test this, we created a small survey (just 2 questions with free-text responses) and ran a 50/50 split test on the opt-in box that appeared on the AWeber dashboard:
- 50% were offered a chance to win one of four $50 gift cards if they responded 💰
- The other 50% were offered no incentive 🚫
To determine the value of the responses, we focused on two main factors:
- Total amount of responses (to achieve statistical significance) 📈
- Quality of the free text responses – measured by comparing the amount of useful vs. unhelpful responses (blank or gibberish) 👌
The results showed that the incentive drove many more responses. Of the 1,014 total responses:
- 78% (793) were incentivized 💰
- 22% (221) were not incentivized 🚫
We also found that the quality of the responses from the two free-text questions was higher with the incentive:
Question 1
- 23% unhelpful responses with the incentive 💰
- 28% unhelpful responses without the incentive 🚫
Question 2
- 7% unhelpful responses with the incentive 💰
- 17% unhelpful responses without the incentive 🚫
The data from this test showed that offering an incentive had a positive impact on the two main factors that we were tracking – total responses AND response quality, with total incentivized responses greatly outnumbering those without an incentive.
This data allowed me to present a case to the AWeber leadership team and get continued buy-in on the value of offering incentives to get meaningful customer feedback. 😎
A final takeaway from this experiment – the data shows a clear benefit to offering an incentive when the amount of responses really matters, like when running a general survey that encompasses the entire product.
That being said, we still saw customer engagement even without an incentive. Depending on how engaged your customers are, it may be worth running smaller, feature-specific surveys without one to keep costs down.
❓- Do you offer your customers research incentives? I’m interested to hear how others get quality user feedback.