Since UX research was introduced at AWeber, it has been standard practice to offer an incentive to users (usually an Amazon gift card) to prompt them to fill out a survey. We wanted to know if those incentives were adding value, or if we could just ask the same questions without the offer of a gift. To test this we ran a 50/50 split test on a survey opt-in that appeared on the dashboard. 50% were offered a chance to win one of four $50 gift cards, and 50% were offered no incentive. To determine the value of the responses, we focused on two main factors:
- Total amount of responses (to achieve statistical significance)
- Quality of the free text responses (measured by comparing the amount of blank or unhelpful responses)
The results showed that the incentive drove many more responses. Of the 1,014 total responses:
- 78% (793) were incentivized
- 22% (221) were not incentivized
We also found that the quality of the responses from the two free-text questions was higher with the incentive:
Question 1
- 23% blank or unhelpful responses with the incentive
- 28% blank or unhelpful responses without the incentive
Question 2
- 7% blank or unhelpful responses with the incentive
- 17% blank or unhelpful responses without the incentive
The data from this test shows that offering an incentive had a positive impact on the two main factors that we were tracking – total responses and response quality, with total responses greatly outnumbering those without an incentive. This shows a clear benefit to offering an incentive when the amount of responses really matters, like when running a general survey that encompasses the entire product.
That being said, we still saw customer engagement even without an incentive, so it may be worth running smaller, feature-specific surveys without one to keep costs down.