How to Get Survey Responses from Reddit

By Max Candocia

November 12, 2020

If you want to get more responses for your survey, you can always check out /r/samplesize on Reddit, where of the respondents are in the younger demographics (18-34), mainly from English-speaking countries. The political lean of the site is generally, but not exclusively, to the left.

How to get the most out of the subreddit is a bit less obvious. Namely, there are a few rules that you need to follow when posting:

1. Your survey link must have a valid URL that matches the destination, which must be a well-trusted survey-collection site, like Qualtric, SurveyMonkey, Google Docs, etc.
2. Your survey must be categorized as [Casual], [Marketing], or [Academic], with one of those respective tags in the title
3. You must indicate what demographics/conditions for responders are in the title
4. You can only repost it once every 24 hours, provided it is no longer one of the top 25 "hot" posts on the subreddit
5. If you do repost, you must indicate it with a [Repost] tag in the title

For example, you could have a survey titled [Academic] How much do you sleep per day (Anyone 18+), or [Repost][Marketing] What kind of fish do you eat? (anyone living in US).

There are additional rules for posting, but these are the ones to keep in mind when setting a strategy for posting.

You also need to create a Reddit account if you haven't already. All you need is an email address, and it's free!

Ideal Outcome

Usually I don't get more than a few dozen responses, although in a few cases I've gotten several hundred. The most important factor in getting responses is getting more upvotes, thus a higher score, on your submission.

The Reddit algorithm will push newer posts up closer to the top of people's pages, as well as posts with higher scores. The process is a positive feedback loop: The higher score you have, the more likely people are to see your post and upvote it and/or fill it out.

There is the question of what the best way to post your survey is. Here are a few guidelines + observations, and then I will back up these claims with data:

• "Casual" posts do much better than "Academic". "Marketing" posts tend to do the worst.
• Including a wider audience in your survey will help.
• A survey being reposted significantly decreases its potential. This might be due to why a survey is reposted in the first place, i.e., why would someone repost a survey after it's successful?
• If you have a "Shares Results" flair, that will significantly boost the likelihood of your post getting > 10 upvotes
• strawpoll.me links seem to perform well. Granted, you can only ask one question with this site.
• Posts on Saturday, 6:00-15:00 UTC (2:00 am- 11:00 am EST) have a tendency to score higher.

Posting Time

Below is a graph of different posting times and the percentage of submissions with scores over 10. Tap on the image to show more technical details in each cell.

The best times to post are generally in the morning for the US, around 12:00-15:00 UTC (or 7:00-10:00 am EST) Saturday

Non-Time Effects

Although planning when to do the survey is important, one can alternately look at factors that improve the average score of a post. The below graph highlights the expected percent increase in score based on the following factors.

Effects on Score Value

Effect on Odds of Score > 10

The main takeaways:

1. Reposting a survey reduces the expected score by about 14%. There is no apparent effect for repeated reposts.
2. If you've previously posted results, you get a "results flair", which gives you an average of 63% more upvotes on your posts. Posting results of a previous survey is one of the easiest ways to increase visibility without otherwise changing your survey or post, and it cancels out the negative effect reposting has on the chance of scoring over 10 upvotes
3. Results posts tend to do very well. Among actual survey posts, “casual” surveys tend to score about 66% higher than Academic. Marketing performs slightly worse by 9%.
4. Ignoring i.reddit.com (Reddit's image host sometimes used for results), surveys posted from strawpoll.me tend to perform the best, although they only allow single questions. docs.google.com is the best for general surveys, although the effect is only about 25%.
5. Surveys that have an inclusive audience (i.e., "everybody") tend to score 20% better

Sampling Methodology

I used my Tree Grabfor Reddit Scraper (GitHub link) to collect the most recent posts on /r/samplesize, and then for each commenter/poster, look through their past 1,000 submissions to see which ones were submitted to /r/samplesize. Posts from April 2020 and onwards were used, and must be at least 19 hours and 12 minutes old as of scraping (0.8 days). The number is a bit arbitrary, but most posts usually reach their maximum potential by that time.

7,533 posts from /r/samplesize were used in this analysis.

Limitations of Analysis

In no particular order, here are some possible caveats to the above analysis:

• The data is observational, as individuals posting to the subreddit generally do so when convenient. True causal effects are harder to identify and enumerate because of this.
• Votes do not correlate 100% with responses to a survey. Usually you will get several times as many responses as upvotes, and the actual number is fudged by Reddit's anti-spam algorithm.
• Individuals who post should have some idea when their target audience is awake. For example, someone targeting Americans would most likely not be posting at midnight local time.
• There is some sampling bias with looking at the history of users who posted surveys in the past. This effect is somewhat limited by limiting the date range, but it is useful for taking a look at how successful reposts are.

Additionally, there appears to be a higher proportion of casual posts on the weekends, and a lower proportion of marketing and academic posts. See the below table for proportion of posts by day of week:

Modeling Methodology

For the effects of different post characteristics, I used a linear regression model using bidirectional stepwise selection (w/BIC). The response variable was log(1+score), which doesn't perfectly correspond to number of votes on a log score, but it is close enough for general interpretation.

GitHub Code

Code and data used for the analysis can be found on https://github.com/mcandocia/samplesize_regression.

Recommended Articles

"Error Bars" on Tiled Heatmaps

Heatmaps are a useful way of plotting 2-dimensional data, such as cross-tabulations. Adding "error bars" can seem non-intuitive, but expressing them in your visualization is possible with a small trick.

Subreddits That Get You the Most Awards

Which Subreddits are most likely to generate awards for their users?