How to Get Survey Responses from Reddit

By Max Candocia

|

November 12, 2020

See What Time Should You Post to Reddit (Part 2) for more general guidelines on posting time on Reddit.

If you want to get more responses for your survey, you can always check out /r/samplesize on Reddit, where of the respondents are in the younger demographics (18-34), mainly from English-speaking countries. The political lean of the site is generally, but not exclusively, to the left.

How to get the most out of the subreddit is a bit less obvious. Namely, there are a few rules that you need to follow when posting:

  1. Your survey link must have a valid URL that matches the destination, which must be a well-trusted survey-collection site, like Qualtric, SurveyMonkey, Google Docs, etc.
  2. Your survey must be categorized as [Casual], [Marketing], or [Academic], with one of those respective tags in the title
  3. You must indicate what demographics/conditions for responders are in the title
  4. You can only repost it once every 24 hours, provided it is no longer one of the top 25 "hot" posts on the subreddit
  5. If you do repost, you must indicate it with a [Repost] tag in the title

For example, you could have a survey titled [Academic] How much do you sleep per day (Anyone 18+), or [Repost][Marketing] What kind of fish do you eat? (anyone living in US).

There are additional rules for posting, but these are the ones to keep in mind when setting a strategy for posting.

You also need to create a Reddit account if you haven't already. All you need is an email address, and it's free!

Ideal Outcome

Usually I don't get more than a few dozen responses, although in a few cases I've gotten several hundred. The most important factor in getting responses is getting more upvotes, thus a higher score, on your submission.

The Reddit algorithm will push newer posts up closer to the top of people's pages, as well as posts with higher scores. The process is a positive feedback loop: The higher score you have, the more likely people are to see your post and upvote it and/or fill it out.

There is the question of what the best way to post your survey is. Here are a few guidelines + observations, and then I will back up these claims with data:

Hint: If you need to repost a survey to /r/SampleSize, you can add ?v=1 to the end of the URL, or any number. If there's already a ? near the end of the URL, you can add &v=1 (or any number) to the end of it instead..

Posting Time

Below is a graph of different posting times and the percentage of submissions with scores over 10. Tap on the image to show more technical details in each cell.

The best times to post are generally in the morning for the US, around 12:00-15:00 UTC (or 7:00-10:00 am EST) Saturday

Additionally, in an earlier analysis for all of Reddit, that time range was among the best for posting, although Saturday and Sunday morning in the US seems to be the best overall, with a wider time window for posting (up to 6 hours earlier and 3 hours later). Thursday does not appear to be a very good day for posting.

See the limitations section at bottom of page for why the above might not be a true cause-effect relationship.

Non-Time Effects

Although planning when to do the survey is important, one can alternately look at factors that improve the average score of a post. The below graph highlights the expected percent increase in score based on the following factors.

Effects on Score Value

plot of chunk coef_graph

Effect on Odds of Score > 10

plot of chunk coef_graph10

The main takeaways:

  1. Reposting a survey reduces the expected score by about 14%. There is no apparent effect for repeated reposts.
  2. If you've previously posted results, you get a "results flair", which gives you an average of 63% more upvotes on your posts. Posting results of a previous survey is one of the easiest ways to increase visibility without otherwise changing your survey or post, and it cancels out the negative effect reposting has on the chance of scoring over 10 upvotes
  3. Results posts tend to do very well. Among actual survey posts, “casual” surveys tend to score about 66% higher than Academic. Marketing performs slightly worse by 9%.
  4. Ignoring i.reddit.com (Reddit's image host sometimes used for results), surveys posted from strawpoll.me tend to perform the best, although they only allow single questions. docs.google.com is the best for general surveys, although the effect is only about 25%.
  5. Surveys that have an inclusive audience (i.e., "everybody") tend to score 20% better

Sampling Methodology

I used my Tree Grabfor Reddit Scraper (GitHub link) to collect the most recent posts on /r/samplesize, and then for each commenter/poster, look through their past 1,000 submissions to see which ones were submitted to /r/samplesize. Posts from April 2020 and onwards were used, and must be at least 19 hours and 12 minutes old as of scraping (0.8 days). The number is a bit arbitrary, but most posts usually reach their maximum potential by that time.

7,533 posts from /r/samplesize were used in this analysis.

Limitations of Analysis

In no particular order, here are some possible caveats to the above analysis:

Additionally, there appears to be a higher proportion of casual posts on the weekends, and a lower proportion of marketing and academic posts. See the below table for proportion of posts by day of week:

ACADEMIC CASUAL MARKETING OTHER RESULTS
Monday 71.3% 20.1% 6.3% 0.2% 2.1%
Tuesday 69.8% 19.2% 8.3% 0.1% 2.5%
Wednesday 72.6% 19.9% 6.0% 0.1% 1.4%
Thursday 73.4% 17.4% 6.4% 0.2% 2.5%
Friday 73.8% 18.8% 4.9% 0.3% 2.2%
Saturday 70.0% 23.8% 4.0% 0.0% 2.2%
Sunday 68.9% 23.6% 4.4% 0.2% 2.8%

Modeling Methodology

For the effects of different post characteristics, I used a linear regression model using bidirectional stepwise selection (w/BIC). The response variable was log(1+score), which doesn't perfectly correspond to number of votes on a log score, but it is close enough for general interpretation.

GitHub Code

Code and data used for the analysis can be found on https://github.com/mcandocia/samplesize_regression.


Tags: 

Recommended Articles

How Would the US Vote for a Candy?

What would it look like if people across the US voted for a candy? Explore different results using different voting methods and different types of representation, such as a national vote versus the Electoral College.

Modifying Naive Bayes for Text Classification

Articles and other bodies of text can be classified with Naive Bayes efficiently by limiting the impact of features on the model.