A/B testing how to ask YouTubers for product reviews

I run Candy Japan, a service that mails surprise boxes of Japanese candy to subscribers around the world.

I previously wrote about an experiment where I spent $13,867 on YouTube ads. Many suggested that instead of spending money on ads, I should just send samples to YouTubers to get them to cover the service for free.

All right, I'll try it.

Gathering a contact list

Being new at this, I decided to start by contacting a larger number of small YouTube channels to learn what kind of an approach would work the best.

I assumed the response rate would be low, so I would need to find a lot of channels to contact. I wanted them to be relevant to people with a strong interest in Japan, and after several evenings of work managed to put together a list of several hundred small cosplay channels. At this point I only had their channel URL, but no way to contact them.

Many list their email address directly in their YouTube profile, while others only have a link to their Tumblr page for example. That page might then link to their other social media profiles, one of which might eventually have a way to contact them. Besides just finding an email address, I also wanted to find names. I read all channel descriptions and watched the beginning of video introductions to make sure I was addressing each recipient appropriately.

This was mind-numbing work that took me close to 10 minutes per channel initially. It's also distracting work in that I needed to spend a lot of time on YouTube and in various social networks without getting sidetracked. To be able to focus, I turned it into a little game for myself. How fast can I go through the next 20 channels? What can I do to improve my time? In the end I had improved enough that I was able to find the name and contact information for a channel in less than 2 minutes.

A little setback happened along the way: Google removed private messaging from YouTube.

I had hoped to use YouTube private messages as a fallback for channels that don't list their email address anywhere. Just as I thought that I had gathered enough channels to contact, private messaging became impossible, and I had to redouble my efforts to make up for all the channels that had now become unreachable.

In the end I had a spreadsheet of 180 cosplay YouTubers with a way to contact each. The average channel in the list has less than 4000 subscribers, with a total of 664k subscribers – power law at play here, combined these roughly match a single popular YouTuber in impact. I wasn't sure if a tiny channel could send enough sales to exceed the cost of sending them a sample, but if it did, that would scale nicely as I could keep sending more and more samples to similar channels.

A/B testing message content

I decided to amuse myself with a little multivariate test and send everyone a slightly different message to learn what works best.

Choosing a title

I tried two different subject lines for the email, but the difference was minor. The worse one turned out to be "Want a Japanese candy review box?" and the slightly better one "Free Candy Japan sample for unboxing video". Of the people who received the former, 25% agreed to do a review. Of the latter, 28.26%.

Message body

I started each message with:

Dear so-and-so,

Would you like to receive a Japanese candy review box for your YouTube channel?

I created five snippets of text, randomly including them in the messages I sent out. Below you can see each snippet, along with what the response rate was when each was absent or present.

Snippet 1: clarifying what I want them to do

I was hoping that if I send you two boxes, you could unbox and try the candies and introduce the service to your subscribers.

With the above snippet included, 29.55% agreed to do a review. Without it, 23.91% agreed. Takeaway: be clear about what you expect people to do for you.

Snippet 2: longer self-introduction

I run a site called Candy Japan, which sends surprise boxes of Japanese sweets to members twice a month.

With the above included 27.63% agreed, without 25.96% did. Takeaway: including an elevator pitch of your service may help.

Snippet 3: appealing to popularity

These types of "trying Japanese candy" videos tend to be very popular, many having millions of views.

With the above included 24.72% agreed, without 28.57% did. Takeaway: don't be patronizing?

Snippet 4: offering their viewers a discount

I can also give your viewers a discount coupon.

With the above included 27.38% agreed, without 26.04% did. Takeaway: it doesn't really matter if you offer a discount or not. In the end I still gave everyone a discount coupon anyway.

Snippet 5: call-to-action

If you would like to receive a review box, please let me know your mailing address.

This had the highest impact: with the above included 21.11% agreed, without 32.22% did. Note that even without this line, in the end everyone still gave me their mailing address, so this only means that it's better not to ask for it right away.

I can think of two reasons for this. Either people need a bit more back-and-forth before parting with such personal information, or possibly it's just more work to reply to this email, so more people will think "I'll reply later", and then forget about it.

Include image or not?

Another variation I tried was attaching the above image of the candy I might send them. I expected this to be a no-brainer in that having the picture would be better, but this actually turned out not to matter. Out of those who got the image 26.97% agreed vs. 26.37% for those who did not get it.

My guess here is that it may actually be beneficial to have the image, but that it is balanced out by more of these emails ending up classified as spam.

Overall response rate

I sent out 180 messages, and guessed that perhaps a dozen people would agree to do a review. In the end 48 YouTubers sent me their shipping address. Only 5 people outright refused to do a review, and even they still wrote polite responses.

Having 26.67% agree to do a review was a much higher rate than I had imagined, and I ended up annoying my wife by having a larger than expected pile of samples in our apartment. I was taking the samples to the post office by bicycle. Below you can see one batch that I was able to carry at once. For some reason it takes close to 30 minutes for the local post office to process each batch, the UI in their POS must be pretty terrible.

One wrong assumption I had was that more popular YouTubers would be harder to get to agree to do a review. Splitting my list of YouTubers evenly into "less popular" and "more popular" groups of less or more than 2290 subscribers, the agree rates were 27.78% and 25.56% respectively – a difference of only two more YouTubers agreeing in the less popular group.

Thanks for reading

It is a fair amount of work to find enough relevant YouTubers to contact, to correspond with them and to actually ship samples.

If you need to ship a YouTuber a physical product sample, don't ask for their address until after they have replied to the initial email. Include an elevator pitch of your service and be clear what you want them to do in the video.

With this approach when contacting relevant less popular YouTubers, you may expect around 30% of recipients to agree to do a review.

I have now sent out most sample boxes, but not everyone has made their videos yet. It will also take some time to know how popular the videos will turn out to be. I will do a part 2 later once I have data on how many sales these videos actually resulted in. Until then 👋

Oh, and if you'd like to try some Japanese candy for yourself, you can sign up here.