How Candy Japan got credit card fraud somewhat under control

I started Candy Japan in summer of 2011 with the idea of mailing Japanese sweets to subscribers around the world on a twice-monthly basis. It worked. Hundreds of people signed up through PayPal and we kept sending them crazy Japanese candy for years.

Not everyone is a big fan of PayPal. Imagined threat or not, it seemed risky to have all subscriptions depending on it. For these reasons I wanted it to be possible to sign up directly with a credit card as well. Just a form on the site where people could enter their payment details.

Back then it was still a fair bit of work to find a suitable credit card processor to accept payments directly on your site (unlike now). I spent a ton of time going through different options, finally finding the magic payment combination that got me the form I wanted.

When I finally got things set up, I was so happy we went to the beach and celebrated it with fireworks. My understanding of how to charge cards at this point was "slap a Recurly widget on your site and call it a day".

Things were good. Until they weren't.

Data breaches affect your store, even if your own database is safe

Credit card numbers leak. The source could be a compromised database, or just an untrustworthy employee skimming cards to supplement their wage at a reputable store. Sometimes millions of card numbers are liberated in a single event.

Because of these leaks, you can go out and buy valid card numbers for about $10 per number, sold in bulk or in smaller packs of confirmed numbers. There are even people who travel the world using flight tickets purchased with stolen cards.

There are plenty of cards already out there.

In June 2015 I started feeling that fallout. I thought we were just having a particularly great month, but it turned out almost all of those new orders were just fraud. It cost thousands of dollars in fees and lost product.

It wasn't a single event either, in fact it is still happening even as I write this. On each fraud attempt I get a notification, and it's blinking often enough that I'm tempted to turn that off. At this point the problem is so bad that when I get a new candy subscriber, it is as likely to be fraud as it is to be a real order.

Trying to fix the issue

My initial reaction was to just turn off credit card orders completely, reverting to PayPal only. After this the amount of new subscribers turned into decline for the first time. Partly because not everyone likes PayPal, but also because I started feeling burned out and spent less time promoting the business.

After recollecting my thoughts, I started reading more about what this fraud is and the motivations of the people behind it. I asked around online and even gave a presentation at a Hacker News meetup to solicit some ideas.

Mostly the suggestions involved either signals for trying to guess which orders might be fraud, or countermeasures to try to make things harder for the fraudsters.

Some examples of signals are checking if many orders originate in a short span of time from the same IP address, whether the IP address country same as the shipping address country, is the email from an anonymous provider such as yopmail and so on.

The problem with signals is that the fraudsters also know about them, and will quickly adapt when they notice you are checking them. It took less than a day for them to start generating new IP addresses for every order once I banned multiple orders in succession from same IP. Same thing when I started requiring a verified email address. They would just generate those on hotmail, outlook or by using throwaway email services.

Countermeasures try to make things harder for the fraudster. The huge issue with these is that you are generally also making things harder for legitimate customers and losing sales as not everyone is willing to jump through fire-lit hoops just to get some candy. This might not seem like a big deal, but it can affect the cost of acquiring customer such that you can no longer afford to promote your site. Some examples are requiring email address verification, connecting with Google+ or Facebook account, SMS / phone verification, captchas and so on.

Key insight #1

Imagine you're in a war. A sniper shot just barely misses you and you take cover. You don't then stand up, spread your arms and shout "I'm still alive!". You want them to think they got you.

On a whim during Christmas holiday in Finland I emailed Derek Sivers to ask how he had solved the problem for his store CD Baby. After flying back to Japan to spend the New Year with my wife's family, I checked my email and was thrilled when I actually got a reply from him!

Characteristically it is also one of the nicest emails I have ever received. It contained a ton of great information, but one insight stood out in particular: don't let the fraudsters know whether their order succeeded.

If you suspect an order is fraud, don't go out and say to the criminal "hey, I declined your super suspicious order!". Instead, play dead. Pretend they got you. Tell them "thank you for your order", behaving exactly the same way as if it really was a successful order.

Now if criminals are using your site to check validity of card numbers... well they can't do that anymore, because it looks like every card they try is valid. If on the other hand they are really trying to get free candy with a stolen card, they'll hopefully be happy after their initial "success" and go away.

Key insight #2

Sadly the Recurly widget wouldn't allow me to play dead so easily, and would rather give fraudsters instant feedback of the success of their charges. I had to do a deeper API integration that took longer than I had hoped for, but I learned another useful thing on the way.

To avoid risking a data breach, you don't ever want to store or even see card numbers hit your server. When you have a payment form that allows typing in a credit card number, you "tokenize" the numbers first. As an example, suppose a customer is typing in the number “1444–4444–4444–4444”. When they hit "submit", you pass the number to a trusted third card number vault, which turns it into a token such as“abcdefGHJIJ1234”.

What's so great about tokens? Tokens are pointers to card numbers not stored on your own server. Whenever you want to charge a card, instead of the number you transmit the token. Unlike card numbers, tokens are only valid for a number of minutes. As they are temporary, they are much less risky to store. If your database got leaked, the attackers would not be able to charge any cards.

Token expiration is a strength, but to me it seemed it could also be a challenge. I wanted to be able to manually confirm any orders before letting them through (to avoid any chargeback fees), but as the tokens are only valid for a number of minutes there wasn't enough time for any manual checking.

Reading up more on the API I learned that while you really do need to use the token within the time limit (20 minutes for Recurly), setting up a charge that happens in the future also counts as "using". Now when a new order comes in, I tell Recurly "charge this token 2 weeks from now". Now I have plenty of time to read the orders before they are actually charged, and this works out just fine since we only ship candy out twice a month in any case.

Conclusion

By not sharing order success with potential fraudsters, using delayed charging of cards and looking at signals for hints on which orders may be fraudulent, I am now at a point where I have been able to re-enable credit cards. Whether this will reduce fraud to a tolerable level or if I will need to go PayPal-only again will remain to be seen.

If you would like me to send you some candy, you can order here. If it's all the same to you, use PayPal so I won't have to manually check your order :-)