It was a really lousy week.
One of 37signals’s key products is Campfire, a real-time chat tool for small businesses. For about a week in mid-December, Campfire, which users access via the Web, kept bouncing on- and offline. This was the first major problem we have had with Campfire since it launched in 2006. For a product that needs to be as reliable as the dial tone on your phone, things couldn’t have been worse.
Thousands of companies rely on Campfire. At 37signals, we use Campfire to run our business. Because we have employees in a dozen cities around the world, Campfire is our lifeline. It’s how we communicate with one another in real time. Campfire is where we make decisions, share designs, debate ideas, broadcast companywide announcements, and keep up to date on what everyone’s working on.
Campfire went down for highly technical reasons, so I won’t go into detail about it. Essentially, three problems emerged at the same time, a combination of hardware and software failures that caused a lot of people a lot of problems. In some cases, we’d be down for only a minute or two. In others, outages would last as long as 45 minutes. For nearly a week, it was impossible to be sure if Campfire would work.
Downtime disgusts us. It’s embarrassing. Knowing that our mistakes are disrupting thousands of people’s lives weighs heavily on us. No one is happy around here when our customers aren’t happy.
And believe me, they let us know when they’re not happy. They flood our inbox with complaints. They broadcast their exasperation on Twitter. Here’s a little bit of what we heard when Campfire crashed: “I wish @37signals could sort their sh*t out, causing me a lot of grief lately.” “Come on guys. Killing my business over here…” “Up until a week ago, I’d suggest @37signals’ Campfire to anyone. Now, I’m telling everyone I know to avoid it.” People were upset, and they were right to be. When we mess up, we deserve to be the punching bag.
Of course, all companies experience episodes like this. How they handle the situation is what counts. I’m not talking about fixing the problem—you have to fix it; that’s a given. I’m talking about how you communicate with your customers, how you accept responsibility, and how you make things right. That’s what people remember.
I’ve been a customer of companies that don’t know how to respond to a crisis. These outfits don’t own up to the problem. They hedge, they tiptoe, they get their PR departments to issue abstract nonapology apologies.
Here’s one of the worst: “We apologize for any inconvenience we may have caused.” If ever there was a nonapology apology, this is it. And just about every company uses it. I Googled the phrase We apologize for any inconvenience. It came up 41 million times.
Let’s break this statement down. We apologize… Come on—when you really mean it, you say, “I’m sorry.” You don’t say, “I apologize.” If you spill hot coffee on someone, you say, “I am so, so sorry.” “I apologize” is renting the problem. “I’m sorry” is owning it. Now, to the second part of the nonapology: …for any inconvenience we may have caused . What a cop-out. For any? How about for all of it? May have caused? Don’t say maybe—say yes. Own it.
We’re very careful about how we explain downtime and other glitches. We don’t beat around the bush. We don’t try to hedge. We don’t pass the blame to a vendor or another party. When our customers are affected, it’s on us.
And we’ve discovered that the more honest we are, the better our customers feel. They just want to know what’s going on and what happened. They want to be informed. Wouldn’t anyone? You don’t win any prizes for confusing people—especially during a crisis. The clearer and more honest you can be, the better off you’ll be. It’s hard to admit mistakes, but you really don’t have a choice.
So here’s what we did when Campfire went down. First, we posted regular updates on the status page of our company’s website. We let people know we were working on the problem. As we figured things out, we shared the results. And if we still didn’t understand something, we admitted as much. That’s OK with us. What isn’t OK is leaving people in the dark. Everyone’s afraid of the dark when their data are involved.
We also took to Twitter. My business partner David Heinemeier Hansson responded to more than 100 tweets from customers. “We’re battling demons on all fronts and losing. It’s pathetic, I know,” David tweeted to one customer. “We’re spending the goodwill we’ve built from years of reliable service like it’s going out of style.” “So sorry for the disruption,” he wrote to another. “You can only say duh! so many times before people just think you’re annoying. We’re way past that,” he wrote.
We responded to every complaint and took the blame every time—even when people went overboard and launched into personal attacks. There was no fighting back, no attempt to save face. We messed up, we knew it, and we let every customer know that we knew it.
And our customers responded with enormous goodwill. “37signals has been giving a free lesson in customer service and honesty the past few weeks,” one customer tweeted. “Way to go on being awesome and communicative to your customers,” said another. Such expressions of support were really heartwarming—and evidence of how honesty, openness, and personal attention to a difficult situation can turn the darkest moment into one of the brightest.
We decided to give every Campfire customer a free month of service. We were down for only a few hours, total, but the downtime was spread out over multiple days. Besides, we didn’t earn our customers’ trust in December, so we didn’t earn their money, either. We have thousands of paying Campfire customers, so this wasn’t a cheap or easy decision. But it was the right thing to do.
Finally, once we figured out what went wrong and took steps to make sure it wouldn’t happen again, we wrote a full post on our product blog detailing exactly what had happened. We started with a general overview that could be understood by everyone. Being in the software business doesn’t give you license to speak in code. Yes, some of our customers are technically gifted. But most of them aren’t, so speaking in tech jargon can cause even more confusion.
That said, we also delved into the technical details for those who care about those kinds of things. And we added a link to the announcement inside Campfire, so all our customers would see it. You can read the product blog post at productblog.37signals.com/products/2010/12/campfire-outage-explanation-and-service-credits.html.
We learned a lot during that awful week. People don’t judge you on the basis of your mistakes—they judge you on the manner in which you own up to them. In my experience, most companies do a terrible job of taking blame. They lob press releases. Or they apologize for the inconvenience. Resist that temptation and say you’re sorry like you’re apologizing to a friend. Be good—and your customers will be good right back to you.