How Much Would You Pay to Get Your Website Back in Service?
$300? If you pay, will you have to pay again? We’re not talking about a website contractor, we’re talking about an extortionist, and your website is under a DDoS attack.
We’re fans of the website and people behind Meetup.com. No doubt, this has been a tough weekend for Meetup. Since Thursday of last week, they faced a massive attack on their servers — a DDoS attack, which is a barrage of traffic intended to make service unavailable. They had many hours of downtime over several days, a first for the company in 12 years of growing the world’s largest network of local community groups.
Meetup’s CEO Scott Heiferman, explained the issue on the company’s Blog and through a variety of other types of crisis communication. He also outlines his reasons for refusing to pay $300 to get the attacks to stop, and we applaud his position. By documenting and sharing all information related to the attack, Scott and his team help every business learn and prepare for like attacks.
Read the below for a complete rundown from Scott:
While the site was down, the Meetup community was not. There were over 60,000 Meetups during the outage period- people meeting up about what’s important in their lives- and saw an incredible outpouring of support.
We’ve been fighting hard since the attacks began.
A little background: We spend millions of dollars every year keeping the Meetup website and apps secure, stable, and reliable. At Meetup HQ we have an amazing team of systems experts who build and manage our secure data centers — they are on-call 24/7 and have been very successful at making Meetup reliable year after year.
We were prepared for most DDoS attacks, but the nature of these attacks is changing (example here).
Here’s what happened. On Thursday morning, I received this email:
Date: Thu, Feb 27, 2014 at 10:26 AM
Subject: DDoS attack, warningA competitor asked me to perform a DDoS attack on your website. I can stop the attack for $300 USD. Let me know if you are interested in my offer.
Simultaneously, the attack began, our servers were overwhelmed with traffic, and our services went down.
We got to work mitigating the attack, but we remained unavailable for nearly 24 hours. Service was restored Friday at 9.30am EST, but it took many hours for the changes we implemented to defend against the attack to be distributed across the Internet. Many folks did not see us come back up before we were hit again.
On Saturday at 4 pm EST, we received another severe DDoS attack. By midnight EST, the engineering team implemented a new solution, and Meetup’s website and apps were widely accessible again.
On Sunday, at 8:09 pm EST, another strong attack began again, taking Meetup down for a third time. We spent the past several days taking every step to ensure the site and apps are available. While we’re confident that we’re taking all the necessary steps to protect against the threat, it’s possible that we’ll face outages in the days ahead.
The natural question I know many of you will ask is why didn’t we pay, especially since the amount of money demanded was ridiculously small ($300 USD).
We chose not to pay because:
1. We made a decision not to negotiate with criminals.
2. The extortion dollar amount suggests this to be the work of amateurs, but the attack is sophisticated. We believe this lowball amount is a trick to see if we are the kind of target who would pay. We believe if we pay, the criminals would simply demand much more.
3. Payment could make us (and all well-meaning organizations like us) a target for further extortion demands as word spreads in the criminal world.
4. We are confident we can protect Meetup from this aggressive attack, even if it will take time.
Please know that while we will not pay the criminals, YOU CAN COUNT ON MEETUP to be stable and reliable soon. We’ll continue to work diligently to restore the site and the apps, to bring back all features, and to minimize the effects of the service outages.
This is an attack on everyone who believes that people are powerful together. We live in a world where criminals can make extortion threats against an organization like ours and temporarily frustrate millions of people. But we also live in a world where organizers start new Meetup Groups, members show up, people start talking, and communities form. Our platform is built around a simple idea — that if Meetup helps people to find the others, we will all be more powerful and will create the kind of world we want to live in together.
For the latest on the service outage, check this blog post, Twitter, or Facebook. Sincere thanks for your patience and support.
—-
Scott Heiferman,
Co-Founder and CEO, Meetup
A Leader leads all of the time, but in times of crisis, a leader’s true character is displayed (and is mirrored by employees). Scott let his employees, customers and community know that Meetup is made of stronger stuff. Reportedly, such ransom demands, especially when no user-confidential information is involved, are not uncommon but are not frequently made public.
According to an article from insurancejournal.com “A report this month by security firm Prolexic said attacks were up 32 percent in 2013, and a December study by the cyber-security-focused Ponemon Institute showed them now responsible for 18 percent of outages at U.S.-based data centers from just 2 percent in 2010. The average cost of a single outage was $630,000, it said.”
While the ROI on fighting may be questionable to some, Meetup did gain a broad audience who applaud the company’s stance. Scott and his team also kept their community updated every step of the way. From Meetup’s FAQs
FAQs about Meetup’s Recent Service Outage
We know our recent service outage has impacted many of you and made it tough for your Meetup Groups to actually Meetup. Please know that we’re listening to your questions and concerns and that YOU CAN COUNT ON MEETUP to be stable and reliable, to restore all features back to normal, and to minimize the effects of the service outages.
1) Is my data secure?
Organizer and Member data is secure, including credit card information. No data has been accessed or stolen. For more detail on the denial of service (DDoS) attack, read this account from Scott Heiferman, Meetup’s Co-Founder and CEO.
2) Will I get credit for the time I wasn’t able to use Meetup?
As a result of this attack on Meetup, the site and apps were largely unavailable for nearly five days. To show our gratitude for your patience during this outage, all Organizers were credited with an additional 7 days.
This extension was added to Organizers’ current Organizer Dues cycle and was automatically applied to Organizers’ accounts. We processed the change, and we emailed Organizers to alert them. It’s reflected on the Organizers’ account page, and there’s nothing further you need to do.
If you are a Meetup Organizer whose Group faced an extraordinary challenge and there is something else we can do to help you out, please email us at [email protected]. We know this was frustrating for you (for us, too!). We also know in many cases, the outage created real problems for your Meetups. It’s critical to us that Meetup Organizers know how much we value your efforts to build community.
3) Why isn’t email working? When will messages be delivered to my members? Will all emails eventually be sent?
After several days with significant periods of downtime, we have accumulated a significant backlog of emails (everything from system generated emails like Event reminders and new Meetup Group announcements, to the emails Organizers send to their Groups). Due to the steps we’ve taken to stop the attacks, it’s also taking longer than usual to deliver email for some domains, notably hotmail.com and roadrunner. Most other domains are working well.
Restoring email functionality to normal is one of our top priorities right now. We may remove some older emails from the queue for a small number of domains, but most email will make it through. Some of that email is no longer relevant and we want to make sure email you are sending from the site now is received in a timely manner.
We expect all email issues to be resolved soon, now that we’re back online.
4) Why couldn’t you send me the email addresses for my Group members so that I could communicate with my Groups during the outage? Can you send them to me now?
We understand that it would have been helpful to have an alternate means to contact Members during the service outage. However, our privacy policy prohibits us from sharing Members’ email addresses without their explicit permission. We take Member privacy seriously, so we can’t bend the rules, even in an extraordinary case like a service outage.
Site downtime is a very rare occurrence (with the past few days being the exception, not the rule).
5) Will new Group announcement emails go out for Groups that should have been announced during the blackout?
Yes, all new Group announcement emails will be sent out. Due to the email backlog issues (see above), it will take time.
6) I was supposed to renew my account and pay my Organizer Dues during the outage. Will I lose Organizer privileges to my Meetup Group?
If you were an Organizer who was supposed to renew your account and pay your Organizer Dues during the site outage, we have automatically extended the renewal period, giving you more time to act. No one will lose ownership or administrative access to their Meetup Group for failure to renew during the site outage. You can find your new deadline date on your Account page.
As further detailed in TechCrunch: according to the company, this was an example of the increasingly common NTP-based DDoS attack. Explains CloudFlare CEO Matthew Prince, who stepped in to help Meetup get back online, NTP-style attacks are a newer choice among criminals when it comes to producing the DDoS flood that can crash websites, and they’re far more powerful, too.
In a nutshell, DDoS attacks attempt to crash servers, usually web servers, by sending a barrage of traffic to overwhelm the receiving ports. The servers crash under the load, taking websites and services down with them. In the past, such as with the high-profile Spamhaus DDoS attacks last year, the previous favorite vector for criminals instigating these attacks was DNS – that is, they would amplify their attacks using the DNS infrastructure.
But now, attackers are beginning to exploit flaws in other, older Internet protocols that were not originally secured particularly well. In Meetup’s case, the attackers use the NTP – or Network Time Protocol – which is a protocol that’s use to sync time clocks between multiple servers.
“What should we do now?” “What should we say?”