We just sent notice we are terminating service for 8chan. There comes a time when enough is enough. But this isn't the end. We need to have a broader conversation about addressing the root causes of hate online.
рдлреЙрд▓реЛ рдХрд░рд╛
рдлреЙрд▓реЛ рдХрд░рд╛ eastdakota рд╡рд░ рдХреНрд▓рд┐рдХ рдХрд░рд╛
Matthew Prince 
@eastdakota
A little bit geek, wonk, and nerd. Repeat entrepreneur, recovering lawyer, and former ski instructor. Co-founder & CEO of Cloudflare (NYSE: NET).
рдмрд╛рдпреЛ рдЕрдиреБрд╡рд╛рджрд┐рдд рдХрд░рд╛
Matthew Prince ЁЯМе рдпрд╛рдВрдЪреА рдЯреНрд╡рд┐рдЯреНрд╕
We just blocked Kiwifarms. The threats on the site escalated enough in the last 48 hours that, in spite of proactively working with law enforcement, it became enough of an imminent emergency we could no longer wait for them to act. Details of our decision:
ThereтАЩs a lot of buzz right now about a тАЬmassive DDoS attackтАЭ targeting the US, complete with scary-looking graphs (see Tweet below). While it makes for a good headline in these already dramatic times, itтАЩs not accurate. The reality is far more boring. 1/X
рдЯреНрд╡рд┐рдЯ рд╡рд░ рднрд╛рд╖реНрдп рдХрд░рд╛
This DDoS attack is serious. It has taken down Instagram, Facebook, T-Mobile, Verizon, and Twitch.... 2020 is something else.
One of the smartest decisions we made at was recognizing that the primary purpose of our blog was attracting employees, not attracting customers.
Oh Paul, you know better. I had to borrow money from my mom to pay my taxes when we were starting Cloudflare. But I certainly came from a relatively privileged background, and so did the AirBnB founders. ItтАЩs hard to take risks if you donтАЩt have a safety net. #bereal
Mistakes happen. The root problem was we didnтАЩt have systems in place to keep them from causing a widespread issue. ThatтАЩs a problem of leadership that I am more responsible for than the engineer who made the typo.
Details on how we caused an 23 min outage for~50% of 's network today. The root cause was a typo in a router configuration on our private backbone. We've applied safeguards to ensure a mistake like this will not cause problems in the future.
Nothing we're seeing related to the Facebook services outage suggests it was an attack. Most likely explanation is that the company's Internet routes (BGP) were withdrawn by mistake during maintenance. #hugops
WeтАЩve made the determination that #Log4J is so bad weтАЩre going to try and roll out at least some protection for all customers by default, even free customers who do not have our WAF. Working on how to do that safely now.
Just sent the last employment offer of 2021. I still personally send all the offers out because nothing is more important than hiring. Some numbers: We received more than 200,000 applications. We extended 1,455 offers. And we had a 92% offer acceptance rate.
Except T-Mobile, which is having a bad day almost certainly entirely of their own teamтАЩs making. So, please, #hugops. And donтАЩt worry, this is one thing that does not need to get added to the list of craziness that has been 2020. 8/8
GIF
рдкреНрд░рддрд┐рдореЗрдЪреЗ рд╡рд░реНрдгрди рд╡рд╛рдЪрд╛
ALT
Why weтАЩre switching to ARM-based servers in one image. Both servers running the same workload at the same performance.
If the team needs help getting DNS back up, stands ready to help restore services temporarily so you can get back on your feet.
Proud of our whole team for creating 1.1.1.1, the InternetтАЩs fastest, privacy-first DNS resolver. ItтАЩs тАЩs first consumer product. And, if youтАЩre wondering whose dopey idea it was to launch on April Fools (and Easter), look no further than me.
рдЯреНрд╡рд┐рдЯ рд╡рд░ рднрд╛рд╖реНрдп рдХрд░рд╛
Announcing 1.1.1.1: the fastest, privacy-first consumer DNS service - cfl.re/2GoNuYP
When I read this story by about how and misfired to cause a huge bill I felt terrible. I reached out to and proposed we split Troy's cost. Scott immediately agreed. Great to support our mutual customers!
It starts with T-Mobile. They were making some changes to their network configurations today. Unfortunately, it went badly. The result has been for around the last 6 hours a series of cascading failures for their users, impacting both their voice and data networks. 2/X
So now people are looking around for an explanation and they stumble across sites like the Arbor Networks attack map. It looks terrifying today! Thing is, it always looks terrifying. ItтАЩs a marketing gimmick put up to sell DDoS mitigation services so thatтАЩs not surprising. 4/X
Traffic to Signal exceeded Telegram in Ukraine for the first time.
The collective hallucination that this is about тАЬtaxesтАЭ rather than finding somewhere talented junior engineers can afford their own apartment and senior engineering managers can afford a nice houseтАФbecause sensible housing policiesтАФwill be what kills SF.
At , we understand the Russian cyber attack capabilities and stand prepared to defend our clients against any cyber retaliation that results from global sanctions.
Earliest evidence weтАЩve found so far of #Log4J exploit is 2021-12-01 04:36:50 UTC. That suggests it was in the wild at least 9 days before publicly disclosed. However, donтАЩt see evidence of mass exploitation until after public disclosure.
We are aware that may have been compromised. There is no evidence that Cloudflare has been compromised. Okta is merely┬аan identity┬аprovider for Cloudflare.┬аThankfully, we have multiple layers of security beyond Okta, and would never consider them to be a standalone option.
Dear , when your team promises to work with companies in exchange for moving significant jobs to Portugal and shows laws to support those promises; then your bureaucrats refuse to follow those laws and promises тАФ is that ok? I feel lied to. Cc:
Lots of reports of Russian censorship of Western media. We are seeing evidence of that. But, generally, consumption of Western media in Russia is up more than 3x in the last month тАФ in spite of censorship. #truthfindsaway
From тАЩs vantage point, we can see a number of things that show there is no massive DDoS attack. First, traffic from WARP to supposedly impacted services is normal and has no increase in errors. 5/X
Just sent out the last employment offers of 2022. Received roughly 400,000 applications, up 49% over 2021. Of those, we interviewed 15,805 candidates to ultimately hire 1,418. 37% of the hires were designated fully remote, up from 14% in 2021. (1/2)
Wow! 's 1.1.1.1 (one.one.one.one) has now passed handling more than 1 trillion requests per day. Not bad for a project we launched on April Fools Day a little over four years ago.
Seeing a marked increase in cyberattacks this evening. Combined with the deeply disturbing headlines, fear the world just turned up the crazy dial another notch. WeтАЩre ready online at . ButтАж worried for the world.
That caused a lot of T-Mobile users to complain on Twitter and other forums that they werenтАЩt able to reach popular services. Then services like Down Detector scraps Twitter and report those services as being offline. 3/X
Second, there is no spike in traffic to any of the major Internet Exchanges, which you do see during actual DDoS attacks and definitely would during one allegedly this disruptive. 6/X
In 1996, I wrote my college thesis on why the Internet was a fad.
Fun fact: AWS hasnтАЩt reduced the price of S3 since December 2016. ЁЯдФ
We are resetting the credentials of any employees whoтАЩve changed their passwords in the last 4 months, out of abundance of caution. WeтАЩve confirmed no compromise. Okta is one layer of security. Given they may have an issue weтАЩre evaluating alternatives for that layer.
When started we didn't want to build a DNS service, so we reached out to Dyn & UltraDNS about becoming a customer. Both blew us off because we were "too small." So we built our own. Good lesson about serving all customers, no matter how small. w3techs.com/technologies/h
This is the sad story about what happened to my friend and тАЩs brilliant third cofounder Lee Holloway (). WeтАЩd never have been able to pull off what we did without him on the team. I miss him every day. wired.com/story/lee-holl
Finally, our team know the network operators at nearly all the other major Internet services and platforms and none of them are reporting anything anomalous. 7/X
Spike in use of Signal (messaging app) in Ukraine in the last 24 hours. Smaller but significant increase in the use of Telegram as well.
We had an issue that impacted some portions of the network. It appears that a router in Atlanta had an error that caused bad routes across our backbone. That resulted in misrouted traffic to PoPs that connect to our backbone. 1/2
As a precaution, weтАЩve removed all customer cryptographic material from servers in Ukraine. We continue to serve traffic there for Ukrainian users, for now, via our #Keyless technology.
Never. Trust. Amazon.
Seeing 's BGP announcements getting published again. Likely means service is on a path to getting restored.
Why we terminated Daily Stormer.
Here's what went wrong on the Internet earlier today causing and several other networks to be unreachable for many users. It's time for providers like to be held responsible for not filtering BGP routes and implementing RPKI.
Were thinking of opening 's Asian office in Singapore. After all the love tonight, thinking maybe Hong Kong instead.
The teams at and should be incredibly embarrassed at their failings this morning which impacted and other large chunks of the Internet. ItтАЩs absurd BGP is so fragile. ItтАЩs more absurd Verizon would blindly accept routes without basic filters.
My sister, , made a custom liner for by suit jacket to celebrate $NET today. She tells me it glows in the dark!!
WeтАЩve seen reports of service outages across the Internet. Confirmed тАЩs services all operational. No uptick in attacks. We are seeing local drops in traffic from some upstream providers. Not yet clear if theyтАЩre related or not. All indications: not a Cloudflare issue.
Aware of major issues impacting us network wide. Team is working on getting to the bottom of whatтАЩs going on. Will continue to update.
The first five employees тАФ┬а, , , , and I тАФ┬аofficially started on January 4, 2010. We were above a nail salon in Palo Alto at 542 Emerson. Our first task was to assemble the BBQ for the deck. It's been quite a decade.
Every ~8 years thereтАЩs a 10x improvement in how computing is provisioned. Bare metal > VMs > Managed VMs/containers. WeтАЩre due for the next step-function improvement and at weтАЩre convinced itтАЩll be built around a technology called Isolates:
#Brexit should serve as a cautionary tale for anyone in the audience sitting back and saying: "There's no way Donald Trump will win."
Wow! 's 1.1.1.1 just crossed handling 300 billion queries per day. Still well shy of the trillion+ that Google's 8.8.8.8 handles, but growing fast!
Completely unacceptable. IтАЩve ordered the account be restored. Called on team to investigate why an automated system took such draconian action without any warning. It may be youтАЩre doing something that breaks plan limits, but we need to have more nuanced solutions. Apologies.
AWSтАЩs bandwidth charges are egregious. Their wholesale cost in AWS-East is likely less than $200/Gbps/mo. That equates to a 10,000%+ markup. #nevertrustamazon
рдЯреНрд╡рд┐рдЯ рд╡рд░ рднрд╛рд╖реНрдп рдХрд░рд╛
1Gbps of sustained outbound transfer on aws is about $21,000/month in us-east-1.
that's it. that's the tweet.
Pretty sure a major DDoS attack would be amazing for us given weтАЩre one of the only companies that could stop it.
Good lesson: when you hear hoof beats, think ЁЯРО not ЁЯжУ.
This Halloween I went as unshowered tech CEO procrastinating from writing his Q3 earnings script.
Incredible that stopped the largest DDoS attack in history and it was just another day at the office. I wasn't even aware of the scale until I read this post.
Years ago, I remember reading that Google's 8.8.8.8 handled 2 trillion queries per day and being blown away. Today, just 3 1/2 years after launch, 1.1.1.1 is a quarter of the way to that same milestone. And >15% of its queries are encrypted! #progress one.one.one.one
All threat research groups have cool names. Most of them are full of BS. is planning to launch a threat research group that's no full of BS. But still need a cool name. Any suggestions? If we choose your suggestion I'll make sure you get lots of branded swag.
Spanish gov't has blocked access to Catalan referendum site. 2 years ago this would have been major story. Now it's just another day online.
Proud of the role played in ensuring the Internet stayed online in 2020. We stand ready for whatever 2021 brings (but, for the record, will be totally cool if itтАЩs less eventful).
Thrilled to partner with NVIDIA to bring AI to the edge!! Workers is the largest, fastest, most used edge computing working. With NVIDIA's hardware running at our edge we open a whole new class of applications for developers. #DeveloperWeek
We isolated the Atlanta router and shut down our backbone, routing traffic across transit providers instead. There was some congestion that caused slow performance on some links as the logging caught up. Everything is restored now and we're looking into the root cause. 2/2
ItтАЩs a disgrace that closed yesterday while extending the season at most their other resorts where they control most of the lodging/dining. They closed with over 100" of base and multiple feet of fresh snow forecast over the next 10 days.
Excited to work with to help customers understand what they're spending on the cloud and how they can optimize those costs using Workers.
Many people know about the wall of lava lamps in San Francisco uses to generate random numbers. Fewer know about the wall of double pendulums in London we use here. #entropy
Writing the S-1 was a fun process for me, perhaps a less fun for the (terrific) bankers we worked with. The number of times I said тАЬMaybe should just start over with a blank pageтАЭ inspired the celebratory cake they made for us.
I could do this in a different, but equally effective, way.
At , we need to manage a lot of services running on a lot of servers distributed across more than 200 cities worldwide. Here's an inside view on one of the tools we use to do that: Nomad.
Ha! AWS required us to remove the term тАЬmulti-cloudтАЭ from our materials at one of their conferences. Someone hasnтАЩt learned the anti-trust lesson from Microsoft yet. They will.
Tomorrow begins Platform Week at . We'll feature all the ways we're extending Cloudflare Workers to be the best serverless development platform. R2's beta will open up. ButтАж┬аthat's not even close to the only exciting announcement for the week! #staytuned
Just finished sending the final batch of job offers for 2020. It's been incredible to see the caliber of the people applying to join our team over the last year. The seeds of what we will become tomorrow are being planted today. (And they're mighty impressive seeds.)
So far the Ukrainian Internet continues to function. Seeing a 50% increase in traffic, day-over-day, likely as people are scouring the news online as they wake up.
All the pieces are now in place for a solo developer to on their own build a $1B company using Workers. Will it be you?
рдЯреНрд╡рд┐рдЯ рд╡рд░ рднрд╛рд╖реНрдп рдХрд░рд╛
We are developing an example feature-complete SaaS application that will be built entirely on the Cloudflare stack. cfl.re/3oGt6Gc #FullStackWeek
Cloudflare's backbone (blue/orange lines) almost circumnavigates the earth. (Mumbai to Marseille in the works, at which point it will.) And our virtual backbone (green lines) powers Argo and connects nearly every major city on earth with secure, encrypted tunnels. #betterinternet
Well, that was fast!! IтАЩm doing the dance of joy! Great news for our mutual customers. And the next step toward the inevitable end of cloud egress.
Additional technical details on the outage experienced earlier today will be posted here. We are committed to being fully transparent:
CloudflareтАЩs unofficial motto: Go fast and fix things.тДв
тАЬAmazon? A threat to us? Never! TheyтАЩre a terrific customer.тАЭ тАФ Senior UPS Executive to me 18 months ago.
рдЯреНрд╡рд┐рдЯ рд╡рд░ рднрд╛рд╖реНрдп рдХрд░рд╛
Amazon just bought 20k delivery vans. ThatтАЩs 20% the size of UPS twitter.com/davehclark/staтАж
To add insult to injury, they chose to close today in spite of the fact that the Park City SchoolsтАЩ Spring Break started today. Dear , think of the children! Hard to imagine what could be more insulting to the community you pretend to respect. Cc:
As it happens, and I are staying in the same hotel in Baltimore. There appears to have been some sort of mixup: my тАЬroomтАЭ has 10 rooms and is larger than my apartment in San Francisco. Ms. Perry, I hope my rightfully issued Deluxe Queen proves inspiring.
On vacation in Rwanda. Something inexplicably incredible about seeing this in a rain forest on the side of a volcano and just a few feet away. He was, to say the leastтАж not small. ЁЯжН
Incidentally: is garbage. It merely looks at mentions on Twitter of people saying something is down. That includes incorrect speculation. And it even amplifies that speculation recursively. ItтАЩs reliably wrong, by design.
May seem big, but because of тАЩs architecture we can handle this and even bigger тАФ even for free customers тАФ without breaking a sweat.
рдЯреНрд╡рд┐рдЯ рд╡рд░ рднрд╛рд╖реНрдп рдХрд░рд╛
Cloudflare blocks an almost 2 Tbps multi-vector DDoS attack. cfl.re/3C9WOrD
So fascinating for long-time customer McLaren to host me behind the scenes at the Belgian Grand Prix today. Good luck to and tomorrow at (rainy) Spa!
Massive spike in CPU usage caused primary and backup systems to fall over. Impacted all services. No evidence yet attack related. Shut down service responsible for CPU spike and traffic back to normal levels. Digging in to root cause.
Would have create 100+ jobs in Portugal in 12 months since beginning of COVID. From basically zero. Good, technical, high-paying jobs. And would keep hiring at that pace. Bureaucrats shooting their country in the foot.
Want to see a real privacy nightmare? Type your friendsтАЩ (or exтАЩs or enemiesтАЩ) phone number in the rewards terminal the next time you check out at CVS or Walgreens. Instantly get a set of coupons detailing their (very intimate) shopping habits. ЁЯджЁЯП╗тАНтЩВя╕П
We're extending our offer to provide for Teams (Access & Gateway) for free for 6 months to companies of all sizes. If we can assist people getting work done securely and efficiently while they work from home, then it's our duty to help.
Appear to have mitigated the issue causing the outage. Traffic restored. Working now to restore all services globally. More details to come as we have them.
Lisbon is so great! Weather, food, people. And тАЩs team here will be over 300 people strong by the end of 2022. WeтАЩre hiring!! cloudflare.com/careers
Love that the only country where Fastly is fastest is now Liberia. So, if the majority of your customers are there, keep using them! OtherwiseтАж┬аmay want to look to the performance leader. And there is literally no country left where Akamai is the fastest. blog.cloudflare.com/speed-week-202
More: appears that the router in Atlanta announced bad routes (effectively a route leak). Only impacted our backbone. Not all of our PoPs are connected to our backbone, so some would not have seen an issue. Appears to have impacted about 50% of our traffic for a bit over 20 min.