05 Feb

Cloudflare issues in South Korea

Over the last months I have tried Cloudflare’s service for one of my websites. Cloudflare is a “Content Delivery Network” or CDN in short. At first I was very excited about it and championed it’s usage to the other executives at our business because of all the advantages a CDN is supposed to give you. After 3 months of using it with their Professional plan (200$ per month) we decided to turn it off as it hasn’t delivered the benefits we would have expected.

The setup

  • The site is hosted on a single DigitalOcean droplet.
  • It is forcing users to use HTTPS. This is done via a .htaccess redirect
  • The homepage has about 60 resources
  • Total file sizeis about 1.2 MB

The website didn’t experience grave performance problems, but it wasn’t exactly fast either. Additionally, we were serving high quality images to our visitors that sometimes took a long time to load.

Speed differences with Cloudflare

From my location in Cambodia, the time for loading the site went down significantly to around one second to load every piece of content.

But for some reasons, the people in South Korea kept reporting to me that the website sometimes loads very slow or even can’t load some images. A few weeks later I was back in South Korea and could expierience all the problems by myself. Sometimes the pages were taking 5-10 seconds to load for no obvious reasons.

In one case I checked together with a customer through skype and looked at his screen. The website was so slow, it would take the 30 seconds before the homepage appeared. And then another full minute until the CSS and most of the images were loaded so that it actually looked similar to what it should look like.

Over the next weeks I could identify three separate problems.

Problem 1: The “Pro Plan” doesn’t work in South Korea

I contacted cloudflare about this issue and got an answer after some back and forth.

Unfortunately I have bad news for you. Being on the Pro plan, almost all KR traffic will be routed through LAX, so you will see increased latency for those requests.

We discuss this here: https://blog.cloudflare.com/bandwidth-costs-around-the-world/

and include a link to ask KT to peer with us in Seoul: http://www.kt.com/eng/etc/contact.jsp

Summary: Traffic in South Korea is much more expensive than in any other region. That’s why Cloudflare reroutes all traffic from the Free and Pro plans through Los Angeles, USA. This should -in the long run- increase costs for Korean ISPs and pressure them and the government to play ball with international companies. To allow peering and speed up the internet around the world.

If you want to have your content be served from their node in Incheon, South Korea, you will need to upgrade to the Business plan which costs 200$ instead of 20$.

 

Problem 2: Korea Telecom has routing problems

What I found when running some networking tools on different connections, was this.

ping_result_problems_cut

On the left side is the result when pinging our webserver directly. On the right side is the result when pinging cloudflares server. I was using a regular home wifi and a LG U+ connection.

After some more back and forth with Cloudflare’s support team, I was told that this is an issue of Korea Telecom and therefore I should talk with them. (I did not talk to them)

Problem 3: Slow TTFB

We opted to upgrade to their Business Plan. At first it looked like it was showing some improvements. But after a while, I can now say it didn’t speed up our website loading times overall. This is most likely caused by Korean ISP’s routing problems, but nevertheless it isn’t helping us so that’s why I decided to stop using cloudflare at all for now. And the increased latency for AJAX calls was becoming a serious issue for us.

Here I’ll break down the different parts of the connection that take place when a page loads.

  1. DNS resolution and connection time
    This was much faster than before. In exact numbers, this metric went down from 150ms to about 20ms.
  2. Download Time
    The time it took for the user to download the HTML page stayed roughly the same. It always stayed between 30ms to 60ms. The larger content such as images started loading significantly faster. A 500 kb image would load in 40ms instead of a very variable 80ms to 400ms.
  3. Server Response Time (TTFB)
    This one went UP from about 250ms to a whopping 900ms.

Here are the metrics collected by Google Analytics. While it might not be the best tool to monitor the site speed, it’s the only one I have for now and it confirms that I’m not the only one with these results.

analytics_ttfb_whitespace

Does the Server Response time (TTFB) matter?

Cloudflare says on an often shared post that you should stop worrying about TTFB.

https://blog.cloudflare.com/ttfb-time-to-first-byte-considered-meaningles/

Sure, it’s not always the most important thing. If your images are takig 3 seconds to download, you have bigger problems than 300ms connection times. But saying it is meaningless is just wrong too.

Our website has a shopping cart feature that is using AJAX. So whenever a user clicks on the “Add to Cart” button we are doing two requests.

  1. Send a request to the server that adds the item to the shopping cart
  2. Reload the basket from the server and display it to the user

These two requests have to happen in sequence, so their times add up. Without cloudflare, it takes around 250ms to send the first request and 250ms to reload the basket. That results in a 500ms time to complete the user action. With cloudflare’s increased TTFB each request takes 900ms to complete. So the total time required to add an item to the cart will take 1.8 seconds.

So what’s the solution?

I started optimizing things one by one instead of using the Cloudflare umbrealla solution.

  1. Cloudflare is now disabled
  2. Now all our user uploaded images are uploaded to an Amazon S3 bucket and then served through their Cloudfront service.
  3. I have yet to find a good solution for our static content like application specific images, Javascript and Stylesheets.

Uploaded images are now served faster than ever before. We do have a slower download time for application specific static content. But once those are cached on the customer’s browser, everything behaves much quicker.