05 Feb

Cloudflare issues in South Korea

Over the last months I have tried Cloudflare’s service for one of my websites. Cloudflare is a “Content Delivery Network” or CDN in short. At first I was very excited about it and championed it’s usage to the other executives at our business because of all the advantages a CDN is supposed to give you. After 3 months of using it with their Professional plan (200$ per month) we decided to turn it off as it hasn’t delivered the benefits we would have expected.

The setup

  • The site is hosted on a single DigitalOcean droplet.
  • It is forcing users to use HTTPS. This is done via a .htaccess redirect
  • The homepage has about 60 resources
  • Total file sizeis about 1.2 MB

The website didn’t experience grave performance problems, but it wasn’t exactly fast either. Additionally, we were serving high quality images to our visitors that sometimes took a long time to load.

Speed differences with Cloudflare

From my location in Cambodia, the time for loading the site went down significantly to around one second to load every piece of content.

But for some reasons, the people in South Korea kept reporting to me that the website sometimes loads very slow or even can’t load some images. A few weeks later I was back in South Korea and could expierience all the problems by myself. Sometimes the pages were taking 5-10 seconds to load for no obvious reasons.

In one case I checked together with a customer through skype and looked at his screen. The website was so slow, it would take the 30 seconds before the homepage appeared. And then another full minute until the CSS and most of the images were loaded so that it actually looked similar to what it should look like.

Over the next weeks I could identify three separate problems.

Problem 1: The “Pro Plan” doesn’t work in South Korea

I contacted cloudflare about this issue and got an answer after some back and forth.

Unfortunately I have bad news for you. Being on the Pro plan, almost all KR traffic will be routed through LAX, so you will see increased latency for those requests.

We discuss this here: https://blog.cloudflare.com/bandwidth-costs-around-the-world/

and include a link to ask KT to peer with us in Seoul: http://www.kt.com/eng/etc/contact.jsp

Summary: Traffic in South Korea is much more expensive than in any other region. That’s why Cloudflare reroutes all traffic from the Free and Pro plans through Los Angeles, USA. This should -in the long run- increase costs for Korean ISPs and pressure them and the government to play ball with international companies. To allow peering and speed up the internet around the world.

If you want to have your content be served from their node in Incheon, South Korea, you will need to upgrade to the Business plan which costs 200$ instead of 20$.

 

Problem 2: Korea Telecom has routing problems

What I found when running some networking tools on different connections, was this.

ping_result_problems_cut

On the left side is the result when pinging our webserver directly. On the right side is the result when pinging cloudflares server. I was using a regular home wifi and a LG U+ connection.

After some more back and forth with Cloudflare’s support team, I was told that this is an issue of Korea Telecom and therefore I should talk with them. (I did not talk to them)

Problem 3: Slow TTFB

We opted to upgrade to their Business Plan. At first it looked like it was showing some improvements. But after a while, I can now say it didn’t speed up our website loading times overall. This is most likely caused by Korean ISP’s routing problems, but nevertheless it isn’t helping us so that’s why I decided to stop using cloudflare at all for now. And the increased latency for AJAX calls was becoming a serious issue for us.

Here I’ll break down the different parts of the connection that take place when a page loads.

  1. DNS resolution and connection time
    This was much faster than before. In exact numbers, this metric went down from 150ms to about 20ms.
  2. Download Time
    The time it took for the user to download the HTML page stayed roughly the same. It always stayed between 30ms to 60ms. The larger content such as images started loading significantly faster. A 500 kb image would load in 40ms instead of a very variable 80ms to 400ms.
  3. Server Response Time (TTFB)
    This one went UP from about 250ms to a whopping 900ms.

Here are the metrics collected by Google Analytics. While it might not be the best tool to monitor the site speed, it’s the only one I have for now and it confirms that I’m not the only one with these results.

analytics_ttfb_whitespace

Does the Server Response time (TTFB) matter?

Cloudflare says on an often shared post that you should stop worrying about TTFB.

https://blog.cloudflare.com/ttfb-time-to-first-byte-considered-meaningles/

Sure, it’s not always the most important thing. If your images are takig 3 seconds to download, you have bigger problems than 300ms connection times. But saying it is meaningless is just wrong too.

Our website has a shopping cart feature that is using AJAX. So whenever a user clicks on the “Add to Cart” button we are doing two requests.

  1. Send a request to the server that adds the item to the shopping cart
  2. Reload the basket from the server and display it to the user

These two requests have to happen in sequence, so their times add up. Without cloudflare, it takes around 250ms to send the first request and 250ms to reload the basket. That results in a 500ms time to complete the user action. With cloudflare’s increased TTFB each request takes 900ms to complete. So the total time required to add an item to the cart will take 1.8 seconds.

So what’s the solution?

I started optimizing things one by one instead of using the Cloudflare umbrealla solution.

  1. Cloudflare is now disabled
  2. Now all our user uploaded images are uploaded to an Amazon S3 bucket and then served through their Cloudfront service.
  3. I have yet to find a good solution for our static content like application specific images, Javascript and Stylesheets.

Uploaded images are now served faster than ever before. We do have a slower download time for application specific static content. But once those are cached on the customer’s browser, everything behaves much quicker.

18 Jan

Setting up Cloudflare for a DigitalOcean droplet

This is a quick recollection of the steps it took to set up Cloudflare for a website that is using Apache on a DigitalOcean Droplet.

What is a CDN and why do I need one?

If you’ve never heard of a CDN or have but are not sure what it is, here is a small explanation. A CDN is a global network of servers that helps distribute your website content to different places across the globe. Most websites start off on a single server. If you set up your own website, you will usually start with a hosting provider and a domain registrar. At the domain registrar you enter the address  of the hosting provider’s DNS servers and/or IP addresses. Sometimes you can have both services from a single company. This is so that when somebody types www.yourwebsite.com the request is correctly routed to your webserver.

In this setup, every visitor to your website will be served by your single server that you rented from your hosting provider. If your hosting provider is in Texas, and your website visitor is in Tokyo, Japan, it will naturally create a little bit of a delay in the communication that happens behind the scenes. The bigger the website you’re serving, the longer it will take to load everything between Texas and Tokyo. Now if you wanted to speed up the loading times for your Japanese visitors, you could set up another server in or close to Japan. But that would be a lot of work and it wouldn’t help your other visitors who might be in South Africa or in Norway.

A Content Distribution network will sit between your hosting provider and your visitors. Because it is a Network, it’s servers are distributed and it will be able to serve your content to much more places, and much faster than you could by setting up a server in every country. Here is an image that I took from Cloudflare’s website and it explains the setup very well I think.

cloudflare_overview

How does the setup process work?

If you’re running a simple website with some information and pictures on it that are not using a secured connection, it’s fairly easy.

  1. Create a cloudflare account
  2. Get your new nameservers from cloudflare
  3. Enter your nameservers in your domain registrar‘s administration panel.
  4. Log in to cloudflare and set up your domains and subdomains
  5. Be sure you recreated all the DNS entries from your old domain registrar. For example, if you are using Google Apps for work, you also have to set up your MX records in your cloudflare account.

What if I use a secure connection over HTTPS?

This was the case for the site I was moving. Cloudflare has a couple of options in their management. They are mostly concerning what certificate to use (cloudflare vs. your own) and what part of the connection should be encrypted with with certificate. Because there are two parts in the connection.

 

Cloudflare's SSL options

Cloudflare’s SSL options

IMPORTANT: If you are using the option Full or Full (strict), cloudflare will require some time to issue a new certificate. I’m not 100% sure what happens behind the scenes, but in our case for the first 24 hours, our website was showing the scary “Your connection is not private” message in all browsers. During that time, the SSL settings in the cloudflare management screen were saying “Issuing new certificate” or something similar. It was eventually resolved automatically, but this is a very important factor if you’re migrating a live website.