By basic on

As things stand today, Drupal.org's mirror network is an essential part of the Drupal.org infrastructure. The ftp.drupal.org infrastructure hosts millions of files, serving everything from Drupal Core to contributed modules and themes, but it's beginning to show its age.

Our current FTP mirrors (co-located, in Oregon, Illinois, and New York) have been behaving erratically: projects have been failing to sync to the mirrors, being deleted before update, and sometimes disappearing from the mirrors for hours or days at a time. Even when working properly, the replication from the primary to additional mirrors can take as much as 45 minutes.

Compounding these issues is the fact that we do not have robust control or access to the existing architecture when problems arise.

So we've taken a step back to ask:

How can we deliver these files in a more reliable way?
On the modern web, the key elements of file delivery are:

  • High availability
  • Peering capacity designed for global delivery
  • Fast replication
  • HTTPS/TLS support

A Content Delivery Network is the answer to these problems, which is why we're evaluating a CDN to replace the ftp.drupal.org infrastructure.

But wait - does this mean the ftp:// protocol will no longer work?
Yes. The FTP protocol is aging as well...

  • In the month of October 2014, ftp:// had 96 unique visitors. Of those 96 unique visitors, only 33 of them made over 10 requests.
  • The ftp pathing differs from http, making the experience of using ftp:// confusing and inconsistent.
  • Replacing the ftp:// protocol with http will enable us to secure Drupal.org with HTTPS across all domains.

How you can help
We need users to help us test MaxCDN as an alternative for file delivery. You can track the issue here, and help us by testing the CDN based downloads. Please report back your findings (good or bad) and let us know if there are any showstoppers.

To test, add this line to your /etc/hosts file:

~$ sudo vim /etc/hosts
198.232.124.192 ftp.drupal.org

And continue using ftp.drupal.org as you normally would through Drupal.org project pages, drush dl, etc.

Comments

opdavies’s picture

opdavies commented

To test, add this line to your /etc/hosts file:

On Windows systems, this is located at %SystemRoot%\system32\drivers\etc\hosts.

mgifford’s picture

mgifford commented

I applied the changes to my host file.  Verified using ping that they were going to the new IP. Then downloaded the D8 Beta3, used drush to download a couple modules (drush dl devel) and also used git to get updates on Core.

I tested this locally on my mac, but also on an Ubuntu instance within a Vagrant box I use. All seems pretty smooth.

Happy to see the transition to HTTPS for file downloads.