Rails Asset Pipeline, CDNs and Serving Cross-Domain Fonts

February 23, 2012      Permalink     Tags:  rails, asset-pipeline, cdn

Recently I was working on a project for a client. My task was straightforward: “Can you make my site faster?” There was only one right answer.

My client’s web site was running on Rails 3.0.1 and using full page caching. The site made extensive use of JavaScript and high-res images and testing with the YSlow plugin for FireFox confirmed my suspicion that most of the bottlenecks could be ameliorated by combining and minifying external JavaScript and CSS files and by serving content from a CDN.

I had been eyeing the Asset Pipeline since it was released as part of Rails 3.1 and this was the perfect excuse to jump in. If you’re not familiar with the Asset Pipeline, you should check it out; explaining it in detail, however, is beyond the scope of this article. Essentially it makes your static assets first-class citizens in your Rails application, right alongside your Ruby code. Leveraging libraries such as Sprockets, Sass and UglifyJS, it makes writing JavaScript and CSS less painful and gives you the ability to create dependency trees and modularize your scripts and styles with imports. The best part, however, is that it can precompile your static assets, compressing all of your scripts and styles into a few core files, minifying them, and even gzipping them. Additionally, it uses file fingerprints to name the compiled files for better caching. Awesome.

First Roadblock: Namespace Chaos

As it goes with Rails, the Asset Pipeline adheres to a few conventions, one of which is dumping all static assets into the same namespace. In order words, once compiled, a stylesheet and an image requested with <%= stylesheet_link_tag "screen.css" %> and <%= image_tag "icon.png" %> will be served from /assets/screen.css and /assets/icon.png, respectively, instead of from /stylesheets/screen.css and /images/icon.png, as they would have been in previous versions of Rails.

This has the advantage of allowing you to reference static assets across libraries without needing to fuss with sub-folders. By default, the Asset Pipeline will bring into the asset search path anything under app/assets/, lib/assets/ and vendor/assets/. So you can require a JavaScript file located at vendor/assets/javascripts/jquery-ui.js from within app/assets/javascripts/application.js without having to worry about the path:

//=require jquery-ui.js

I imagine for most this works fine. However, I see a few problems with it. First, there is potential for files to overlap across libraries. For example, it is conceivable that two libraries might have a stylesheet called screen.css – which one would be imported? Second, it makes it difficult to organize and serve assets from multiple Content Delivery Networks, as your files are lumped together in one namespace. (Why multiple CDNs? More on that in a bit.) Finally, it pollutes the public/assets/ directory with any uncompiled files that your third-party libraries may contain – even READMEs, tests and documentation. This became especially apparent when I dropped the Google Closure library into vendor/assets/. When I ran rake assets:precompile, I ended up with this:

$ ls public/static/
AUTHORS
LICENSE
README
all_tests-2b7a9ce7ac70114656773b6a5eb47ffc.html
all_tests.html
application-9302184a59033b8f84b5c0c84d58beaf.js
application-9302184a59033b8f84b5c0c84d58beaf.js.gz
application-c57e646675665776d02ceda5774b69dc.css
application-c57e646675665776d02ceda5774b69dc.css.gz
application.css
application.css.gz
application.js
application.js.gz
arrow-left_20x14-102858d7051ca3cb338ec444dd1e41d5.png
arrow-left_20x14.png
beta_59x33-bbe810b7f16c710832057168e248cf9e.png
beta_59x33.png
bg-gradient-d1045408f33a832534950ec06b195fe3.jpg
bg-gradient.jpg
bullet_green-0995accbacf9bc3e11f805378cb085bc.png
bullet_green.png
bullet_orange-cf2e4090499d4dee019244b27993737b.png
bullet_orange.png
closure
# ... and 86 more. You get the idea.

As you can see, any non-JS or CSS files found in the assets search paths are dumped into public/static/, including the AUTHORS and README files from Google Closure. How was I supposed to maintain that? As I upload my static assets to a CDN like most, what would happen when I wanted to remove or upgrade a third-party library? Rather than find out, I decided to pass on dealing with that mess.

So I set out to fight the framework and namespace my static assets. There’s probably a really clever way to do this, but simply nesting my assets within sub-folders accomplished what I was hoping for. So, app/assets/fonts/ became app/therealretouch/assets/fonts/, vendor/assets/closure-library/ became vendor/assets/third_party/closure-library/, and so on.

When I compiled my static assets again, public/assets/ looked like this:

$ ls public/assets/
closure-library  images       jquery-ui-1.8.10  manifest.yml
fonts            javascripts  jqzoom            stylesheets

Sanity had been restored.

Second Roadblock: Relative Image Paths in Third-Party Library Stylesheets

Moving assets into logical namespaces made managing assets easier in the long run, but it created a new problem: relative image paths in third-party libraries were broken. Fortunately, this only affected jQuery UI, as the Google Closure library used absolute paths for images (on Google’s CDN). After considering my options, I decided to search-and-replace relative path strings with the Rails asset helper in the jQuery UI stylesheet like so: <%= asset_path 'jquery-ui/images/ui-bg_flat_75_ffffff_40x100.png' %>. Again, there may be a more elegant solution, but this worked for my purposes.

Third Roadblock: JavaScript File Size

The Asset Pipeline will by default compile two files: app/assets/application.js and app/assets/application.css. You can instruct it to precompile additional files like so:

config.assets.precompile += ['admin/admin.js', 'admin/admin.css', 'admin/editor.comp.js']

I had previously built a rich text editor with the Google Closure library for use in administrative back-ends. With closure-library now integrated into the Asset Pipeline, I decided to add the editor to the array of assets to precompile. Weighing in at 1,365 lines of JavaScript – after compiling it with calcdeps.py (hey, Closure isn’t exactly the most concise JavaScript library out there…) – the editor proved more than Sprockets could handle, as the latter died with stack overflow errors when trying to compile it.

Since breaking up my Closure editor into multiple files wasn’t an option, I removed it from the precompile array and simply wrote a Capistrano deploy task to copy it to the assets directory and add an entry for it to the manifest.yml file (so that the Asset Pipeline would recognize it) after the other assets were precompiled. It goes something like this:

run "cp #{release_path}/app/assets/therealretouch/javascripts/admin/editor.comp.js #{release_path}/public/assets/javascripts/admin/editor.comp.js"
run %{echo "javascripts/admin/editor.comp.js: javascripts/admin/editor.comp.js" >> #{release_path}/public/assets/manifest.yml}

With the my static assets organized into sane, modular namespaces, and with my rich text editor in place, I set about uploading everything to a CDN.

Fourth Roadblock: Firefox and Serving Fonts across Domains

My client required a CDN to host all JavaScript, CSS, image and font files. As she was already relying on Amazon S3 for storing large order files, Amazon CloudFront seemed like a logical choice for a CDN.

It turns out that Firefox will not load a font file served from one domain accessed within another domain. So, a font hosted at cdn.therealretouch.com would be inaccessible from www.therealretouch.com when the website were viewed in Firefox.

The workaround for this is to use the Access-Control-Allow-Origin header to specify the domain which would be accessing the resource (or, more flexibly, *, to permit any domain). Unfortunately, the Amazon S3 API does not allow one to set the Access-Control-Allow-Origin header. As a result, I needed another CDN solution that would allow setting this header when uploading fonts.

As the site was already being hosted with Rackspace, CloudFiles seemed a logical alternative. Thankfully CloudFiles has supported setting the Access-Control-Allow-Origin header via their API since May 2011.

Since my client was already tied to Amazon S3 for order files, I decided to use CloudFiles for serving fonts only and to serve all other static content from Amazon CloudFront.

Finish Line

Here is the Rake task for uploading the compiled assets to Amazon S3 and Rackspace CloudFiles according to the hybrid CDN solution described above:

require 'aws-sdk'
require 'cloudfiles'
require 'mime/types'

aws_interface = ::AWS::S3.new(
  :s3_endpoint => 's3.amazonaws.com',
  :access_key_id => AWS_KEY,
  :secret_access_key => AWS_SECRET
)
aws_bucket = aws_interface.buckets[AWS_BUCKET]

cloudfiles_connection = CloudFiles::Connection.new(
  :username => CF_USERNAME,
  :api_key => CF_API_KEY
)
cloudfiles_container = cloudfiles_connection.container(CF_CONTAINER)

namespace :cdn do
  namespace :upload do
    desc "Upload all static assets"
    task :all => [:fonts, :images, :javascripts, :stylesheets, :third_party] do
    end

    desc "Upload font files to CDN"
    task :fonts do
      fonts = get_files('public/assets/fonts/**/*')
      fonts.each do |filename|
        headers = { 'Access-Control-Allow-Origin' =>  '*' }
        write_cloudfiles_file(cloudfiles_container, filename, headers)
      end
    end

    desc "Upload image files to CDN"
    task :images do
      assets = get_files('public/assets/images/**/*')
      assets.each { |filename| write_aws_file(aws_bucket, filename) }
    end

    desc "Upload javascript files to CDN"
    task :javascripts do
      assets = get_files('public/assets/javascripts/**/*')
      assets.each { |filename| write_aws_file(aws_bucket, filename) }
    end

    desc "Upload stylesheet files to CDN"
    task :stylesheets do
      assets = get_files('public/assets/stylesheets/**/*')
      assets.each { |filename| write_aws_file(aws_bucket, filename) }
    end

    desc "Upload third party files to CDN"
    task :third_party do
      assets = get_files('public/assets/jquery-ui/**/*', 'public/assets/jqzoom/**/*')
      assets.each { |filename| write_aws_file(aws_bucket, filename) }
    end
  end # :upload
end # :cdn

def write_cloudfiles_file(container, filename, headers = {}, check_md5 = true)
  type = MIME::Types.type_for(filename)
  type = type.empty? ? get_font_mime_type(filename) : type.first.content_type
  headers['Content-Type'] = type unless type.nil?
  name = cdn_file_path(filename)
  puts "Writing to CloudFiles: #{name} (#{type})"
  object = container.object_exists?(name) ? container.object(name) : container.create_object(name, true)
  object.load_from_filename(filename, headers, check_md5)
end

def write_aws_file(bucket, filename, options = {})
  options[:acl] ||= :public_read
  options[:content_encoding] = 'gzip' if filename =~ /\.gz$/
  options[:content_type] ||= MIME::Types.type_for(filename.gsub(/\.gz$/, '')).first.content_type
  name = cdn_file_path(filename)
  puts "Writing to S3: #{name} (#{options[:content_type]})"
  object = bucket.objects[name]
  object.write(File.open(filename, 'rb').read, options)
end

def get_files(*dir_names)
  assets = Array.new.tap do |ary|
    dir_names.each do |dir|
      Dir[dir].each do |file|
        next if File.ftype(file) == 'directory'
        ary << file
      end
    end
  end
  assets
end

def cdn_file_path(path)
  path.gsub(%r{^public/}, '')
end

def get_font_mime_type(filename)
  extname = File.extname(filename)
  case extname
  when '.ttf' then 'application/x-font-ttf'
  when '.otf' then 'application/x-opentype'
  when '.eot' then 'application/vnd.ms-fontobject'
  when '.woff' then 'application/x-font-woff'
  else
    nil
  end
end

With assets encapsulated in logical namespaces (e.g., javascripts, images, fonts, etc.), additional assets marked for pre-compilation, large JavaScript files manually integrated into the Asset Pipeline, third-party relative paths resolved, and a hybrid CDN solution laid out for serving cross-domain fonts, all the pieces were in place. Here is the Capistrano task to bring it all together:

namespace :assets do
  desc "Compile assets for production"
  task :compile, :roles => :web do
    run "cd #{release_path} && RAILS_ENV=#{rails_env} bundle exec rake assets:precompile --trace"
    run "cp #{release_path}/app/assets/therealretouch/javascripts/admin/editor.comp.js #{release_path}/public/assets/javascripts/admin/editor.comp.js"
    run %{echo "javascripts/admin/editor.comp.js: javascripts/admin/editor.comp.js" >> #{release_path}/public/assets/manifest.yml}
    run "cd #{release_path} && RAILS_ENV=#{rails_env} bundle exec rake cdn:upload:all --trace"
  end
end

after 'deploy:update_code', 'assets:compile'

References:

  1. Asset Pipeline. Sep. 2011. Rails Guides. 31 Jan. 2012 <http://guides.rubyonrails.org/asset_pipeline.html>.
  2. Understanding the Asset Pipeline. Aug. 2011. Railscasts. 31 Jan. 2012 <http://railscasts.com/episodes/279-understanding-the-asset-pipeline>.
  3. Upgrading to Rails 3.1. Sep 2011. Railscasts. 31 Jan. 2012 <http://railscasts.com/episodes/282-upgrading-to-rails-3-1>.

I'm interested in full-stack Web development with Rails and JavaScript.

Find me on Twitter: @bryandragon