Fast Server, Slow Browser

Here’s the situation: Your amazing developer team has rocked your world with caching and other optimizations to get data from your client’s Rails server over to the users’ browsers. Your monitoring shows requests that were taking a whole 80-90 seconds to serve are now only taking 1500ms (and all that time is actually spent transferring megabytes of html to the browser, actual data retrieval is ~10ms). These numbers come from a recent project I did for a client with production data. Now, to tackle the last piece of speeding it up… Chrome browser is taking MINUTES (6 minutes, actually) to fully build and render the page after it’s received the data. 100% unacceptable. Time to drop in some asynchronous data transferral and a specialized Javascript library to keep things performant on the client side!

Diving In

My last two posts have been about dramatically speeding up how you serve large, complicated data sets to your users. As described above, serving the data was now quick, but rendering the data in the browser took a full 6 minutes; I timed it, repeatedly. For reference, this was done on a current generation MBP, so the CPU is no slouch (i7 4770HQ @ 2.2ghz). The issue here is that multiple MB of HTML were being served up in a report to the browser as a DataTable. I detailed the caching process in my earlier post, but the gist is that each <tr>...</tr> worth of data is saved in a solr cache for each individual row of the report. These rows are styled, with each individual <td>...</td> also having styling. That adds a significant amount of processing time to display data to the user.

Clusterize.js

Clusterize solves this exact problem. Check out the demo on their page to see it in action. The basic mechanism is that it takes an array of data (as table rows, each row is one element) and only renders the ones you can see at any given time. Absolutely perfect for our needs, since no matter how big a user’s screen is, they’ll only be displaying a tiny fraction, 0.1% absolute max, of the report at a given time. Perfect for our needs.

The Solution

Setting up our app to use clusterize will involve the following steps:

• Put the clusterize js and css assets in the app
• Create a model to fetch the data from solr
• Set up a data serving route and controller to return a JSON array of info
• Point clusterize at our endpoint
• Apply needed clusterize classes and ids to the table to make it display properly

Assets

Grab the two following files and throw them in your css and javascripts directories, respectively:

https://github.com/NeXTs/Clusterize.js/blob/master/clusterize.min.js
https://github.com/NeXTs/Clusterize.js/blob/master/clusterize.css

Don’t forget to add them to your precompile list in initializers/assets.rb if needed.

Solr data retrieval

This model exists solely to retrieve the prerendered contract data as an array from solr given a solr search on Contract.

# app/models/admin/reports/contracts.rb
class Admin::Reports::Contracts
  def contract_data(solr_search)
   contract_rows = []
   solr_search.hits.each do |hit|
     contract_rows << hit.stored(:mar_partial)
   end
   contract_rows
  end
end

The Controller and Routes to Serve JSON

# config/routes.rb

namespace :admin do
  namespace :reports do
    resources :contracts, only: [:index]
  end
end

# app/controllers/admin/reports/contracts_controller.rb

class Admin::Reports::ContractsController < Admin::BaseController
  def index
    search = DealJacket.solr_search do
      with(:contract_status, params[:contract_status]) if params[:contract_status].present? #If no filter, include all
      paginate(per_page: Contract.count)
    end

    render json: Admin::Reports::Contracts.new.contract_data(search)
  end
end

This simply takes a search param to scope down contracts (if provided) and renders the json data out. Try visiting the route in your browser, if you see a screen full of raw json text, it’s working as planned.

Coffeescript and Views

# app/assets/javascripts/contracts_report.js.coffee

build_clusterize = ->
  return new Clusterize({
        rows: window.contract_data,
        scrollId: 'scrollArea',
        contentId: 'contentArea'
  });

get_json = (url) ->
  $.getJSON url

$ ->
  $.getJSON(
    "/admin/reports/contracts",
    (data) ->
      window.contract_data = data["contracts"]
      $.clusterize = build_clusterize(window.contract_data)
  )

Coffee/JS is definitely my week point, so I’m sure this code can be improved, but it works great. What this does is the following:

• Asynchronously gets json data from our Reports::ContractsController
• Builds Clusterize with the provided array found in the JSON["contracts"]
• Due to weird CoffeeScript variable scoping, data is attached to window for later usage

= content_for :footer_js do
  =javascript_include_tag "asset_registers"
  =javascript_include_tag "clusterize.min"
  =stylesheet_link_tag "clusterize", media: "all"

.clusterize
  table
    thead
      tr
        th Contract ID
        ...
        th Balance
  #scrollArea.clusterize-scroll
    table
      tbody#contentArea.clusterize-content
        tr.clusterize-no-data
          td Loading data...

The main points here are the following:

• Encompass your whole table in the .clusterize div
• Give your table rows their own table (makes the header row act as a sticky header)
• tag tbody with #contentArea.clusterize-content, this tells Clusterize where to manage the displayed rows
• Provide a data loading message to placate users while the JSON is fetched

End Result

All together, this loads the page with static table info (that “Loading Data…” row), and pulls your table rows into a new Clusterize object. As I mentioned, before clusterize, start to finish on a fast computer it took at least 6 minutes (that’s 360 seconds) to load and render the data so that it was usable. Now, it’s done consistently under 2 seconds, every single time. Hooking this all together made displaying a backend optimized report 180x faster.

I’ll call that a good accomplishment for the day.