Here’s the situation: Your amazing developer team has rocked your world with caching and other optimizations to get data from your client’s Rails server over to the users’ browsers. Your monitoring shows requests that were taking a whole 80-90 seconds to serve are now only taking 1500ms (and all that time is actually spent transferring megabytes of html to the browser, actual data retrieval is ~10ms). These numbers come from a recent project I did for a client with production data. Now, to tackle the last piece of speeding it up… Chrome browser is taking MINUTES (6 minutes, actually) to fully build and render the page after it’s received the data. 100% unacceptable. Time to drop in some asynchronous data transferral and a specialized Javascript library to keep things performant on the client side!
My last two posts have been about dramatically speeding up how you serve large, complicated data sets to your users. As described above, serving the data was now quick, but rendering the data in the browser took a full 6 minutes; I timed it, repeatedly. For reference, this was done on a current generation MBP, so the CPU is no slouch (i7 4770HQ @ 2.2ghz). The issue here is that multiple MB of HTML were being served up in a report to the browser as a DataTable. I detailed the caching process in my earlier post, but the gist is that each <tr>...</tr>
worth of data is saved in a solr cache for each individual row of the report. These rows are styled, with each individual <td>...</td>
also having styling. That adds a significant amount of processing time to display data to the user.
Clusterize solves this exact problem. Check out the demo on their page to see it in action. The basic mechanism is that it takes an array of data (as table rows, each row is one element) and only renders the ones you can see at any given time. Absolutely perfect for our needs, since no matter how big a user’s screen is, they’ll only be displaying a tiny fraction, 0.1% absolute max, of the report at a given time. Perfect for our needs.
Setting up our app to use clusterize will involve the following steps:
Grab the two following files and throw them in your css and javascripts directories, respectively:
https://github.com/NeXTs/Clusterize.js/blob/master/clusterize.min.js
https://github.com/NeXTs/Clusterize.js/blob/master/clusterize.css
Don’t forget to add them to your precompile list in initializers/assets.rb
if needed.
This model exists solely to retrieve the prerendered contract data as an array from solr given a solr search on Contract
.
# app/models/admin/reports/contracts.rb
class Admin::Reports::Contracts
def contract_data(solr_search)
contract_rows = []
solr_search.hits.each do |hit|
contract_rows << hit.stored(:mar_partial)
end
contract_rows
end
end
# config/routes.rb
namespace :admin do
namespace :reports do
resources :contracts, only: [:index]
end
end
# app/controllers/admin/reports/contracts_controller.rb
class Admin::Reports::ContractsController < Admin::BaseController
def index
search = DealJacket.solr_search do
with(:contract_status, params[:contract_status]) if params[:contract_status].present? #If no filter, include all
paginate(per_page: Contract.count)
end
render json: Admin::Reports::Contracts.new.contract_data(search)
end
end
This simply takes a search param to scope down contracts (if provided) and renders the json data out. Try visiting the route in your browser, if you see a screen full of raw json text, it’s working as planned.
# app/assets/javascripts/contracts_report.js.coffee
build_clusterize = ->
return new Clusterize({
rows: window.contract_data,
scrollId: 'scrollArea',
contentId: 'contentArea'
});
get_json = (url) ->
$.getJSON url
$ ->
$.getJSON(
"/admin/reports/contracts",
(data) ->
window.contract_data = data["contracts"]
$.clusterize = build_clusterize(window.contract_data)
)
Coffee/JS is definitely my week point, so I’m sure this code can be improved, but it works great. What this does is the following:
Reports::ContractsController
JSON["contracts"]
window
for later usage= content_for :footer_js do
=javascript_include_tag "asset_registers"
=javascript_include_tag "clusterize.min"
=stylesheet_link_tag "clusterize", media: "all"
.clusterize
table
thead
tr
th Contract ID
...
th Balance
#scrollArea.clusterize-scroll
table
tbody#contentArea.clusterize-content
tr.clusterize-no-data
td Loading data...
The main points here are the following:
tbody
with #contentArea.clusterize-content
, this tells Clusterize where to manage the displayed rowsAll together, this loads the page with static table info (that “Loading Data…” row), and pulls your table rows into a new Clusterize object. As I mentioned, before clusterize, start to finish on a fast computer it took at least 6 minutes (that’s 360 seconds) to load and render the data so that it was usable. Now, it’s done consistently under 2 seconds, every single time. Hooking this all together made displaying a backend optimized report 180x faster.
I’ll call that a good accomplishment for the day.