Fast Server, Slow Browser
My last two posts have been about dramatically speeding up how you serve large, complicated data sets to your users. As described above, serving the data was now quick, but rendering the data in the browser took a full 6 minutes; I timed it, repeatedly. For reference, this was done on a current generation MBP, so the CPU is no slouch (i7 4770HQ @ 2.2ghz). The issue here is that multiple MB of HTML were being served up in a report to the browser as a DataTable. I detailed the caching process in my earlier post, but the gist is that each
<tr>...</tr> worth of data is saved in a solr cache for each individual row of the report. These rows are styled, with each individual
<td>...</td> also having styling. That adds a significant amount of processing time to display data to the user.
Clusterize solves this exact problem. Check out the demo on their page to see it in action. The basic mechanism is that it takes an array of data (as table rows, each row is one element) and only renders the ones you can see at any given time. Absolutely perfect for our needs, since no matter how big a user’s screen is, they’ll only be displaying a tiny fraction, 0.1% absolute max, of the report at a given time. Perfect for our needs.
Setting up our app to use clusterize will involve the following steps:
- Put the clusterize js and css assets in the app
- Create a model to fetch the data from solr
- Set up a data serving route and controller to return a JSON array of info
- Point clusterize at our endpoint
- Apply needed clusterize classes and ids to the table to make it display properly
Don’t forget to add them to your precompile list in
initializers/assets.rb if needed.
Solr data retrieval
This model exists solely to retrieve the prerendered contract data as an array from solr given a solr search on
# app/models/admin/reports/contracts.rb class Admin::Reports::Contracts def contract_data(solr_search) contract_rows =  solr_search.hits.each do |hit| contract_rows << hit.stored(:mar_partial) end contract_rows end end
The Controller and Routes to Serve JSON
# config/routes.rb namespace :admin do namespace :reports do resources :contracts, only: [:index] end end
# app/controllers/admin/reports/contracts_controller.rb class Admin::Reports::ContractsController < Admin::BaseController def index search = DealJacket.solr_search do with(:contract_status, params[:contract_status]) if params[:contract_status].present? #If no filter, include all paginate(per_page: Contract.count) end render json: Admin::Reports::Contracts.new.contract_data(search) end end
This simply takes a search param to scope down contracts (if provided) and renders the json data out. Try visiting the route in your browser, if you see a screen full of raw json text, it’s working as planned.
Coffeescript and Views
Coffee/JS is definitely my week point, so I’m sure this code can be improved, but it works great. What this does is the following:
- Asynchronously gets json data from our
- Builds Clusterize with the provided array found in the
- Due to weird CoffeeScript variable scoping, data is attached to
windowfor later usage
The main points here are the following:
- Encompass your whole table in the .clusterize div
- Give your table rows their own table (makes the header row act as a sticky header)
#contentArea.clusterize-content, this tells Clusterize where to manage the displayed rows
- Provide a data loading message to placate users while the JSON is fetched
All together, this loads the page with static table info (that “Loading Data…” row), and pulls your table rows into a new Clusterize object. As I mentioned, before clusterize, start to finish on a fast computer it took at least 6 minutes (that’s 360 seconds) to load and render the data so that it was usable. Now, it’s done consistently under 2 seconds, every single time. Hooking this all together made displaying a backend optimized report 180x faster.
I’ll call that a good accomplishment for the day.