datamapper_rack_streamer – giving something back to the community

- Luke Metcalfe

As we developers from crealytics like the idea of open source technology and most of our projects rely on work, someone from “the community” has done before, it’s time to give something back now and then. Today, this is the datamapper_rack_streamer project, hosted at github.

Now, what’s it all about?

The datamapper_rack_streamer is a little application written in Ruby, which takes data from a database using the marvelous datamapper ORM. (That means, you don’t have to struggle with SQL any longer just for retrieving that 5 billion data records you always wanted to export!)

Trying to export a huge dataset via a webinterface with datamapper and a Sinatra/Rack combo, we ran into performance and memory problems. The problems arose from our little Sinatra application fetching all the requested data, building the csv file (still on the server side) and then sending the file to the client. This works for smaller datasets. It really doesn’t for huge ones.

The solution then was to change the approach and stream the data directly to the client as it’s read from the database. And TADA – here we are. That’s all, the datamapper_rack_streamer does. Nothing more, nothing less. It comes with the very liberal BSD license. (So you could take it and build a jumbo jet from it. If you like.)

We hope, someone else has use for our small application and we also hope to give some more stuff back to the community soon.


Luke is a Content Marketer at Crealytics

Leave a Reply

Your email address will not be published. Required fields are marked *