Dumber is faster

Jeremy Zawodny provides evidence that old-style sequential searching can be faster than using SQL.

He had to run queries involving over 100 million records and found that using conventional SQL took him several days, whereas performing minimal sorting and outputting a stream of tens of millions of records sequentially, to be analysed by code, took only 8 hours to ahcieve the same result.

He calls it “linear” data processing – “reading sequentially through a batch of files rather than trying to stuff everything in a database and doing big join queries.”

Leave a Reply

Your email address will not be published. Required fields are marked *