DuckDB benchmarked against Spark

You Don't Always Need A Sledgehammer

Introduction

Apache Spark has been the de facto open source data processing for fifteen years. It was invented to solve a major problem that traditional data warehousing was not built to solve - processing massive amounts of data horizontally at scale (Zach used Spark to process 2000 TBs per day at Netflix), whether in a structured or semi-structured for…