Presto: Distributed sql query engine
Distributed sql query engine
Published on: Mar 4, 2016
Transcripts - Presto: Distributed sql query engine
Problem to solve
Huge production of data.
As data is growing enormously to the point of peta bytes
, querying the database has become a big issue.
So we should be able to run more interactive queries and get
results faster .
Presto is a open source distributed sql query engine.
For running queries against of all sizes ranging from
gigabytes to petabytes .
It supports ANSI SQL ,including complex
queries,aggresgations,joins and window functions .
It is implemented in java.
Presto: I can query
Client sends sql to presto coordinator.
Coordinator parses ,analyzes and plans the query execution.
The scheduler wires together the execution pipeline ,assigns
work to nodes closest to data and monitors the progress.
The client pulls the data from output stage which in turn pulls
data from underlying stages.
Hive/Mapreduce Execution model
Hive translates queries into multiple stage of mapreduce
tasks and execute them one after the other.
Each task reads input from disk and writes intermediate
output back to disk.
Presto engine does not use Mapreduce.
It employs a custom query and execution engine with
operators designed to support sql semantics.
Processing is in memory and pipelined across the network
between stages which avoids unnecessary I/O and
associated latency overhead.
Pipelined execution model runs multiple stages at once and
streams data from one stage to next as it becomes available
which reduces end-to-end latency
Presto dynamically compiles certain portions of query plan to
byte code which lets JVM optimize and generate native
Presto was designed with a simple storage abstraction that
makes its easy to provide sql query capability against
disparate data sources.
Connectors only need to provide interfaces for fetching meta
data, getting data locations and accessing data itself.
Size limitation on the join tables and cardinality of unique
Lacks the ability to write output back to tables. Currently
query results are streamed to client.
Presto developers claim:
Presto is 10x better than hive/Mapreduce in terms of cpu
efficiency and latency for most queries.
Supports ANSI sql, including joins, left/right outer
joins,subqueries,most of the common aggregate and scalar
functions, including approximate distinct counts,