golang: bigslice

Bigslice is a system for fast, large-scale, serverless data processing using Go.

Bigslice provides an API that lets users express their computation with a handful of familiar data transformation primitives such as map, filter, reduce, and join. When the program is run, Bigslice creates an ad hoc cluster on a cloud computing provider and transparently distributes the computation among the nodes in the cluster.

Bigslice is similar to data processing systems like Apache Spark and FlumeJava, but with different aims:

  • Bigslice is built for Go. Bigslice is used as an ordinary Go package, users use their existing Go code, and Bigslice binaries are compiled like ordinary Go binaries.
  • Bigslice is serverless. Requiring nothing more than cloud credentials, Bigslice will have you processing large datasets in no time, without the use of any other external infrastructure.
  • Bigslice is simple and transparent. Bigslice programs are regular Go programs, providing users with a familiar environment and tools. A Bigslice program can be run on a single node like any other program, but it is also capable of transparently distributing itself across an ad hoc cluster, managed entirely by the program itself.
Publikováno v Go