Bigslice is a system for fast, large-scale, serverless data processing using Go.
Bigslice provides an API that lets users express their computation with a handful of familiar data transformation primitives such as map, filter, reduce, and join. When the program is run, Bigslice creates an ad hoc cluster on a cloud computing provider and transparently distributes the computation among the nodes in the cluster.
Bigslice is similar to data processing systems like Apache Spark and FlumeJava, but with different aims:
- Bigslice is built for Go. Bigslice is used as an ordinary Go package, users use their existing Go code, and Bigslice binaries are compiled like ordinary Go binaries.
- Bigslice is serverless. Requiring nothing more than cloud credentials, Bigslice will have you processing large datasets in no time, without the use of any other external infrastructure.
- Bigslice is simple and transparent. Bigslice programs are regular Go programs, providing users with a familiar environment and tools. A Bigslice program can be run on a single node like any other program, but it is also capable of transparently distributing itself across an ad hoc cluster, managed entirely by the program itself.