Apache REEF™ - a stdlib for Big Data

Apache REEF™ (Retainable Evaluator Execution Framework) is a library for developing portable applications for cluster resource managers such as Apache Hadoop™ YARN or Apache Mesos™. Apache REEF drastically simplifies development of those resource managers through the following features:

Features

  • Centralized Control Flow: Apache REEF turns the chaos of a distributed application into events in a single machine, the Job Driver. Events include container allocation, Task launch, completion and failure. For failures, Apache REEF makes every effort of making the actual Exception thrown by the Task available to the Driver.
  • Task runtime: Apache REEF provides a Task runtime called Evaluator. Evaluators are instantiated in every container of a REEF application. Evaluators can keep data in memory in between Tasks, which enables efficient pipelines on REEF.
  • Support for multiple resource managers: Apache REEF applications are portable to any supported resource manager with minimal effort. Further, new resource managers are easy to support in REEF.
  • .NET and Java API: Apache REEF is the only API to write YARN or Mesos applications in .NET. Further, a single REEF application is free to mix and match Tasks written for .NET or Java.
  • Plugins: Apache REEF allows for plugins (called “Services”) to augment its feature set without adding bloat to the core. REEF includes many Services, such as a name-based communications between Tasks, MPI-inspired group communications (Broadcast, Reduce, Gather, …) and data ingress.

REEF @ Apache Big Data 2016

Latest Release

The latest release of Apache REEF is 0.15.0. Please go to the downloads page to find out more.