Virtual Distributed File System: Alluxio
Alluxio (2018) is the world’s first memory speed virtual distributed storage. Alluxio is an open source project with a community of over 700 contributors from over 150 organizations. In the big data stack, it is a layer between applications such as Apache Spark (2018) and Apache MapReduce (Apache Hadoop 2018a) and data stores such as Amazon S3 (Amazon 2018) (Simple Storage System) and Apache HDFS (Apache Hadoop 2018b) (Hadoop Distributed File System).
Alluxio presents a set of disparate data stores as a single file system, greatly reducing the complexity of storage APIs (Application Programming Interface), locations, and semantics exposed to applications. For example, an object storage in the cloud and an on-premise distributed file system both appear to applications as subtrees in the Alluxio file system namespace.
Alluxio is designed with a memory centric architecture, enabling applications to leverage memory speed I/O by simply using Alluxio. Because...
- Alluxio (2018) Alluxio. https://alluxio.org
- Alluxio, Baidu (2016) Baidu queries data 30 times faster with Alluxio. http://alluxio-com-site-prod.s3.amazonaws.com/resource/media/Baidu-Case-Study.pdf
- Alluxio, Barclays (2016) Making the impossible possible w/ Alluxio. http://alluxio-com-site-prod.s3.amazonaws.com/resource/media/Making_the_Impossible_Possible_w_Alluxio.pdf
- Alluxio, Qunar (2017) Qunar performs real-time data analytics up to 300x faster with Alluxio. http://alluxio-com-site-prod.s3.amazonaws.com/resource/media/Qunar_Performs_Real-Time_Data_Analytics_up_to_300x_Faster_with_Alluxio.pdf
- Amazon (2018) Amazon S3. https://aws.amazon.com/s3
- Apache Hadoop (2018a) Apache Hadoop MapReduce. http://hadoop.apache.org
- Apache Hadoop (2018b) Apache Hadoop distributed file system. http://hadoop.apache.org
- Apache Spark (2018) Apache Spark. https://spark.apache.org
- Apache Zookeeper (2018) Apache Zookeeper. https://zookeeper.apache.org
- Google (2018) Protocol buffers. https://developers.google.com/protocol-buffers
- Liu S, Dawei S (2017) PerceptIn robotics get a performance boost from Alluxio distributed storage. http://thenewstack.io/powering-robotics-clouds-alluxio