HDFS>> VeloxMR>> 返回
项目作者： DICL

项目描述 ：
Data processing component of the Velox Big Data Framework (VBDF)
高级语言： C++
项目地址: git://github.com/DICL/VeloxMR.git
创建时间： 2016-03-05T03:06:27Z
项目社区：https://github.com/DICL/VeloxMR
开源协议：Apache License 2.0
下载


Build Status
Slack room
ZenHub

Analytics

VeloxMR {#mainpage}

VeloxMR is a MapReduce framework implemented in C++ on the top of VeloxDF

Key features of current VeloxMR include:

  • There’s no central directory service such as HDFS NameNode.
  • The decentralized file metadata are managed by SQLite in each data node.
  • It is fault tolerant. (Each block is replicated in successor and decessor by default)

USAGE

VeloxMR default launcher is not included in this repository, if you want to use it you can find it [here][eclipsed].

The reason not to include the launcher inside the package is to let the user to choose any launcher or service managers such as:

  • systemd/init.d
  • puppet/chef/salt

Once the system is up and running, you can interact with VeloxDFS with the following commands:

  1.  $ dfs put|get|cat|ls|rm|format|pget|update|append

COMPILING & INSTALLING

Detailed information can be found in the wiki of this repository

Compiling requirements

  • C++14 support, this is: GCC >= 4.9, Clang >= 3.4, or ICC >= 16.0.
  • Boost library >= 1.53.
  • Sqlite3 library.
  • GNU Autotools (Autoconf, Automake, Libtool).
  • Unittest++ [optional].

For single user installation for developers

  1. $ mkdir -p local_eclipse/{tmp,sandbox}                 # Create a sandbox directories
  2. $ cd local_eclipse                                     # enter in the directory
  3. $ git clone git@github.com:DICL/VeloxDFS.git           # Clone the project from github
  4. $ cd VeloxDFS
  5. $ sh autogen.sh                                        # Generate configure script 
  6. $ cd ../tmp                                            # Go to building folder
  7. $ sh ../VeloxDFS/configure --prefix=`pwd`/../sandbox # Check requirements and generate the Makefile
  9. # If you get a boost error go the FAQ section of the README
  11. ### This last command will be needed whenever you want to recompile the source
  12. $ make [-j#] install                                   # Compile & install add -j flag to speed up

Now edit in your ~/.bashrc or ~/.profile:

  1. export PATH="/home/*..PATH/To/eclipse/..*/sandbox/bin":$PATH
  2. export LIBRARY_PATH="/home/*..PATH/To/eclipse/..*/sandbox/lib"
  3. export C_INCLUDE_PATH="/home/*..PATH/To/eclipse/..*/sandbox/include"

Default settings for VELOXDFS 

  1. "log" : {
  2.   "type" : "LOG_LOCAL6"
  3.   "name" : "ECLIPSE"
  4.   "mask" : "DEBUG"
  5. },
  7. "cache" : {
  8.   "numbin"      : 100,
  9.   "size"        : 200000,
  10.   "concurrency" : 1
  11. },
  13. "filesystem" : {
  14.   "block"    : 137438953,
  15.   "buffer"   : 512,
  16.   "replica"  : 1
  17. }

FAQ

  • Question : configure stops with errors related to boost library.
  • Answer : It probably means that you do not have boost library installed in
    the default location, in such case you should specify the boost library location.
    1. sh ../VeloxDFS/configure --prefix ~/sandbox --with-boost=/usr/local --with-boost-libdir=/usr/local/lib
    In this example we assume that the boost headers are in /usr/local/include while the library files
    are inside /usr/local/lib.

AUTHORS