YACOP

YACOP(Yet Another Container Orchestration Platform) is a container orchestration platform for YARN.

Features

Roadmap

• Simple short-lived Docker container scheduling
• Retreive meta data of Docker container
• Enable Docker container access to secured Hadoop services
• Read only volume mount
• Task re-run when failure
• Application master fault tolerance
• Client and AM RPC
• Dynamically scale tasks
• Remove appname submission to AM
• Use MD5 hash to check if image has already been loaded
• Support "YARN" containers

Setting up and running YACOP

1. A LinuxContainerExecutor enabled hadoop cluster
2. Enable YARN Registry for the cluster
3. Prepare your Docker image. It should have the same user (and uid) in it which is also used when run YACOP
4. Clone, build and run YACOP with the Docker image

git clone https://github.com/intel-hadoop/yacop.git
cd yacop
mvn clean package -DskipTests

docker pull centos:latest
cat <<'EOF' > artifact.json
{
    'name': 'hello-docker',
    'cpus': 2.0,
    'mem': 32.0,
    'instances': 2,
    'cmd': 'echo \"hello docker\"',
    'engine': {
        'type': 'DOCKER',
        'image': 'centos',
        'localImage': true
    }
}
EOF

yarn jar target/yacop-1.0-SNAPSHOT.jar run -configFile artifact.json -jar target/yacop-1.0-SNAPSHOT.jar

YACOP client

To list Docker container status of an applicatoin

yarn jar target/yacop-1.0-SNAPSHOT.jar resolve -applicationId <applicationId>

To list all applications history

yarn jar target/yacop-1.0-SNAPSHOT.jar registry