Skip to content

Latest commit

 

History

History
65 lines (55 loc) · 3.93 KB

README.md

File metadata and controls

65 lines (55 loc) · 3.93 KB
Raw

Crawling@Home

Help us build a billion-scale image-caption dataset by filtering Common Crawl with OpenAI CLIP

Google Colab

Open In Colab

  1. Change the value for YOUR_NICKNAME_FOR_THE_LEADERBOARD and make sure you are connected to a gpu runtime to maximize efficiency.
  2. Then just run all (Ctrl+F9) to install dependencies and start Crawling!

Other options

  • Open In Colab
    • If you want to run a cpu only worker (don't use a gpu runtime)
  • Open In Colab
    • If you want to run a gpu only worker (please use a gpu runtime)

Docker file

  1. Get the docker image using docker pull arkseal/cah-worker:hybrid-cpu
  2. Run docker image using docker run --name cahworker --shm-size=4G -d arkseal/cah-worker:hybrid-cpu
    • add -e NAME={nickname} to specify display name
      • Ex: docker run --name cahworker -e NAME=ARKseal --shm-size=4G -d arkseal/cah-worker:hybrid-cpu
You can use this one liner: docker pull arkseal/cah-worker:hybrid-cpu && docker run --name cahworker --shm-size=4G -d arkseal/cah-worker:hybrid-cpu
  • add -e NAME={nickname} to specify display name
    • Ex: docker pull arkseal/cah-worker:hybrid-cpu && docker run --name cahworker -e NAME=ARKseal --shm-size=4G -d arkseal/cah-worker:hybrid-cpu

Other options

  • Gpu enabled hybrid worker: docker pull arkseal/cah-worker:hybrid-gpu && docker run --name cahworker --gpus all --shm-size=4G -d arkseal/cah-worker:hybrid-gpu
    • add -e NAME={nickname} to specify display name
      • Ex: docker pull arkseal/cah-worker:hybrid-gpu && docker run --name cahworker --gpus all -e NAME=ARKseal --shm-size=4G -d arkseal/cah-worker:hybrid-gpu
    • This requries NVIDIA Container Toolkit on host device
  • Cpu only worker: docker pull arkseal/cah-worker:cpu && docker run --name cahworker --shm-size=4G -d arkseal/cah-worker:cpu
    • add -e NAME={nickname} to specify display name
      • Ex: docker pull arkseal/cah-worker:cpu && docker run --name cahworker -e NAME=ARKseal --shm-size=4G -d arkseal/cah-worker:cpu

Setup

  1. wget https://raw.githubusercontent.com/ARKseal/crawlingathome-worker/master/setup/setup_hybrid.sh
  2. bash setup.sh, to install dependencies.
  3. export PYTHONHASHCODE=0 && python3 crawlingathome.py, to start Crawling!
    • use --name {nickname} to specify your display name

Other Options

  • CPU Only Worker:
    1. wget https://raw.githubusercontent.com/ARKseal/crawlingathome-worker/master/setup/setup_cpu.sh
    2. bash setup.sh, to install dependencies.
    3. export PYTHONHASHCODE=0 && python3 crawlingathome.py --cpu, to start Crawling!
      • use --name {nickname} to specify your display name
  • GPU Only Worker:
    1. wget https://raw.githubusercontent.com/ARKseal/crawlingathome-worker/master/setup/setup_gpu.sh
    2. bash setup.sh, to install dependencies.
    3. export PYTHONHASHCODE=0 && python3 crawlingathome.py --gpu, to start Crawling!
      • use --name {nickname} to specify your display name

Droplet Setup

  1. use cloud-config.yaml script to init the droplet
  2. ssh with this command ssh -oIdentitiesOnly=yes -i~/.ssh/id_cah crawl@{your-droplet-ip}}
  3. check the script by running tail -f crawl.log

TODO

  • Save image embedding
  • Convert images to tfrecords
  • Upload to google drive
  • Prevent corrupt image to be processed
  • Shard of chunk (it needs to read all WAT file which will be bad for low ram server)
  • Crawling@Home integration
  • Verify output