Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Problems with attaching the nullworkload application instance to the experiment in GCE #399

Open
marcinkaczor opened this issue Sep 3, 2020 · 0 comments
Open

Problems with attaching the nullworkload application instance to the experiment in GCE #399

marcinkaczor opened this issue Sep 3, 2020 · 0 comments

Comments

@marcinkaczor
Copy link

marcinkaczor commented Sep 3, 2020 •
edited
Loading

Hello. As part of a master's thesis, I would like to perform some experiments in GCE cloud via CBTOOL. My first aim is to run nullworkload appliction with the usage of nested containers. I took the following steps:

...

[VM_DEFAULTS : GCE_CLOUDCONFIG]
NEST_CONTAINERS_ENABLED = $True

[VM_TEMPLATES : GCE_CLOUDCONFIG]
NEST_CONTAINERS_BASE_IMAGE = size:NA, imageids:1, imageid1:my-ubuntu-1804-lts, cloudinit_packages:openvpn;bc;jq;docker.io;python;redis-server;ntp;redis-tools

Unfortunately, when I'm trying to attach the nullworkload application instance to my experiment by the command:

  • aiattach nullworkload

I'm getting the following errors:

marcin@orchestrator:~/cbtool$ ./cb --hard_reset
Cbtool version is "82962c1f"
Parsing "cloud definitions" file..... "/home/marcin/cbtool/lib/auxiliary//../..//configs/marcin_cloud_definitions.txt" opened and parsed successfully.

Checking "Object Store"..... An Object Store of the kind "Redis" (shared) on node 10.142.0.45, TCP port 6379, database id "0" seems to be running.
Checking "Log Store"..... A Log Store of the kind "rsyslog" (private) on node 10.142.0.45, UDP port 5114 seems to be running.
Checking "Metric Store"..... A Metric Store of the kind "MongoDB" (shared) on node 10.142.0.45, TCP port 27017, database id "metrics" seems to be running.
Checking "File Store"..... A File Store of the kind "rsync" (private) on node 10.142.0.45, TCP port 873 seems to be running.

Executing "hard" reset: (killing all running toolkit processes and flushing stores) before starting the experiment......
Only flush registered cloud: MYGCE
    Killing all processes... done
    Flushing Object Store... done
    Flushing Log Store... done
    Flushing Metric Store... done


Checking for a running API service daemon..... API Service daemon was successfully started. The process id is ['9770'] (http://10.142.0.45:7070).
Checking for a running GUI service daemon..... GUI Service daemon was successfully started. The process id is ['9794', '9795'], listening on port 8080. Full url is "http://10.142.0.45:8080".


 ############################# Executing command "cldattach gce MYGCE" (specified on the configuration file) 

 status: VPN configuration for this cloud already generated: /home/marcin/cbtool/lib/auxiliary//../../configs/generated/MYGCE_server-cb-openvpn.conf

 status: Checking if the ssh key pair "marcin_cbtool_rsa" is created on VMC us-east1-b....
 status: Checking if the imageids associated to each "VM role" are registered on VMC "us-east1-b"....
 status:  WARNING Image id for VM roles "acmeair,liberty": "0000040123242511792" ("cb_acmeair") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "apache,wrk": "0000074975611073851" ("cb_wrk") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "bonnie": "0000098139396296240" ("cb_bonnie") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "btest": "0000066264935330811" ("cb_btest") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "caffe": "0000014349494917245" ("cb_caffe") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "cassandra,mongo_cfg_server,mongodb,mongos,redis,seed,ycsb": "0000059666729869556" ("cb_ycsb") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "client_ibm_daytrader,db2,driver_daytrader,was": "0000052349521107553" ("cb_daytrader") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "client_open_daytrader,geronimo,mysql": "0000094629655075256" ("cb_open_daytrader") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "client_rubis,rubis_db,rubis": "0000039793387843519" ("cb_rubis") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "client_tradelite,driver_tradelite": "0000032206402890225" ("cb_tradelite") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "client_windows,windows": "0000086095781128510" ("cb_windows") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "cn_hpc,fen_hpc": "0000059788337076558" ("cb_hpcc") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "coremark,driver_coremark": "0000047083639541179" ("cb_coremark") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "ddgen": "0000005771270606395" ("cb_ddgen") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "driver_filebench,filebench": "0000085224470736523" ("cb_filebench") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "driver_fio,fio": "0000026752411224710" ("cb_fio") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "driver_hadoop,hadoopmaster,hadoopslave": "0000059622458305109" ("cb_hadoop") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "driver_netperf,netclient,netserver": "0000030769860987776" ("cb_netperf") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "giraphmaster,giraphslave": "0000037124466263610" ("cb_giraph") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "iperfclient,iperfserver": "0000092958431606560" ("cb_iperf") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "kernbench": "0000080711670236658" ("cb_kernbench") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "lb,tinyvm,yatinyvm": "0000008579795908899" ("cb_nullworkload") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "linpack": "0000012440708337052" ("cb_linpack") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "memtier": "0000019136157107221" ("cb_memtier") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "multichase": "0000028256533409339" ("cb_multichase") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "mumblerhead,mumblerremote": "0000009149670589484" ("cb_mumbler") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "nodejs,wrk2": "0000074311680735376" ("cb_wrk2") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "nuttcpclient,nuttcpserver": "0000079306480810013" ("cb_nuttcp") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "oldisimdriver,oldisimlb,oldisimleaf,oldisimroot": "0000045288972569114" ("cb_oldisim") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "parboil": "0000097024708351132" ("cb_parboil") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "postmark": "0000094921030386462" ("cb_postmark") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "rubbos": "0000011291181587207" ("cb_rubbos") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "scimark": "0000039118455565926" ("cb_scimark") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "slurmcontroller,slurmcompute": "0000091371939603605" ("cb_slurm") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "sparkmaster,sparkslave": "0000061986612664611" ("cb_spark") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "specjbb": "0000066628738687283" ("cb_specjbb") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "stress": "0000071693065843989" ("cb_stress") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "sysbench": "0000001916935026064" ("cb_sysbench") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "tpcc": "0000046969466279100" ("cb_tpcc") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "unixbench": "0000011554790667004" ("cb_unixbench") is NOT registered (attaching VMs with any of these roles will result in error).
          WARNING Image id for VM roles "xpingreceiver,xpingsender": "0000055828933983073" ("cb_xping") is NOT registered (attaching VMs with any of these roles will result in error).
          INFO    Image id for VM roles "nest_containers_base_image" is "8762128895458517523" ("my-ubuntu-1804-lts") and it is already registered
 status: VMC "us-east1-b" was successfully tested.


The GCE cloud named "MYGCE" was successfully attached to this experiment.
The experiment identifier is EXP-09-03-2020-08-06-08-PM-UTC


 ############################# Executing command "vmcattach all" (specified on the configuration file) 

 status: (default) Removing all VMs previously created on VMC "us-east1-b" (only VM names starting with "cb-marcin-MYGCE").....
 status: (default) Removing all VVs previously created on VMC "us-east1-b" (only VV names starting with "cb-marcin-MYGCE").....
 status: VMC us-east1-b (7BC1252F-3A74-50D5-A6B7-59550BA0A4A9) was successfully registered on Google Compute Engine "MYGCE" 
 status: The host list for VMC "us-east1-b" is empty ("discover_hosts" was set to "false"). Skipping Host OS performance monitor daemon startup
 status: Attribute "collect_from_host" was set to "false". Skipping Host OS performance monitor daemon startup

All VMCs successfully attached to this experiment.
(MYGCE) aiattach nullworkload
 status: ai_1 (AC6A0B1D-CA2B-5202-8E74-CA53C158D9E9) was successfully defined on Google Compute Engine "MYGCE"  (will now be fully deployed)
 status: VM size "01-512" (vcpus-vmem GB) mapped to cloud-specific flavor "n1-standard-1".
 status: Will instruct cloud-init to install packages "openvpn bc jq docker.io python redis-server ntp redis-tools" on vm_1 (D2AA1B53-14B0-5936-BD17-F7EE0481F467), part of ai_1 (AC6A0B1D-CA2B-5202-8E74-CA53C158D9E9)
 status: Appended userdata...
 status: Starting instance "cb-marcin-mygce-vm1-tinyvm-ai-1" on Google Compute Engine, using the image "my-ubuntu-1804-lts" (8762128895458517523) and size "n1-standard-1", container: ubuntu_cb_nullworkload, connected to network "private", on VMC "us-east1-b", under tenant "default", injecting the contents of the pub ssh key "marcin_cbtool_rsa" (userdata is "False").
 status: Waiting for vm_1 (D2AA1B53-14B0-5936-BD17-F7EE0481F467), part of ai_1 (AC6A0B1D-CA2B-5202-8E74-CA53C158D9E9), to start...
 status: Trying to establish network connectivity to vm_1 (D2AA1B53-14B0-5936-BD17-F7EE0481F467), part of ai_1 (AC6A0B1D-CA2B-5202-8E74-CA53C158D9E9), on IP address 10.142.15.206 (using method "tcp_on_22")...
 status: ai_1 (AC6A0B1D-CA2B-5202-8E74-CA53C158D9E9) was successfully defined on Google Compute Engine "MYGCE"  (will now be fully deployed)
 status: Performing generic application instance post_boot configuration on all VMs belonging to ai_1 (AC6A0B1D-CA2B-5202-8E74-CA53C158D9E9)...
 status: Command "~/cbtool/scripts/common/cb_post_boot.sh" failed to execute on hostname 10.142.15.206, port 22 after attempt 0. Will try 30 more times.
 status: Command "~/cbtool/scripts/common/cb_post_boot.sh" failed to execute on hostname 10.142.15.206, port 22 after attempt 1. Will try 29 more times.
 status: Command "~/cbtool/scripts/common/cb_post_boot.sh" failed to execute on hostname 10.142.15.206, port 22 after attempt 2. Will try 28 more times.
 status: Command "~/cbtool/scripts/common/cb_post_boot.sh" failed to execute on hostname 10.142.15.206, port 22 after attempt 3. Will try 27 more times.
 status: Command "~/cbtool/scripts/common/cb_post_boot.sh" failed to execute on hostname 10.142.15.206, port 22 after attempt 4. Will try 26 more times.
 status: Command "~/cbtool/scripts/common/cb_post_boot.sh" failed to execute on hostname 10.142.15.206, port 22 after attempt 5. Will try 25 more times.

When I'm executing a command:

  • tail -f /var/log/cloudbench/marcin_remotescripts.log

I'm getting the following errors (there are 400 such attempts):

Sep  3 21:26:22 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Starting generic VM post_boot configuration
Sep  3 21:26:22 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): VMs need to be able to perform passwordless SSH between each other. Updating ~/.ssh/id_rsa with the contents from "~/cbtool/credentials/cbtool_rsa" to be the same on all VMs..
Sep  3 21:26:23 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Relaxing all security configurations
Sep  3 21:26:24 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Stopping service "ufw" with command "sudo systemctl stop ufw"...
Sep  3 21:26:24 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Disabling service "ufw" with command "sudo systemctl disable ufw"...
Sep  3 21:26:25 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Disabling Apparmor...
Sep  3 21:26:26 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Stopping service "apparmor" with command "sudo systemctl stop apparmor"...
Sep  3 21:26:26 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Disabling service "apparmor" with command "sudo systemctl disable apparmor"...
Sep  3 21:26:27 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Done
Sep  3 21:26:27 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Starting (AI) Log store...
Sep  3 21:26:28 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Local (AI) Log store started
Sep  3 21:26:28 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Starting (AI) Object store...
Sep  3 21:26:28 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Stopping service "redis-server" with command "sudo systemctl stop redis-server"...
Sep  3 21:26:29 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Disabling service "redis-server" with command "sudo systemctl disable redis-server"...
Sep  3 21:26:29 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Updating object store configuration template
Sep  3 21:26:30 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Restarting service "redis-server", with command "sudo service redis-server restart", attempt 1 of 7...
Sep  3 21:26:30 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Service "redis-server" was successfully restarted
Sep  3 21:26:30 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Enabling service "redis-server", with command "sudo update-rc.d -f redis-server defaults"...
Sep  3 21:26:30 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Local (AI) Object store started
Sep  3 21:26:30 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Making sure all NICs on this instance have IPs configured ...
Sep  3 21:26:30 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Refreshing hosts file ... 
Sep  3 21:26:31 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Creating directory "/nulltest"
Sep  3 21:26:31 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Changing ownership of /nulltest with command "sudo chown -R cbuser:cbuser /nulltest"
Sep  3 21:26:31 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Changing ownership of /nulltest with command "sudo chown -R cbuser:cbuser /nulltest"
Sep  3 21:26:31 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Enabling firewall via ufw commands...
Sep  3 21:26:35 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Executing "post_boot_steps" function
Sep  3 21:26:35 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Restarting service "docker", with command "sudo systemctl restart docker", attempt 1 of 7...
Sep  3 21:26:37 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Service "docker" was successfully restarted
Sep  3 21:26:37 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Enabling service "docker", with command "sudo systemctl enable docker"...
Sep  3 21:26:37 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Downloading container image from: ibmcb/ubuntu_cb_nullworkload
Sep  3 21:26:38 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Image pulled, starting container...
Sep  3 21:26:38 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Stopping service "sshd" with command "sudo systemctl stop sshd"...
Sep  3 21:26:38 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Disabling service "sshd" with command "sudo systemctl disable sshd"...
Sep  3 21:26:39 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Container started, settling... (sudo docker run -u ${username} -it -d --name cbnested --privileged --net host --env CB_SSH_PUB_KEY="$(cat ~/.ssh/id_rsa.pub)" -v ~/:${NEST_EXPORTED_HOME} ${image} bash -c "sudo rm -f ${NEST_POST_BOOT_FLAG}; sudo mkdir ${userpath}/$username; sudo rsync -arz ${NEST_EXPORTED_HOME}/* ${NEST_EXPORTED_HOME}/.* ${userpath}/${username}/; sudo chmod 755 ${userpath}/${username}; sudo chown -R ${username}:${username} ${userpath}/${username}; (sudo bash /etc/my_init.d/inject_pubkey_and_start_ssh.sh &); cd; source ~/cbtool/scripts/common/cb_common.sh; syslog_netcat \"Running post_boot inside container...\"; ~/cbtool/scripts/common/cb_post_boot_container.sh; code=\$?; if [ \$code -gt 0 ] ; then echo post boot failed; exit \$code; fi; touch ${NEST_POST_BOOT_FLAG}; sleep infinity")
Sep  3 21:26:39 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Return code: 1 Error: No such container: cbnested
Sep  3 21:26:39 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Still waiting on container startup. Attempts left: 399
Sep  3 21:26:39 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Recreating container...
Sep  3 21:26:40 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Stopping service "sshd" with command "sudo service sshd stop"...
Sep  3 21:26:42 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Disabling service "sshd" with command "sudo update-rc.d -f sshd remove"...
Sep  3 21:26:42 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Restart command: sudo docker run -u ${username} -it -d --name cbnested --privileged --net host --env CB_SSH_PUB_KEY="$(cat ~/.ssh/id_rsa.pub)" -v ~/:${NEST_EXPORTED_HOME} ${image} bash -c "sudo rm -f ${NEST_POST_BOOT_FLAG}; sudo mkdir ${userpath}/$username; sudo rsync -arz ${NEST_EXPORTED_HOME}/* ${NEST_EXPORTED_HOME}/.* ${userpath}/${username}/; sudo chmod 755 ${userpath}/${username}; sudo chown -R ${username}:${username} ${userpath}/${username}; (sudo bash /etc/my_init.d/inject_pubkey_and_start_ssh.sh &); cd; source ~/cbtool/scripts/common/cb_common.sh; syslog_netcat \"Running post_boot inside container...\"; ~/cbtool/scripts/common/cb_post_boot_container.sh; code=\$?; if [ \$code -gt 0 ] ; then echo post boot failed; exit \$code; fi; touch ${NEST_POST_BOOT_FLAG}; sleep infinity"
Sep  3 21:26:43 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Recreated.
Sep  3 21:26:45 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Return code: 1 Error: No such container: cbnested
Sep  3 21:26:45 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Still waiting on container startup. Attempts left: 398
Sep  3 21:26:45 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Recreating container...
Sep  3 21:26:46 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Stopping service "sshd" with command "sudo service sshd stop"...
Sep  3 21:26:47 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Disabling service "sshd" with command "sudo update-rc.d -f sshd remove"...
Sep  3 21:26:47 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Restart command: sudo docker run -u ${username} -it -d --name cbnested --privileged --net host --env CB_SSH_PUB_KEY="$(cat ~/.ssh/id_rsa.pub)" -v ~/:${NEST_EXPORTED_HOME} ${image} bash -c "sudo rm -f ${NEST_POST_BOOT_FLAG}; sudo mkdir ${userpath}/$username; sudo rsync -arz ${NEST_EXPORTED_HOME}/* ${NEST_EXPORTED_HOME}/.* ${userpath}/${username}/; sudo chmod 755 ${userpath}/${username}; sudo chown -R ${username}:${username} ${userpath}/${username}; (sudo bash /etc/my_init.d/inject_pubkey_and_start_ssh.sh &); cd; source ~/cbtool/scripts/common/cb_common.sh; syslog_netcat \"Running post_boot inside container...\"; ~/cbtool/scripts/common/cb_post_boot_container.sh; code=\$?; if [ \$code -gt 0 ] ; then echo post boot failed; exit \$code; fi; touch ${NEST_POST_BOOT_FLAG}; sleep infinity"
Sep  3 21:26:48 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Recreated.
Sep  3 21:26:50 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Return code: 1 Error: No such container: cbnested
Sep  3 21:26:50 cb-marcin-mygce-vm1-tinyvm-ai-1.us-east1-b.c.thesis-273119.internal cb-marcin-mygce-vm1-tinyvm-ai-1_ cloudbench EXP-09-03-2020-09-19-16-PM-UTC cb_common.sh (4573): Still waiting on container startup. Attempts left: 397
...

Have you got any ideas what causes the problem Error: No such container: cbnested (cause it is the main problem as I guess...) and how can I fix it?

Thank you.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@marcinkaczor