Building for the RaspberryPi?

[chrisruk]

Hi,

I'm just wondering if anyone has managed to build rumprun for the Pi?

I'm trying the following so far:

docker run -it resin/rpi-raspbian:jessie 

Then in the docker:

sudo apt-get update
sudo apt-get install make
sudo apt-get install git
sudo apt-get install gcc-4.9
sudo apt-get install zlib1g-dev

git clone https://github.com/dave-tucker/rumprun.git
git checkout raspberrypi

Modify:

./src-netbsd/sys/arch/arm/include/aeabi.h
./src-netbsd/sys/sys/cdbr.h
./src-netbsd/lib/libc/locale/setlocale_local.h
./src-netbsd/sys/crypto/camellia/camellia.h
./src-netbsd/crypto/external/bsd/openssl/dist/crypto/camellia/camellia.h

adding:

typedef	unsigned int size_t;

I also commented out

TARGETS+=	compiler_rt

In platform/Makefile.inc.

Then ran:

export HOST_CC=gcc-4.9
export CC=gcc-4.9

./build-rr.sh -b rpi -j4 hw -- -r -j4 -F ACFLAGS="-march=armv6k -march=armv6zk -mcpu=arm1176jz-s -mfloat-abi=hard" 

Which gave:

>>
>> Finished ./build-rr.sh for hw
>>
>> For Rumprun developers (if you're not sure, you don't need it):
. "/root/tucker2/rumprun/./obj-evbearmhf-el-hw-raspberrypi/config"
>>
>> toolchain tuple: arm-rumprun-netbsdelf-eabihf
>> cc wrapper: arm-rumprun-netbsdelf-eabihf-gcc
>> installed to "/root/tucker2/rumprun/./rumprun-raspberrypi"
>>
>> ./build-rr.sh ran successfully

I then did:

Created file rumprun-bake2.conf:

#
# rumpbake default configuration file
#

version 20150930

#
# First, define internal convenience sets.
#
create _foundation "basic components for the Rumprun unikernel"
	add _foundation				\
		-lrumpvfs			\
		-lrumpkern_bmktc		\
		-lrumpkern_mman			\
		-lrumpdev			\
		-lrumpfs_tmpfs			\
		-lrumpnet_config		\
		-lrumpnet			\
		-lrumpdev_bpf			\
		-lrumpdev_vnd			\
		-lrumpdev_rnd

create _virtio "virtio drivers"
	add _virtio				\
		-lrumpdev_virtio_if_vioif	\
		-lrumpdev_virtio_ld		\
		-lrumpdev_virtio_viornd		\
		-lrumpdev_pci_virtio		\
		-lrumpdev_pci

create _audio "audio subsystem and some PCI audio device drivers"
	add _audio				\
		-lrumpdev_audio			\
		-lrumpdev_audio_ac97		\
		-lrumpdev_pci_auich		\
		-lrumpdev_pci_eap		\
		-lrumpdev_pci_hdaudio		\
		-lrumpdev_hdaudio_hdafg

create _netinet "TCP/IP (v4)"
	add _netinet				\
		-lrumpnet_netinet		\
		-lrumpnet_net			\
		-lrumpnet

create _netinet6 "TCP/IP (v6)"
	add _netinet6				\
		-lrumpnet_netinet6		\
		-lrumpnet_net			\
		-lrumpnet

create _netunix "local domain sockets"
	add _netunix				\
		-lrumpnet_local			\
		-lrumpnet

create _stdfs "selection of file system drivers"
	add _stdfs				\
		-lrumpfs_ffs			\
		-lrumpfs_cd9660			\
		-lrumpfs_ext2fs			\
		-lrumpdev_disk			\
	-lrumpvfs

create _pciether "PCI Ethernet NIC drivers"
	add _pciether				\
		-lrumpdev_pci_if_wm		\
		-lrumpdev_miiphy		\
		-lrumpdev_pci

create _usb "USB stack incl. PCI USB host controllers"
	add _usb				\
		-lrumpdev_pci_usbhc		\
		-lrumpdev_usb			\
		-lrumpdev_umass			\
		-lrumpdev_scsipi		\
		-lrumpdev_pci

# this is a config so that it can be assimilated
create _sysproxy "system call proxy support"
	add _sysproxy				\
		-lrumpkern_sysproxy

create _miconf "general useful MI/pseudo driver set"
	assimilate _miconf		\
		_foundation		\
		_stdfs			\
		_sysproxy



create hw_generic2 "generic targets, includes (almost) all drivers"
	assimilate hw_generic2		\
		_miconf			\

CC=rumprun/rumprun-raspberrypi/bin/arm-rumprun-netbsdelf-eabihf-gcc
$CC 1.c -o a.out
rumprun-bake -c rumprun-bake2.conf hw_generic2 test2.bin a.out 
qemu-system-arm -cpu arm1176 -m 128 -M raspi2 -kernel test2.bin
qemu-system-arm -nographic -cpu arm1176 -m 256 -M raspi2 -serial mon:stdio -kernel test2.bin

However no output is produced either graphically or via the serial output (I listed the command for the serial version above). And it seems to give high CPU usage on all cores.

Without the custom conf file I get:

root@56604f6e742f:~# rumprun-bake hw_generic a2.bin a.out 

!!!
!!! NOTE: rumprun-bake is experimental. syntax may change in the future
!!!

/usr/bin/ld: cannot find -lrumpdev_virtio_if_vioif
/usr/bin/ld: cannot find -lrumpdev_virtio_ld
/usr/bin/ld: cannot find -lrumpdev_virtio_viornd
/usr/bin/ld: cannot find -lrumpdev_pci_virtio
/usr/bin/ld: cannot find -lrumpdev_pci
/usr/bin/ld: cannot find -lrumpdev_audio_ac97
/usr/bin/ld: cannot find -lrumpdev_pci_auich
/usr/bin/ld: cannot find -lrumpdev_pci_eap
/usr/bin/ld: cannot find -lrumpdev_pci_hdaudio
/usr/bin/ld: cannot find -lrumpdev_hdaudio_hdafg
/usr/bin/ld: cannot find -lrumpdev_pci_if_wm
/usr/bin/ld: cannot find -lrumpdev_miiphy
/usr/bin/ld: cannot find -lrumpdev_pci_usbhc
collect2: error: ld returned 1 exit status

Any suggestions would be most appreciated!

Cheers
Chris

[zatkh]

I am also wondering if there is any update on rumprun rpi or aarch64 support?

[ananos]

Hi @chrisruk, @zatkh,

we have a port of https://github.com/nabla-containers for aarch64. Maybe you'll find that useful:

https://github.com/cloudkernels/rumprun (branch solo5-tests-aarch64).

You can also find some info at https://blog.cloudkernels.net

cheers,
Tassos

[zatkh]

That's great @ananos ,thanks, exactly was looking for aarch64 support of nabla to test with my solo5 backend extension! will try now

[zatkh]

Hi @ananos, I am testing this, it seems the blog post is not a match with current version of solo5 and nabla. (for example install solo5.o-->libsolo5_ukvm is before adding spt), so not sure what is solo5_stock and why you need two solo5 repos?

[ananos]

Hi @zatkh,

our aarch64 support has been accepted upstream so you should be able to build for a 64-bit RPi3 using the https://github.com/nabla-containers/rumprun

We are in the process of updating the blog posts to point to the updated repos. I'll ping you when we manage to do that.

In any case, please try using the Nabla version (we have tested hvt & spt on aarch64) and let us know how it goes!

[ananos]

Hi @zatkh,

we have updated the posts to point to the latest versions of the repos (nabla/solo5/cloudkernels). Please have a look and let us know if you manage to get something running, or if we missed something in the update process.

thanks!

[zatkh]

Hi @ananos ,

Sorry for the delay (I'm traveling), and thanks for letting me know. I quickly tested basic packages (hello,python3) on both my aarch64 server and rpi3 and didn't have time to fix some of the problems yet.

• On my aarch64 server: Linux arm-build-2 4.15.0-20-generic fix mailing list address #21-Ubuntu SMP Tue Apr 24 06:16:20 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux, both gcc 7.3 and 5.5 , rumprun build is ok but python3-sqlite3 fails.

sqlite3: In function sqlite3VXPrintf':
/home/zahra/git/rumprun-packages/sqlite/build/sqlite3.c:22563: undefined reference to __lttf2' /home/zahra/git/rumprun-packages/sqlite/build/sqlite3.c:22582: undefined reference to __gttf2'
/home/zahra/git/rumprun-packages/sqlite/build/sqlite3.c:22605: undefined reference to __getf2'

• On my rpi3, Linux raspberrypi 4.18.20-v8+ Export RUMPMAKE #1 SMP PREEMPT Wed Feb 20 11:29:25 GMT 2019 aarch64 GNU/Linux, gcc 5.4.1 rumprun build fails.

compile rumpkern/atomic_add_16.o
/home/pi/git/rumprun/src-netbsd/sys/rump/librump/rumpkern/../../../lib/libkern/../../../common/lib/libc/arch/arm/atomic/atomic_add_16.S:78: Error: selected processor does not support `strexh r2,r0,[ip]' in ARM mode

*** [atomic_add_16.o] Error code 1

nbmake[5]: stopped in /home/pi/git/rumprun/src-netbsd/sys/rump/librump/rumpkern
1 error

[ananos]

Hi @zatkh,

regarding the pi rumprun issue, I'm using the following and the build completes:

root@pi:/build/rumprun# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/aarch64-linux-gnu/6/lto-wrapper
Target: aarch64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 6.3.0-18+deb9u1' --with-bugurl=file:///usr/share/doc/gcc-6/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-6 --program-prefix=aarch64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libquadmath --enable-plugin --enable-default-pie --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-6-arm64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-6-arm64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-6-arm64 --with-arch-directory=aarch64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-multiarch --enable-fix-cortex-a53-843419 --enable-checking=release --build=aarch64-linux-gnu --host=aarch64-linux-gnu --target=aarch64-linux-gnu
Thread model: posix
gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1) 
root@pi:/build/rumprun# uname -a
Linux pi 4.16.18-v8+ #1 SMP PREEMPT Sat Sep 15 19:33:42 EEST 2018 aarch64 GNU/Linux
root@pi:/build/rumprun# dpkg -l |grep gcc   
ii  gcc                             4:6.3.0-4                      arm64        GNU C compiler
ii  gcc-6                           6.3.0-18+deb9u1                arm64        GNU C compiler
ii  gcc-6-base:arm64                6.3.0-18+deb9u1                arm64        GCC, the GNU Compiler Collection (base package)
ii  libgcc-6-dev:arm64              6.3.0-18+deb9u1                arm64        GCC support library (development files)
ii  libgcc1:arm64                   1:6.3.0-18+deb9u1              arm64        GCC support library

BTW on an arm server, the build completes using the following:

# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/aarch64-linux-gnu/5/lto-wrapper
Target: aarch64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.10' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libquadmath --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-arm64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-arm64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-arm64 --with-arch-directory=aarch64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-multiarch --enable-fix-cortex-a53-843419 --disable-werror --enable-checking=release --build=aarch64-linux-gnu --host=aarch64-linux-gnu --target=aarch64-linux-gnu
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.10) 
root@kmaxgw-arm64:~# uname -a
Linux arm64 4.17.0-00001-g010275f #2 SMP PREEMPT Sun Jul 22 12:33:44 EEST 2018 aarch64 aarch64 aarch64 GNU/Linux
root@kmaxgw-arm64:~# dpkg -l |grep gcc
ii  gcc-5                                  5.4.0-6ubuntu1~16.04.10                    arm64        GNU C compiler
ii  gcc-5-base:arm64                       5.4.0-6ubuntu1~16.04.10                    arm64        GCC, the GNU Compiler Collection (base package)
ii  libgcc-5-dev:arm64                     5.4.0-6ubuntu1~16.04.10                    arm64        GCC support library (development files)
ii  libgcc1:arm64                          1:8.1.0-12ubuntu1                          arm64        GCC support library

regarding the python3 issue, we are working on trying to understand why this happens. My initial assumption is that the long double type assumes it's 128-bit and the processor needs to use sw emulation for these built-in operations. If we manually link -lgcc from the system we're able to build python3 -- but we need to figure it out as this approach is far from clean ;-)

Will let you know as soon as we have something -- please do let us know if you manage to figure it out!

thanks!

[QiZhang1997]

Hi @ananos
When I tried to run unikernel on Raspberry pi 3B + following the instructions in your bloghere.

When I enter the following command/build/solo5/tenders/spt/solo5-spt --mem=32 --net=tap0 --disk=dummy ./bin/hello.spt.

I met an errorsolo5-spt: ./bin/hello.spt: No Solo5 ABI information found in executable

I don’t know what went wrong, do you have any suggestion? Thanks a lot, I'm looking forward to your reply. I'm very interested in your project.

[ananos]

just for reference, as the question was already answered offline -- rumprun over solo5 works with solo5 v0.4.1, so the solution to this question is to use tender binaries built from version v0.4.1

[hazcod]

I wonder if there is a way to run microkernels directly on the Pi without the use of Docker/firecracker/...