Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Toolchain for ARM 32 bit fails on android host with exit code 159 #71

Open
4ntoine opened this issue Jan 21, 2021 · 38 comments
Open

Toolchain for ARM 32 bit fails on android host with exit code 159 #71

4ntoine opened this issue Jan 21, 2021 · 38 comments

Comments

@4ntoine
Copy link

4ntoine commented Jan 21, 2021

When compiled for ARM 32bit CT_HOST=arm-linux-gnueabihf archiever fails to put .o files into .a with return code 159. If not archiving .o into .a and pass to linker linker fails to link with the same return code.

I will provide the details later.
Is it a known issue?

100% the same command-lines that use toolchain compiled for arm64 (CT_HOST=aarch64-linux-gnu) works just fine.

@4ntoine 4ntoine changed the title Toolchain for ARM produces corrupted files Toolchain for ARM 32 bit produces corrupted files Jan 21, 2021
@jcmvbkbc
Copy link
Owner

I know that there's been a user who ran the toolchain on armhf host, they've reported a real issue in the compiler, but after it was fixed they were happy, so presumably it worked for them. I guess it doesn't happen with arbitrary .o files? I'll try to reproduce it anyway.

I will provide the details later.

That would be helpful, thank you. Please provide the version of ct-ng and configuration used to build the toolchan as well as the toolchain invocation commands that lead to the crash.

@4ntoine
Copy link
Author

4ntoine commented Jan 21, 2021

Here is how the extensa toolchain is built.

Common

git clone https://github.com/jcmvbkbc/crosstool-NG.git
cd crosstool-NG.git
git checkout xtensa-1.24.x
./bootstrap
./configure --prefix=./installed
make
make install
cd ./installed/bin

ARM64 (works fine)

sudo apt install g++-aarch64-linux-gnu
sudo apt install gcc-aarch64-linux-gnu

mkdir overlays
download https://github.com/jcmvbkbc/crosstool-NG/blob/xtensa-1.22.x/overlays/xtensa_lx106.tar to overlays (and extract).
./ct-ng list-samples
./ct-ng aarch64-linux-gnu,xtensa-lx106-elf
./ct-ng build.4

Grab the files from ~/x-tools/HOST-aarch64-linux-gnu

ARM32 (bad instructions)

sudo apt install gcc-arm-linux-gnueabihf
sudo apt install g++-arm-linux-gnueabihf

edit .config and set CT_HOST=arm-linux-gnueabihf
./ct-ng build.4

Grab the files from ~/x-tools/HOST-arm-linux-gnueabihf.

The files

Compiled toolchain for ARM, SDK distrib (including compiled toolchain), screenshot and build .b files for xtensa in:
https://dropmefiles.com/SFH4q
Note README for the explanation.

Compilation log (how to reproduce):
https://dropmefiles.com/2nSaW

bin/xtensa-lx106-elf-ar cru /data/data/../coreBuild/core.a /data/data/../coreBuild/cont.S.o
xtensa-lx106-elf-ar returned 159

@4ntoine
Copy link
Author

4ntoine commented Jan 21, 2021

@jcmvbkbc Let me know if you need anything else to repro. Meanwhile i'll try to refine it to have minimum reproducible example

@4ntoine
Copy link
Author

4ntoine commented Jan 21, 2021

Yup, just compile any .cpp to .o and archieve it to .a and it fails with return code 159:
https://dropmefiles.com/uShD4

@4ntoine
Copy link
Author

4ntoine commented Jan 22, 2021

@jcmvbkbc Is there anything i can assist you with? Probably try with/without some arguments, compare with file compiled with ARM64 version of toolchain, anything?

@jcmvbkbc
Copy link
Owner

jcmvbkbc commented Jan 22, 2021

I don't have easy access to old ARM hardware, I tried your instructions in qemu linux-user, it worked for me. Looking through the emulated instructions trace I've noticed that the following VFP/NEON instructions were used:

vand
vceq.i8
vdup.8
vld1.8
vldr
vmov
vmov.32
vmov.f32
vpadd.i8
vpop
vpush
vst1.8
vstmia
vstr

@4ntoine does you hardware support these instructions? Do you have any way to check whether ar crash on your hardware happens on one of these instructions? Maybe run your reproducer in the debugger or under strace -i and check disassembly at the crash address?

@jcmvbkbc
Copy link
Owner

jcmvbkbc commented Jan 22, 2021

xtensa-lx106-elf-ar returned 159

OTOH this seems to correspond to termination by SIGSYS. In any case running reproducer on hardware under strace -i should shed some light. Here's what I see in emulation:

QEMU_STRACE= xtensa-lx106-elf-ar cru z.a BearSSLHelpers.cpp.o
5698 brk(NULL) = 0x00108000
5698 brk(0x00108d0c) = 0x00108d0c
5698 uname(0xfffef208) = 0
5698 readlink("/proc/self/exe",0xfffee350,4096) = 147
5698 brk(0x00129d0c) = 0x00129d0c
5698 brk(0x0012a000) = 0x0012a000
5698 openat(AT_FDCWD,"/usr/lib/locale/locale-archive",O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
5698 fstat64(3,0x001067d0) = 0
5698 mmap2(NULL,2097152,PROT_READ,MAP_PRIVATE,3,0) = 0xff5ef000
5698 mmap2(NULL,2596864,PROT_READ,MAP_PRIVATE,3,0x6b) = 0xff375000
5698 close(3) = 0
5698 openat(AT_FDCWD,"/usr/lib/arm-linux-gnueabihf/gconv/gconv-modules.cache",O_RDONLY) = -1 errno=2 (No such file or directory)
5698 openat(AT_FDCWD,"/usr/lib/arm-linux-gnueabihf/gconv/gconv-modules",O_RDONLY|O_CLOEXEC) = -1 errno=2 (No such file or directory)
5698 stat64("z.a",0xfffef218) = -1 errno=2 (No such file or directory)
5698 prlimit64(0,7,0,-69256,1091000,734348) = 0
5698 stat64("z.a",0xfffef190) = -1 errno=2 (No such file or directory)
5698 openat(AT_FDCWD,"z.a",O_RDWR|O_CREAT|O_LARGEFILE|O_TRUNC,0666) = 3
5698 fcntl64(3,F_GETFD) = 0
5698 fcntl64(3,F_SETFD,1) = 0
5698 fstat64(3,0xfffed098) = 0
5698 write(3,0x10c670,8) = 8
5698 close(3) = 0
5698 openat(AT_FDCWD,"z.a",O_RDONLY|O_LARGEFILE) = 3
5698 fcntl64(3,F_GETFD) = 0
5698 fcntl64(3,F_SETFD,1) = 0
5698 fstat64(3,0xfffef000) = 0
5698 read(3,0x10c6d8,4096) = 8
5698 read(3,0x10c6d8,4096) = 0
5698 openat(AT_FDCWD,"BearSSLHelpers.cpp.o",O_RDONLY|O_LARGEFILE) = 4
5698 fcntl64(4,F_GETFD) = 0
5698 fcntl64(4,F_SETFD,1) = 0
5698 gettimeofday(-69160,0,263565312,263565312,0,28) = 0
5698 getpid() = 5698
5698 openat(AT_FDCWD,"stp3bMzo",O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE,0600) = 5
5698 close(5) = 0
5698 stat64("stp3bMzo",0xfffef1f8) = 0
5698 openat(AT_FDCWD,"stp3bMzo",O_RDWR|O_CREAT|O_LARGEFILE|O_TRUNC,0666) = 5
5698 fcntl64(5,F_GETFD) = 0
5698 fcntl64(5,F_SETFD,1) = 0
5698 stat64("BearSSLHelpers.cpp.o",0xfffed248) = 0
5698 lstat64("/home",0xfffebf90) = 0
5698 lstat64("/home/jcmvbkbc",0xfffebf90) = 0
5698 lstat64("/home/jcmvbkbc/ws",0xfffebf90) = 0
5698 lstat64("/home/jcmvbkbc/ws/tensilica",0xfffebf90) = 0
5698 lstat64("/home/jcmvbkbc/ws/tensilica/bug-triage",0xfffebf90) = 0
5698 lstat64("/home/jcmvbkbc/ws/tensilica/bug-triage/20210121-ct-ng-armhf-ar",0xfffebf90) = 0
5698 lstat64("/home/jcmvbkbc/ws/tensilica/bug-triage/20210121-ct-ng-armhf-ar/crosstool-NG",0xfffebf90) = 0
5698 lstat64("/home/jcmvbkbc/ws/tensilica/bug-triage/20210121-ct-ng-armhf-ar/crosstool-NG/build",0xfffebf90) = 0
5698 lstat64("/home/jcmvbkbc/ws/tensilica/bug-triage/20210121-ct-ng-armhf-ar/crosstool-NG/build/HOST-arm-linux-gnueabihf",0xfffebf90) = 0
5698 lstat64("/home/jcmvbkbc/ws/tensilica/bug-triage/20210121-ct-ng-armhf-ar/crosstool-NG/build/HOST-arm-linux-gnueabihf/xtensa-lx106-elf",0xfffebf90) = 0
5698 lstat64("/home/jcmvbkbc/ws/tensilica/bug-triage/20210121-ct-ng-armhf-ar/crosstool-NG/build/HOST-arm-linux-gnueabihf/xtensa-lx106-elf/bin",0xfffebf90) = 0
5698 lstat64("/home/jcmvbkbc/ws/tensilica/bug-triage/20210121-ct-ng-armhf-ar/crosstool-NG/build/HOST-arm-linux-gnueabihf/xtensa-lx106-elf/bin/xtensa-lx106-elf-ar",0xfffebf90) = 0
5698 openat(AT_FDCWD,"/home/jcmvbkbc/ws/tensilica/bug-triage/20210121-ct-ng-armhf-ar/crosstool-NG/build/HOST-arm-linux-gnueabihf/xtensa-lx106-elf/bin/../bin/../lib/bfd-plugins",O_RDONLY|O_DIRECTORY|O_LARGEFILE|O_NONBLOCK|O_CLOEXEC) = -1 errno=2 (No such file or directory)
5698 fstat64(4,0xfffecf40) = 0
5698 read(4,0x1120f8,4096) = 4096
5698 _llseek(4,0,425984,0xfffecf50,SEEK_SET) = 0
5698 read(4,0x1120f8,4096) = 4096
5698 fstat64(4,0xfffecff8) = 0
5698 _llseek(4,0,434176,0xfffecf50,SEEK_SET) = 0
5698 read(4,0x1120f8,4096) = 2628
5698 _llseek(4,0,425984,0xfffecf50,SEEK_SET) = 0
5698 read(4,0x1120f8,4096) = 4096
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 read(4,0x1120f8,4096) = 4096
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 read(4,0x1120f8,4096) = 2628
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecff8) = 0
5698 fstat64(4,0xfffecfa8) = 0
5698 _llseek(4,0,417792,0xfffecee8,SEEK_SET) = 0
5698 read(4,0x1120f8,4096) = 4096
5698 read(4,0x116b2c,4096) = 4096
5698 read(4,0x1120f8,4096) = 4096
5698 _llseek(4,0,0,0xfffecea0,SEEK_SET) = 0
5698 read(4,0x1120f8,4096) = 4096
5698 _llseek(4,0,258048,0xfffece58,SEEK_SET) = 0
5698 read(4,0x1120f8,4096) = 4096
5698 fstat64(4,0xfffecf10) = 0
5698 _llseek(4,0,262144,0xfffece50,SEEK_SET) = 0
5698 read(4,0x1120f8,4096) = 4096
5698 _llseek(4,0,258048,0xfffece58,SEEK_SET) = 0
5698 read(4,0x1120f8,4096) = 4096
5698 _llseek(4,0,262144,0xfffece58,SEEK_SET) = 0
5698 read(4,0x1120f8,4096) = 4096
5698 _llseek(4,0,266240,0xfffece58,SEEK_SET) = 0
5698 brk(0x0014b000) = 0x0014b000
5698 _llseek(4,0,4096,0xfffece10,SEEK_SET) = 0
5698 read(4,0x1120f8,4096) = 4096
5698 _llseek(4,0,8192,0xfffece10,SEEK_SET) = 0
5698 read(4,0x1120f8,4096) = 4096
5698 _llseek(4,0,110592,0xfffece10,SEEK_SET) = 0
5698 read(4,0x1120f8,4096) = 4096
5698 _llseek(4,0,114688,0xfffece10,SEEK_SET) = 0
5698 read(4,0x1120f8,4096) = 4096
5698 _llseek(4,0,135168,0xfffece10,SEEK_SET) = 0
5698 read(4,0x1120f8,4096) = 4096
5698 _llseek(4,0,139264,0xfffece10,SEEK_SET) = 0
5698 _llseek(4,0,139264,0xfffece10,SEEK_SET) = 0
5698 _llseek(4,0,155648,0xfffece10,SEEK_SET) = 0
5698 read(4,0x1120f8,4096) = 4096
5698 _llseek(4,0,249856,0xfffece28,SEEK_SET) = 0
5698 read(4,0x1120f8,4096) = 4096
5698 fstat64(5,0xfffed100) = 0
5698 _llseek(4,0,253952,0xfffecfd0,SEEK_SET) = 0
5698 read(4,0x1120f8,4096) = 4096
5698 read(4,0x133754,4096) = 4096
5698 read(4,0x1120f8,4096) = 4096
5698 gettimeofday(-77532,0,0,263565312,1031064,0) = 0
5698 _llseek(4,0,0,0xfffed0e8,SEEK_SET) = 0
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0xfffed248,8192) = 8192
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0xfffed5da,4096) = 4096
5698 read(4,0x1120f8,4096) = 2628
5698 write(5,0x12c218,4096) = 4096
5698 write(5,0x12c218,1714) = 1714
5698 close(5) = 0
5698 brk(0x00134000) = 0x00134000
5698 close(3) = 0
5698 lstat64("z.a",0xfffef1f8) = 0
5698 rename("stp3bMzo","z.a") = 0
5698 chmod("z.a",0644) = 0
5698 chown32(1112336,1000,1000,1,0,1112336) = 0
5698 chmod("z.a",0644) = 0
5698 exit_group(0)

@4ntoine
Copy link
Author

4ntoine commented Jan 22, 2021

@jcmvbkbc Thanks for looking into it.

it worked for me

jic i've tried on 2 arm32 devices and both did not work (different CPUs) - Redmi 7a and Huawei Honor 9s. BTW i have a couple of really old devices, i can double check..

does you hardware support these instructions? Do you have any way to check whether ar crash on your hardware happens on one of these instructions?

At the moment i have only Honor 9s device, can you find the specs?

I'm afraid strace requires root access and my device is not rooted.

I will try to get the stacktrace. BTW i can see the binaries are stripped (in ~/x-tools/..). Any chance to get them unstripped (probably from ct-ng build dir)? Will it help if you check ARM64 sys calls and compare them (i can provide distrib similar to what i've already uploaded for arm32)?

@jcmvbkbc
Copy link
Owner

At the moment i have one 32bit cpu device for the testing - it's Huawei Honor 9s device, can you find the specs?

We can approach it from the other side: could you use arm-linux-gnueabi host triplet and see if the toolchain built with it works?

@4ntoine
Copy link
Author

4ntoine commented Jan 22, 2021

At the moment i have one 32bit cpu device for the testing - it's Huawei Honor 9s device, can you find the specs?

We can approach it from the other side: could you use arm-linux-gnueabi host triplet and see if the toolchain built with it works?

Yup, will do that and return back to you soon. Upd: nope, it will take few hours..

@jcmvbkbc
Copy link
Owner

I'm afraid strace requires root access and my device is not rooted.

Can you try running ulimit -c unlimited before running ar, collect the core dump and load it together with ar into cross gdb (arm-linux-gnueabihf-gdb -c core xtensa-lx106-elf-ar)?

@jcmvbkbc
Copy link
Owner

At the moment i have only Honor 9s device, can you find the specs?

What I find says that's based on ARM cortex-a53, which is 64-bit ARMv8. Is there a reason to tun 32-bit code on it?
Could you also check /proc/cpuinfo contents on your devices?

@4ntoine
Copy link
Author

4ntoine commented Jan 22, 2021

At the moment i have only Honor 9s device, can you find the specs?

What I find says that's based on ARM cortex-a53, which is 64-bit ARMv8. Is there a reason to tun 32-bit code on it?

Nope, it's detected as 32 bit by Android:
https://stackoverflow.com/questions/65769808/device-is-incorrectly-detected-as-32bit-by-android

It's armv8l, not armv8a. I'm not an expert here but it seems it makes a difference.

Could you also check /proc/cpuinfo contents on your devices?

will do that

ulimit

I believe it's just absent on "regular android device". Doesn't it require root too?

@4ntoine
Copy link
Author

4ntoine commented Jan 22, 2021

adb shell ls -l /proc/cpuinfo
-r--r--r-- 1 root root 0 2021-01-22 23:29 /proc/cpuinfo

can't access it without rooting

@4ntoine
Copy link
Author

4ntoine commented Jan 22, 2021

device-2021-01-22-233036

@4ntoine
Copy link
Author

4ntoine commented Jan 22, 2021

No stacktrace/addressed in the logcat, will try not hard float toolchain

@jcmvbkbc
Copy link
Owner

can't access it without rooting

try adb shell cat /proc/cpuinfo instead.

@4ntoine
Copy link
Author

4ntoine commented Jan 22, 2021

adb shell cat /proc/cpuinfo

processor	: 0
model name	: ARMv8 Processor rev 4 (v8l)
BogoMIPS	: 26.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 1
model name	: ARMv8 Processor rev 4 (v8l)
BogoMIPS	: 26.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 2
model name	: ARMv8 Processor rev 4 (v8l)
BogoMIPS	: 26.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 3
model name	: ARMv8 Processor rev 4 (v8l)
BogoMIPS	: 26.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 4
model name	: ARMv8 Processor rev 4 (v8l)
BogoMIPS	: 26.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 5
model name	: ARMv8 Processor rev 4 (v8l)
BogoMIPS	: 26.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 6
model name	: ARMv8 Processor rev 4 (v8l)
BogoMIPS	: 26.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 7
model name	: ARMv8 Processor rev 4 (v8l)
BogoMIPS	: 26.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

Hardware	: MT6762V/WR

@4ntoine
Copy link
Author

4ntoine commented Jan 23, 2021

@jcmvbkbc arm-linux-gnueabi does not work too: same error code 159. I'm going to check if symlinking hurts (though i doubt) and targeting Android 9 (usually targeting Android 10)

@4ntoine
Copy link
Author

4ntoine commented Jan 23, 2021

@jcmvbkbc Nope, i've just tried without symlinking and targeting Android 28
Снимок экрана 2021-01-23 в 15 16 24

same issue.

@4ntoine
Copy link
Author

4ntoine commented Jan 23, 2021

@jcmvbkbc anything i can try locally just to have more info to decide on? step-by-step for non-rooted device guide appreciated

@4ntoine
Copy link
Author

4ntoine commented Jan 29, 2021

@jcmvbkbc Any progress on it? Is there anything i can help with?

@jcmvbkbc
Copy link
Owner

I'm setting things up for debugging on android. Expecting to get to it this weekend.

@4ntoine
Copy link
Author

4ntoine commented Jan 29, 2021

@jcmvbkbc Great! Thank you

@jcmvbkbc
Copy link
Owner

I just tried xtensa-lx106-elf-ar cru z.a BearSSLHelpers.cpp.o in 32-bit arm android emulator with clean android-10.0.0_r47 build from AOSP. It works when I do it as a root. But I couldn't start xtensa-lx106-elf-ar as a non-root.

How do you set up permissions and security context for the toolchain?

@4ntoine
Copy link
Author

4ntoine commented Jan 31, 2021

@jcmvbkbc I don't do it and it works with regular apps permissions on arm 64 bit (armv8a).

@4ntoine
Copy link
Author

4ntoine commented Jan 31, 2021

@jcmvbkbc BTW make sure you target android 9 for testing. Android 10 has more strict security model (thus that symlink trick, fore more infor about the trick here)

@jcmvbkbc
Copy link
Owner

Ok, after disabling selinux with setenforce 0 non-root user was able to run xtensa-lx106-elf-ar cru z.a BearSSLHelpers.cpp.o successfully.
I've also noticed that I could run commands under strace as non-root user with selinux enabled. Would you like to try it?

@4ntoine
Copy link
Author

4ntoine commented Jan 31, 2021

@jcmvbkbc Do i understand you correctly that it will work with just disabling selinux (i guess it requires root anyway)? sure i will do my best, please provide set-by-step guide on how to do it. My main concern is that even if it works, there is some difference between 32 and 84 toolchain that makes a difference (and requires some additional configuration on 32 but doe snot on 64)

@jcmvbkbc
Copy link
Owner

My main concern is that even if it works, there is some difference between 32 and 84 toolchain that makes a difference

Maybe. My suggestion is to try to run the reproducer under the strace to see where exactly it fails.
I guess that doing adb push strace /data/local/tmp with the strace binary available here followed by adb shell and chmod +x /data/local/tmp/strace should give you working strace.

@jcmvbkbc
Copy link
Owner

I've put xtensa-lx106-elf-ar to /data/local/tmp and xtensa-lx106-elf-ar cru z.a BearSSLHelpers.cpp.o runs successfully under non-root with selinux enabled. Looks like this system doesn't reproduce your issue.

@4ntoine
Copy link
Author

4ntoine commented Jan 31, 2021

I'm not sure how it works but on Android 10 the app can't execute the executables even in it's own app sandbox (/data/data/%package%, only from /data/app/%package% - that's why the trick with executables put as shared libs). So if you can execute anything from data/local/tmp i'm afraid it does not reproduce the real non-rooted device.

Please provide step-by-step guide what should i try?

@jcmvbkbc
Copy link
Owner

if you can execute anything from data/local/tmp

My understanding is that's because it is specifically designated to the user shell, who's also the user that adb shell runs under.
I would suggest that you try the same thing: put your reproducer binaries to the /data/local/tmp and run the reproducer from the adb shell there. Success would mean that the issue is in the permissions/selinux setup.

@4ntoine
Copy link
Author

4ntoine commented Jan 31, 2021

@jcmvbkbc That surprisingly worked (on a real production device):

> adb shell /data/local/tmp/libesp8266_xtensa-lx106-elf-ar.so cru /data/local/tmp/z.a /data/local/tmp/BearSSLHelpers.cpp.o
/tmp developer
> ls -l /data/local/tmp
ls: /data/local/tmp: No such file or directory
/tmp developer
> adb shell ls -l /data/local/tmp
total 2074
-rw-rw-rw- 1 shell shell  436800 2021-01-31 23:44 BearSSLHelpers.cpp.o
drwxrwxr-x 2 shell shell    3488 2021-01-31 23:44 device-explorer
-rwxrwxrwx 1 shell shell 1234300 2021-01-23 22:48 libesp8266_xtensa-lx106-elf-ar.so
drwxrwxrwx 4 shell shell    3488 2021-01-31 23:22 perfd
-rw-rw-rw- 1 shell shell  439982 2021-01-31 23:48 z.a

What does it mean? Why does it work for 64 bit out of box? (from app dir)? Can it be done to work the same way on 32 bit (working only in the app sandbox dir)?

@jcmvbkbc
Copy link
Owner

What does it mean?

That the toolchain binaries that you have are capable of working correctly on your system. My interpretation is that the original issue is in the permissions/selinux setup.

Why does it work for 64 bit out of box? Can it be done to work the same way on 32 bit?

These questions are about the android and its application permissions of which I don't know much. I would expect that there might be a tool for debugging this kind of issues.

@jcmvbkbc jcmvbkbc changed the title Toolchain for ARM 32 bit produces corrupted files Toolchain for ARM 32 bit fails on android host with exit code 159 Jan 31, 2021
@4ntoine
Copy link
Author

4ntoine commented Feb 1, 2021

@jcmvbkbc Well, regular android app on Android 10 can't put files to /data/local/tmp (i've just checked and got AccessDeniedException) so this workaround does not work. That's still strange that some special tricks are required to make it working on 32..

@4ntoine
Copy link
Author

4ntoine commented Feb 2, 2021

@jcmvbkbc Ok, so what's the plan? I can prepare a simple app that extracts the sdk and compiles the final binary for the testing, if it helps.

@jcmvbkbc
Copy link
Owner

jcmvbkbc commented Feb 2, 2021

so what's the plan?

I don't have any further plan. Specifically I'm not planning to debug android application permissions/selinux configuration.

You may want to find out specific syscall that gets SIGSYS and figure out why that happens. Like I said I don't know android well enough to give you any guidance on this, the only thing that comes to my mind is that perhaps generic selinux policy debug techniques could be useful. When you know which syscall fails and for what reason you may be able to find a fix or workaround for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants