Make of ACK frozen on MSYS2

[Wysardry]

I was using the MSYS2-MINGW-64 window to make ACK for the first time and it has frozen (for 30 minutes so far) on step 2143 out of 8876.

The last few lines of output look like this:-

[1285/8876] C:/msys64/tmp/ack-build/obj/util/cmi...-build/obj/lang/cem/libcc.ansi/ctype_tab/char.c
Warning: redefinition of index 12
Warning: redefinition of index 10
Warning: redefinition of index 13
Warning: redefinition of index 9
Warning: redefinition of index 11
[2143/8876] cp plat/em/libsys/access.e C:/msys64...4/tmp/ack-build/obj/plat/em/libsys/srcs/write.e

I previously installed git and base-devel (which includes lexx and yacc).

Prior to cloning the repository, I updated everything using pacman -Syu, then installed ninja and mingw-w64-x86_64-lua via pacman.

These are the commands I used to start the build:-

git clone https://github.com/davidgiven/ack.git
cd ack
make

Does anyone have an idea of what might be wrong? I'll leave the MSYS window open for a while longer in case it is actually still doing something.

Edit: I left MSYS running for three more hours, then exited. I then tried the make command again and my anti-virus program started to display a load of "checking new file..." windows, so I exited again.

I guessed the (Avast) anti-virus program caused the previous session to freeze, so I added the MSYS directory to its list of exceptions so it wouldn't keep doing a realtime scan every build.

So, I rebooted my computer to end any possible running processes, deleted the ack directory and the contents of the MSYS tmp directory, then opened MSYS again.

I repeated my earlier commands and the build seemed to be going well until the build stopped itself with the following error:-

[8422/8876] ACKDIR=C:/msys64/tmp/ack-build/stagi...obj/examples/hilo_b_cpm/main/hilo_b_cpm/main.a
FAILED: C:/msys64/tmp/ack-build/obj/examples/hilo_b_cpm/hilo_b_cpm
ACKDIR=C:/msys64/tmp/ack-build/staging C:/msys64/tmp/ack-build/staging/bin/ack -mcpm -.b -o C:/msys64/tmp/ack-build/obj/examples/hilo_b_cpm/hilo_b_cpm C:/msys64/tmp/ack-build/obj/examples/hilo_b_cpm/main/hilo_b_cpm/main.a
C:/msys64/tmp/ack-build/staging/lib/ack/em_led: C:/msys64/tmp/ack-build/staging/share/ack/cpm/libem.a: C:/msys64/tmp/ack-build/staging/share/ack/cpm/libem.a: no table of contents (fatal)
C:\msys64\tmp\ack-build\staging\bin\ack.exe: C:/msys64/tmp/ack-build/staging/lib/ack/em_led died with signal 1
[8427/8876] ACKDIR=C:/msys64/tmp/ack-build/stagi...uild/obj/examples/hilo_b_linuxppc/platstamp -O6
ninja: build stopped: subcommand failed.
make: *** [Makefile:104: build-plus-goals] Error 1

As you can see, it got a lot further this time and stopped with an error message, but I'm not sure what to do to fix it.

[davidgiven]

Yes, something very odd seems to be going on --- for some reason .a files never get symbol tables attached to them on Windows, but they work fine everywhere else! I'm investigating. I'd suggest trying WSL or WSL2 instead.

[Wysardry]

Thanks. At least I know the problem is not with my setup and that you're looking into it.

I tried installing via WSL2 (Debian version) which additionally needed Flex and Bison to be installed as they aren't included in base-devel, even though they are in the MSYS version.

The final message after the tests was "HAPPY FACE" so it looks like that worked.

[davidgiven]

I found the issue --- one of the standard libc functions doesn't work properly in msvcrt. I've added some proper error checking so it reports this, but unfortunately the only real solution is to switch to the UCRT64 build environment. Then it builds and runs perfectly on my local system.

Unfortunately the CI build still doesn't work, and I still don't know why.

[Wysardry]

I just tried using the UCRT64 environment instead and had to install a different version of lua, as the binary packages are different.

After fixing that problem, I got the following when running make from inside the ack directory:-

UCRT64 ~/ack
$ make
ninja: error: unknown target '+ack'
make: *** [Makefile:104: build-plus-goals] Error 1

Edit: I removed all files from the "ack" and "tmp" directories before cloning the repository again to make sure I had the lastest files.

Edit 2: I also tried using the CLANG64 version (after installing a suitable Lua package) and got the same error as above.

Edit 3: I also tried the standard MSYS2 shortcut (gcc/cygwin), MINGW32 and MINGW64 (again) with the same result as above.

[davidgiven]

make clean; make. If it doesn't find the Lua script, it ends up with an incomplete generated makefile and doesn't work. (The build system needs an overhaul. Unfortunately the ACK is ridiculously hard to build.)

The definitive set of packages is in the CI file, here: https://github.com/davidgiven/ack/blob/default/.github/workflows/ccpp.yml Bear in mind that you can't use UCRT64 unless you launch it with the UCRT64 shortcut, or else the paths won't be set up. Smooth, the process isn't.

[Wysardry]

Thanks. That list of packages helped, as they aren't all in the list of requirements on the main page.

It is now building. It will be a while before it completes.

I copied the MSYS shortcuts to my desktop to make life easier. It doesn't help that some packages install for all instances and others are specific to each environment.

Edit: The build failed again.

[1799/8876] C:/msys64/tmp/ack-build/obj/util/cmi...-build/obj/lang/cem/libcc.ansi/ctype_tab/char.c
Warning: redefinition of index 12
Warning: redefinition of index 10
Warning: redefinition of index 13
Warning: redefinition of index 9
Warning: redefinition of index 11
[2663/8876] ACKDIR=C:/msys64/tmp/ack-build/stagi....o lang/cem/libcc.ansi/sys/stdio/tmpfile.c  -O6
FAILED: C:/msys64/tmp/ack-build/obj/lang/cem/libcc.ansi/lib_linuxmips/tmpfile/tmpfile.o
ACKDIR=C:/msys64/tmp/ack-build/staging C:/msys64/tmp/ack-build/staging/bin/ack -mlinuxmips -c.o -o C:/msys64/tmp/ack-build/obj/lang/cem/libcc.ansi/lib_linuxmips/tmpfile/tmpfile.o lang/cem/libcc.ansi/sys/stdio/tmpfile.c  -O6
"ack.ack-31f6080c.so", line 17: non-terminated string
C:\msys64\tmp\ack-build\staging\bin\ack.exe: C:/msys64/tmp/ack-build/staging/lib/ack/linuxmips/as died with signal 1
[2668/8876] ACKDIR=C:/msys64/tmp/ack-build/stagi.../lang/cem/libcc.ansi/ctype_files/isalpha.c  -O6
ninja: build stopped: subcommand failed.
make: *** [Makefile:104: build-plus-goals] Error 1

[davidgiven]

Technically the Windows build doesn't work yet --- it got added to the CI system, I saw that it was failing, and didn't get round to fixing it. So this isn't a regression!

The error you're seeing is the one that shows up on the CI builds which I can't replicate, so it's really good that you're seeing it. Could you try:

ACKDIR=C:/msys64/tmp/ack-build/staging C:/msys64/tmp/ack-build/staging/bin/ack -mlinuxmips -c.o -o C:/msys64/tmp/ack-build/obj/lang/cem/libcc.ansi/lib_linuxmips/tmpfile/tmpfile.o lang/cem/libcc.ansi/sys/stdio/tmpfile.c  -O6 -v -v -t

This will generate a tempfile.so in the current directory; could you attach it, please?

[Wysardry]

Okay, I had to add a .txt extension to the filename for GitHub to allow it to be uploaded.

tmpfile.so.txt

There are quite a few files starting with tmpfile with the same timestamp.

[davidgiven]

Thank you! That was what I needed to fix it. There was a buffer overrun which on Linux and my Windows installation was reading zeroes; on your system and the CI build servers it was reading garbage. But it's done. If you pull from git you should get the fixed version.

Yes, the ACK uses temporary files a lot --- it's how it managed to run on Minix on the 8086 with 128kB maximum per process. You need a fast filesystem.

[Wysardry]

No problem. At least I've contributed something, even though I don't really know what I'm doing.

Does this mean I need to use the UCRT64 version again, or will this have fixed the MINGW64 version too?

[davidgiven]

You still have to use the UCRT64 version, unfortunately. There's a showstopping bug in tmpfile for the mingw version (that's Microsoft's fault, not mingw's). However, now that the CI builds work, it should be possible to generate an installer. Let me have a look...

[Wysardry]

That's okay, I don't have a preference. It's just that MINGW seems to be used more for other projects I've looked at, Nelua being one of them.

I'll try again and let you know what happens. It may take a while to reach the end though.

[davidgiven]

We have automatically built installers! I need to tell it to upload them to the release page, but here's one. It may even work: https://github.com/davidgiven/ack/actions/runs/8056011291/artifacts/1276801105

[Wysardry]

Does that mean there will be a nightly build for Windows that you can just unpack to a directory and use?

z88dk has that, and it makes life so much easier (I'm assuming for the devs too as they can see when something fails).

The building process is on step 3385/8876 so far, btw.

[davidgiven]

Here we go --- nightly builds: https://github.com/davidgiven/ack/releases/tag/dev

[Wysardry]

The build reached the last step, but didn't seem to run any tests or print "HAPPY FACE" as the WSL2 version did.

Nightly builds will make the process so much easier, even when building from source, as Windows users will know the zipped source will compile if there's an installer file from the same day.