Updating to Lua 5.4.2 crashes AwesomeWM

[Nuc1eoN]

Today I've updated from Lua v5.3.5 to v5.4.2 which is in the repos.

Unfortunately it seems, that update crashes the Xserver, or more specifically AwesomeWM.

This is the X crash message:

awesome: symbol lookup error: awesome: undefined symbol: lua_newuserdata

[sage-etcher]

TLDR

After upgrading Lua you have re-build Awesome and re-install Lua modules (LuaRocks stuff). Currently the stable version of LuaRocks:lgi does not have 5.4 compatibility. However, the experimental git version (non-released) does have 5.4 support, so if you manually install LGI from git, Awesome should work. However, currently I see no way of installing the needed LGI version with Alien.

awesome: symbol lookup error: awesome: undefined symbol: lua_newuserdata

The error message is due to lua_newuserdata behing defined as a preprocessor macro in version 5.4, as mentioned in this StackOverflow thread.

Full Process (so far)

Section 1: Installing Lua54 Compatible Compile Recipes

Install/update programs (and deps) for Lua54 support using Compile:

• Lua >= 5.4.0, < 5.5.0
• LuaRocks >= 3.5.0
• CMake >= 3.18.0

Re-Source Bash's Environment

It is important that LUA_PATH_5_4 and LUA_CPATH_5_4 are set correctly later, otherwise startx will complain. Bash will not source the new versions of Lua or LuaRocks's Environment files without a log-out or reboot. Either of them should work, or we can source them manually as follows. The first command re-sources all the Environment files, while the second commands re-source only Lua and LuaRocks

for f in $goboEnvironment/*--*; source $f
or
source /Programs/Lua/Current/Resources/Environment
source /Programs/LuaRocks/Current/Resources/Environment

Section 2: Installing LuaRocks Modules

Gobo provides a handful of LuaRocks modules to make AwesomeWM look and feel as it does. By default these modules are installed for Lua53, but now that Lua54 is being used, they have to be re-installed.

These modules should be easily installed using the sudo Alien --install LuaRocks:PACKAGE script:

• gobo-awesome
• gobo-awesome-alttab
• gobo-awesome-battery
• gobo-awesome-bluetooth
• gobo-awesome-gobonet
• gobo-awesome-light
• gobo-awesome-sound
• gobo-awesome-screenlock
• luafilesystem

If you're lazy like me, you could use this instead, to install all of them

for pkg in luafilesystem gobo-awesome{,-{alttab,battery,bluetooth,gobonet,light,sound}}; do
   sudo Alien-LuaRocks --install $pkg
done

Section 3: Installing LuaRocks Git Modules

Awesome requires LGI to be installed, however LuaRocks:lgi 0.9.2-1 does not have Lua54 support yet. If we want to use Lua 5.4 we will have to install the GIT version instead.

git clone https://github.com/lgi-devs/lgi.git
cd lgi
sudo luarocks make

Section 4: Compiling Awesome

After all of this you should be able to compile Awesome as normal.

sudo Compile Awesome 4.3
startx

Notes

The versions listed in Section 1 are based on when each software added Lua54 support, I have not tested them individually; durring testing I have used the following software/versions:

• Lua 5.4.3
• LuaRocks 3.9.2
• CMake 3.22.2
• Awesome 4.3

[Nuc1eoN]

Sounds like quite the rabbit hole, good work so far 💪

Inside CMake, Lua cannot find the Alien modules; Awesome cannot find LGI, so the build fails. However outside of CMake Lua has no issues finding them with require()???

Maybe try running Compile --debug, IIRC it will list which directories get pulled into the build environment. Also do not use --pure. Sounds like for some reason it doesn't include Aliens? Maybe it has to be added to the Dependencies file? Eg.
LuaRocks:LGI (out of my head, actual phrase may vary).

Is LGI properly installed? Is it installed inside /Programs (and properly symlinked) or inside the Aliens directory?

Right after Compiling, Lua's package.path variable doesn't include the correct Alien paths, a reboot seems to fix this.

This is probably worth investigating at some point.

[sage-etcher]

This was done earlier, I'm not at the system atm, and am done for the night. It's putting up quite a hasle though lmao

Is LGI properly installed? Is it installed inside /Programs (and properly symlinked) or inside the Aliens directory?

luarocks make installs the package into the Aliens subtree. It installs same as Alien --install LuaRocks:PACKAGE would, just from source instead.

earlier testing was done with installing LGI in /Programs. this works almost flawlessly, asside from that Compile complains that LuaRocks:lgi doesn't exist so you'd gotta either skip the dependency or add it to the exclude list. My concern with it though, once the LuaRocks version is updated, we'll be back at having the current issues.

Sounds like for some reason it doesn't include Aliens? Maybe it has to be added to the Dependencies file?

LuaRocks:lgi is already added in the dependency file

I wasn't running with --pure

I tested with Compile --debug and none of the Alien tree got pulled in by Runner.

The build fails on lgi-check (a build tool from Awesome). I was playing around with it some, and lgi-check, passes when ran normally, I ran it under Runner using Awesome's Dependency file and it passed there too, suprisingly?? but then it fails under Compile. This was happening no matter the version of Lua i was running.

though Lua is quite nice in showing what directories it looks in, and it appeared that in Compile Lua is only checking the /System/Index tree. So I figured it was going based on the default CPATH and LPATHs specified in Lua's luaconf.h file, so i wrote a patch for that, tested it and still only checking in /usr/include directories.

I have no clue where it's pulling it's package.path/package.cpath from. The lua shared libraries and installed headers, all included the alien subtree as defaults, but then still it decided to be a pain.

I investicated the lgi-check.c file aswell, and it was REAL BASIC, nothing that should be causing an issue.

Tried defining LUA_PATH and LUA_CPATH variables in the Recipe environment and still nothing changed

I tested it with the --no-sandbox flag in the Recipe aswell, but the issue still persisted. I'm at a bit a loss atm

I plan on investiging under the hood what Compile is doing and playing it by ear from there, but that's for tommorow.

This is probably worth investigating at some point.

and yes investigating that is a plan for the future, I'd like to get things working first, but yes.

one note, if the luaconf.h file is patched to include the Alien subtree as a default, then the restart isn't required. However, it's not ideal, and after a restart, the Alien directories are listed twice in package.path. not good, but thought it was interesting.

[Nuc1eoN]

I tested with Compile --debug and none of the Alien tree got pulled in by Runner.

I'd probably look into this part. Compile should pull in the Aliens tree, if it doesn't then that is an issue.

earlier testing was done with installing LGI in /Programs. this works almost flawlessly, asside from that Compile complains that LuaRocks:lgi doesn't exist so you'd gotta either skip the dependency or add it to the exclude list.

Seems to make sense. If LGI is installed under /Programs rather than Aliens, then the correct Dependency declaration would be LGI, not LuaRocks:lgi.

[sage-etcher]

Part that's confusing me the most is that when Lua panics and cant find a package, it prints out every directory it looked in. Whether the directory exists or not, doesn't matter, it gets printed all the same, no file: PATH/EXPECTED_FILE. And in Compile it doesn't look like Lua is even trying to check the Aliens/LuaRocks tree. So even if eveything else is in place through Runner, if Lua isn't checking the right locations, well, it isn't gonna find jack.

I am going to look more into Runner, but it seems to me that Lua is a bigger issue here.

if i recall correctly

$ sudo Runner -d LUA543_DEPENDENCIES_FILE /Data/Compile/Sources/awesome-4.3/_build/lgi-check

succeeds, and has propper paths. So I dont think the issue is coming directly from Runner. I've only been able to replicate the issue inside Compile.

[Nuc1eoN]

Yeah I get you. Sometimes GoboLinux feels like the first steps on the Moon. But slow and steady approach will lead to the result. Anyways have a good rest.

[sage-etcher]

Okay, I think I found the cause, it was Runner related.

Case	Dependency	Result
A	only LuaRocks	PASSES
B	only Lua	FAILS
C	neither	PASSES
D	both	PASSES

Case A passes because LuaRocks lists Lua as a dependency for itself.
Case C is fairly confusing, but isn't an issue in this scenario.

When running Compile, Runner pulls in dependencies from both Compile and Awesome. Awesome list neither and Compile lists only Lua, placing us firmly in Case B.

Compile already lists Lua # used by Lua rocks, so it would seem adding LuaRocks to Compile's dep-list would make the most sense.

[sage-etcher]

Right after Compiling, Lua's package.path variable doesn't include the correct Alien paths, a reboot seems to fix this.

restarting the machine is no longer necessary, though the Environment files do need to be re-sourced by the user. Unfortunately it doesn't appear that this can be done automatically

The main comment was updated with instructions and reasoning for such (Section 1 > Re-Source Bash's Environment).

[Nuc1eoN]

Case C is fairly confusing, but isn't an issue in this scenario.

Case C is definitely fairly confusing. However it does sound familiar, although I cannot remember what the exact case was for me.

---

@hishamhm as @sage-etcher mentioned, gobo-awesome-screenlock seems to be missing from LuaRocks. Although we can install it from git, this is probably not ideal. Do you know why it has been removed? Would you kindly re-publish it to LuaRocks?

[hishamhm]

@Nuc1eoN I don't think gobo-awesome-screenlock was ever uploaded to luarocks.org! I've just cloned the repo and uploaded it there using my luarocks account.

[hishamhm]

(I've never used that package myself; I wrote the other gobo-awesome entries, but this one was made by Lucas.)

[Nuc1eoN]

I don't think gobo-awesome-screenlock was ever uploaded to luarocks.org! I've just cloned the repo and uploaded it there using my luarocks account.

Great, thank you!:)

@sage-etcher , one note regarding:

It is important that LUA_PATH_5_4 and LUA_CPATH_5_4 are set correctly later, otherwise startx will complain. Bash will not source the new versions of Lua or LuaRocks's Environment files without a log-out or reboot. Either of them should work, or we can source them manually as follows. The first command re-sources all the Environment files, while the second commands re-source only Lua and LuaRocks

for f in $goboEnvironment/*--*; source $f
or
source /Programs/Lua/Current/Resources/Environment
source /Programs/LuaRocks/Current/Resources/Environment

Maybe let's add this re-sourcing to a PostInstall section of the Lua/LuaRocks recipes? That would prevent potential future users from potential issues.

At the very least I would suggest something like post_install_message="\n\nIMPORTANT:\nYou must run 'source /Programs/Lua/Current/Resources/Environment; source /Programs/LuaRocks/Current/Resources/Environment', or reboot/re-login the system.\n"

@hishamhm Do we have some in-build/native hook in Compile that we could use to re-source the ENVIRONMENT files?

EDIT:
I have found RebuildLinks which probably does just that. So we could use that unless Hisham knows an even nicer way:)

[hishamhm]

since each process keeps its own environment, there's no real way to automatically re-source the environment in a complete way after a package upgrade. You'll always have things like terminal windows still open running the old environment, etc. For things like the window manager, which is the parent process of all graphical applications, then yeah, I think it's reasonable to require a re-login to get the whole subprocess tree refreshed.

If re-sourcing the environment is something that is needed in between building dependencies so that the whole compilation succeeds, then code could be added to Compile for it to do that for its own process (I don't recall us reloading Environment files from within Compile... maybe it does already?), but I don't think there's a way to have a more automagical solution that refreshes everything.

[Nuc1eoN]

@sage-etcher I am trying to follow your guide now in order to update my Lua/Awesome setup.

But upon running

source /Programs/LuaRocks/Current/Resources/Environment

as suggested in your guide, I am getting:

/Programs/LuaRocks/Current/Resources/Environment:export:8: not valid in this context: LUA_PATH+

It looks like there is some issue with this script:

Recipes/LuaRocks/3.9.2/Resources/Environment

Lines 1 to 19 in 542a759

	# LuaRocks Environment
	for vk in 5.1= 5.2=_5_2 5.3=_5_3 5.4=_5_4
	do
	v="${vk%=*}"
	k="${vk#*=}"
	export LUA_PATH$k+="\
	;$goboSystem/Aliens/LuaRocks/share/lua/$v/?.lua;\
	$goboSystem/Aliens/LuaRocks/share/lua/$v/?/init.lua;\
	$goboSystem/Aliens/LuaRocks/lib/lua/$v/?.lua;\
	$goboSystem/Aliens/LuaRocks/lib/lua/$v/?/init.lua"
	export LUA_CPATH$k+="\
	;$goboSystem/Aliens/LuaRocks/lib/lua/$v/?.so;\
	$goboSystem/Aliens/LuaRocks/lib/lua/$v/loadall.so"
	done
	unset v k vk

Possibly related to dcd7442 of PR #234 ?

Right now my LUA_PATH is:

$ printenv LUA_PATH
/System/Aliens/LuaRocks/share/lua/5.1/?.lua;/System/Aliens/LuaRocks/share/lua/5.1/?/init.lua;/System/Aliens/LuaRocks/lib/lua/5.1/?.lua;/System/Aliens/LuaRocks/lib/lua/5.1/?/init.lua;;

EDIT: Seems to be specific to zsh, since running it from pure bash succeeds.

EDIT2: So yes, running startx from bash succeeds, while when running startx from ZSH the awesome setup starts but is severely degraded (broken theme etc).

EDIT3: reverting dcd7442 works for zsh. However I have to run source /Programs/LuaRocks/Current/Resources/Environment manually every time before startx. Even after a reboot.

[sage-etcher]

Interesting. From the error message it appears that ZSH doesn't support export VARIABLE+="appendedvalue". I'm not at a system where I can do any testing, however it would be reasonable to assume that += may be incompatible between ZSH and Bash. If that is the case, the repo could benefit from removing instances of it where/when ever possible, for example changing it to:

export LUA_PATH$k="\
${LUA_PATH${k}}
;$goboSystem/Aliens/LuaRocks/share/lua/$v/?.lua;\
...
"

NOTE: the above code is untested, I am uncertain if the ${LUA_PATH${k}} is valid syntax.

EDIT: LUA_PATH is for Lua version 5.1; I assume you're using 5.4, in which case LUA_PATH_5_4 would need checked instead.

EDIT2: So yes, running startx from bash succeeds, while when running startx from ZSH the awesome setup starts but is severely degraded (broken theme etc).

The broken themes in ZSH are thanks to Awesome/Lua failing to find the Lua modules required for the theme, this is expected if the Environment fails to source.

EDIT2: looks like ${LUA_PATH${k}} is not valid bash

[sage-etcher]

Okay, I rewrote the LuaRocks Environment file to use less magic, however it is now fairly large. The new one is using Bash4 arrays instead of variable expansion magic like the previous.

https://github.com/sage-etcher/Recipes/blob/LuaRocks-ZSH-Compatibility/LuaRocks/3.9.2/Resources/Environment

@Nuc1eoN can you verify functionality in your ZSH environment, I'm stuck in WSL bash atm?

EDIT: I over complicated it on first pass, I put a 2nd commit with the simplified version. no arrays are needed, lol

[Nuc1eoN]

Hm, wouldn't something like:

# LuaRocks Environment

for vk in  5.1=  5.2=_5_2  5.3=_5_3 5.4=_5_4
do
  v="${vk%=*}"
  k="${vk#*=}"

export LUA_PATH$k="$LUA_PATH$k\
$goboSystem/Aliens/LuaRocks/share/lua/$v/?.lua;\
$goboSystem/Aliens/LuaRocks/share/lua/$v/?/init.lua;\
$goboSystem/Aliens/LuaRocks/lib/lua/$v/?.lua;\
$goboSystem/Aliens/LuaRocks/lib/lua/$v/?/init.lua"

export LUA_CPATH$k="$LUA_CPATH$k\
$goboSystem/Aliens/LuaRocks/lib/lua/$v/?.so;\
$goboSystem/Aliens/LuaRocks/lib/lua/$v/loadall.so"

done
unset v k vk

suffice?
If we want the curly braces, then ${LUA_PATH$}${k} (rather then ${LUA_PATH${k}}) should be valid syntax?

In my testing zsh, as well as bash could source this correctly (although I am not yet sure how to best handle the semicolons).

In any case I would try your solution, but got difficulties reproducing this issue right now. 😅

For some reason even if I delete /P/LuaRocks/Current/Resources/Environment and /P/Lua/Current/Resources/Environment my awesome configuration starts fine! (maybe that's a good thing)

And for instance my LUA_PATH_5_4 shows:

/System/Aliens/LuaRocks/share/lua/5.4/?.lua;/System/Aliens/LuaRocks/share/lua/5.4/?/init.lua;/System/Aliens/LuaRocks/lib/lua/5.4/?.lua;/System/Aliens/LuaRocks/lib/lua/5.4/?/init.lua;;

I do not understand the magic yet :)

[sage-etcher]

no not quite, the original is using variable expansions (as it is called when used in Windows Batch, I am unsure what name bash uses for it). This means that while k is equal to _5_4, we write to variable LUA_PATH_5_4, and while k is empty it writes to LUA_PATH (version 5.1 path). however to my knowledge variable expansion in bash is only allowed in writing to a variable, not in reading from it.

and so, to modify the above example

k="_5_4"
LUA_PATH="version51path"
LUA_PATH_5_4="version54path"
LUA_PATH$k="$LUA_PATH$k;appendedLuaRocksPath;;"

echo "LUA_PATH = ${LUA_PATH}"
echo "LUA_PATH_5_4 = ${LUA_PATH_5_4}

# outpus:
# LUA_PATH = version51path
# LUA_PATH_5_4 = version51path_5_4;appendedLuaRocksPath;;

# intended output should be
# LUA_PATH = version51path
# LUA_PATH_5_4 = version54path;appendedLuaRocksPath;;

this output is because within the string "$LUA_PATH$k;appendedLuaRocksPath;;" every variable is inserted left to right separately from one another, so $LUA_PATH expands to version51path, and $k expands separately to _5_4. The original only worked because in export LUA_PATH$k="..." variable k is expanded before the variable name is assigned.

The original code had frustrated me a bit initially because of bash's poor handling of variable expansion, but it was left because it worked and wasn't causing any issues, so long as += was permitted.

oh and btw, incase k and v are unclear in the original code, for pair kv="key=value", the bash parameter expansion says that k="key" and v="value". This is the case for

for vk in  5.1=  5.2=_5_2  5.3=_5_3 5.4=_5_4
do
  v="${vk%=*}"
  k="${vk#*=}"

done

EDIT:

For some reason even if I delete /P/LuaRocks/Current/Resources/Environment and /P/Lua/Current/Resources/Environment my awesome configuration starts fine! (maybe that's a good thing)

This is the case because the environment variables are read once, so the changes to the Environment files will only be noticed on resourcing them, while deletions will only be noticed on a system reboot.

EDIT2: the $LUA_PATH$k and ${LUA_PATH}${k} are valid syntax, however it would constitute a logical error instead.

[sage-etcher]

I would suggest something simple with no funky half supported magic, like the following. should be easy to follow and easy to maintain. its not fully featured but it should work.

gen_luarocks_path ()
{
   local version="$1"

   local luarocks_path="\
${goboSystem}/Aliens/LuaRocks/share/lua/${version}/?.lua;\
${goboSystem}/Aliens/LuaRocks/share/lua/${version}/?/init.lua;\
${goboSystem}/Aliens/LuaRocks/lib/lua/${version}/?.lua;\
${goboSystem}/Aliens/LuaRocks/lib/lua/${version}/?/init.lua"

  echo "${luarocks_path}"
}

gen_luarocks_cpath ()
{
   local version="$1"

   local luarocks_cpath="\
${goboSystem}/Aliens/LuaRocks/lib/lua/${version}/?.so;\
${goboSystem}/Aliens/LuaRocks/lib/lua/${version}/loadall.so"

  echo "${luarocks_cpath}"
}


export LUA_PATH="${LUA_PATH};$(gen_luarocks_path 5.1)"
export LUA_PATH_5_2="${LUA_PATH_5_2};$(gen_luarocks_path 5.2)"
export LUA_PATH_5_3="${LUA_PATH_5_3};$(gen_luarocks_path 5.3)"
export LUA_PATH_5_4="${LUA_PATH_5_4};$(gen_luarocks_path 5.4)"

export LUA_CPATH="${LUA_CPATH};$(gen_luarocks_cpath 5.1)"
export LUA_CPATH_5_2="${LUA_CPATH_5_2};$(gen_luarocks_cpath 5.2)"
export LUA_CPATH_5_3="${LUA_CPATH_5_3};$(gen_luarocks_cpath 5.3)"
export LUA_CPATH_5_4="${LUA_CPATH_5_4};$(gen_luarocks_cpath 5.4)"