error loading module 'lua-utf8'

[no-vici]

Compatibility issues with lua-utf8 have forced me to downgrade SILE to SILE v0.14.9 (Lua 5.4) after each system update. Has anyone else experienced this problem?

Your sile document is being processed...

/usr/bin/luajit: error loading module 'lua-utf8' from file '/usr/lib/lua/5.4/lua-utf8.so':
	/usr/lib/lua/5.4/lua-utf8.so: undefined symbol: lua_callk
stack traceback:
	[C]: at 0x7f15fe737d70
	[C]: in function 'require'
	/usr/share/sile/core/sile.lua:24: in main chunk
	[C]: in function 'require'
	/usr/bin/sile:26: in main chunk
	[C]: at 0x55c110f25cd0
lua: ebook-builder-v0.lua:110: Command failed: sile Diary_of_Saigon_1975.sil
Exit: exit
Status: 1
...

[alerque]

What OS are you an and what install methods are you using for Lua Rocks and SILE?

Also why are you using Lua 5.4 specifically? It will still work of course, but we've switched the default/recommendation to LuaJIT because it is just so much faster.

[no-vici]

Thank you for replying.

I'm running Manjaro-Sway and I installed SILE and LuaRocks by pacman -S sile luarocks

I have lua51 and luajit 2.1.1.17 (drop-in replacement for Lua 5.1) installed. Lua 5.4 was shipped with my fresh installation of the OS. Should I go back to Lua 5.3?

[alerque]

Something doesn't add up there. The Manjaro packages are just going to be copies of the Arch Linux packaging, which I build. The default Lua used for the Arch packages is LuaJIT, so installing pacman -S sile should get you a LuaJIT package and a bunch of lua51-* packages for the LuaRocks dependencies. This should include lua51-luautf8.

The error messages you are showing in your first post however are showing an a version of SILE that is build against Lua 5.4. You appear to have mixed and matched packages somehow. I don't know what your "downgrade" process is (since no such thing is really supported in Arch/Manjara) but I believe you have cross wired something and are not really using the packages & their dependencies as they were built.

I suggest a pacman -Syu to get everything freshened up into a supported fully updated set, then lets look at pacman -Qi sile to understand which build you have, sile --version to review what SILE thinks it is build against, and perhaps sile -e 'SU.dump(pl.stringx.split(package.path, ";"))' -e 'os.exit()' to have a look at the module loader paths that got configured.

[alerque]

Two other things to confirm: have you ever done an installation from source too or just the system package manager? Are you by chance trying to run SILE from inside a checkout of the SILE sources?

[no-vici]

Something doesn't add up there. The Manjaro packages are just going to be copies of the Arch Linux packaging, which I build. The default Lua used for the Arch packages is LuaJIT, so installing pacman -S sile should get you a LuaJIT package and a bunch of lua51-* packages for the LuaRocks dependencies. This should include lua51-luautf8.

Manjaro uses mainly the same repositories as Arch. And I have that bunch of lua51-* and obviously lua51-luautf8. I don't get why the new luajit always looks for Lua 5.4's lua-utf8.so. I went over to luajit's github and found no related problem.

lua51-luautf8 /usr/
lua51-luautf8 /usr/lib/
lua51-luautf8 /usr/lib/lua/
lua51-luautf8 /usr/lib/lua/5.1/
lua51-luautf8 /usr/lib/lua/5.1/lua-utf8.so
lua51-luautf8 /usr/lib/luarocks/
lua51-luautf8 /usr/lib/luarocks/rocks-5.1/
lua51-luautf8 /usr/lib/luarocks/rocks-5.1/luautf8/
lua51-luautf8 /usr/lib/luarocks/rocks-5.1/luautf8/0.1.5-1/
lua51-luautf8 /usr/lib/luarocks/rocks-5.1/luautf8/0.1.5-1/doc/
lua51-luautf8 /usr/lib/luarocks/rocks-5.1/luautf8/0.1.5-1/doc/LICENSE
lua51-luautf8 /usr/lib/luarocks/rocks-5.1/luautf8/0.1.5-1/doc/README.md
lua51-luautf8 /usr/lib/luarocks/rocks-5.1/luautf8/0.1.5-1/luautf8-0.1.5-1.rockspec
lua51-luautf8 /usr/lib/luarocks/rocks-5.1/luautf8/0.1.5-1/rock_manifest
lua51-luautf8 /usr/share/
lua51-luautf8 /usr/share/licenses/
lua51-luautf8 /usr/share/licenses/lua51-luautf8/
lua51-luautf8 /usr/share/licenses/lua51-luautf8/LICENSE

The error messages you are showing in your first post however are showing an a version of SILE that is build against Lua 5.4. You appear to have mixed and matched packages somehow. I don't know what your "downgrade" process is (since no such thing is really supported in Arch/Manjara) but I believe you have cross wired something and are not really using the packages & their dependencies as they were built.

The SILE version I installed is 0.14.13-1 built by you on Arch's packages. I do run pamac upgrade from time to time and the system updates itself. The downgrade process I used is DOWNGRADE_FROM_ALA=1 sudo downgrade sile a list of sile's versions from the system cache shows up.

I suggest a pacman -Syu to get everything freshened up into a supported fully updated set, then lets look at pacman -Qi sile to understand which build you have, sile --version to review what SILE thinks it is build against, and perhaps sile -e 'SU.dump(pl.stringx.split(package.path, ";"))' -e 'os.exit()' to have a look at the module loader paths that got configured.

Running pacman -Qi sile after sudo pacman -Syu, I got:

Running sile --version, I got the same error as mentioned:

/usr/bin/luajit: error loading module 'lua-utf8' from file '/usr/lib/lua/5.4/lua-utf8.so':
	/usr/lib/lua/5.4/lua-utf8.so: undefined symbol: lua_callk
stack traceback:
	[C]: at 0x7fc080691d70
	[C]: in function 'require'
	/usr/share/sile/core/sile.lua:24: in main chunk
	[C]: in function 'require'
	/usr/bin/sile:26: in main chunk
	[C]: at 0x5646f35eccd0

Thank you so much for your time looking into this. I may have to reinstall my system.

[no-vici]

Two other things to confirm: have you ever done an installation from source too or just the system package manager? Are you by chance trying to run SILE from inside a checkout of the SILE sources?

SILE before v0.14.11 worked beautifully, so I did not dive into its source and try to install it from there, and I've never tried to run SILE from inside a checkout either. The distro's package manager does a very good job of updating everything though.

[alerque]

First, the output of the path dump I requested above would be useful:

$ /usr/bin/sile -e 'SU.dump(pl.stringx.split(package.path, ";"))' -e 'os.exit()'

Also which sile might suggest if you're actually running the system installed SILE package.

[alerque]

It looks to me like you somehow have a partial SILE install in your personal user's home directory, possibly installed with luarocks install sile. Possibly those fragments are just from resilient or some other 3rd party library. Something about those is probably cross wired with SILE itself. In particular to use 3rd party modules you need to install them using the same version of Lua as SILE uses. But even with something from a different Lua version installed on the system or in your home user, that doesn't explain why running SILE is finding the wrong bits.

Do you have anything in your user's shell RC files for using local rocks? Fore xample eval $(luarocks path) or anything along those lines?

[no-vici]

$ /usr/bin/sile -e 'SU.dump(pl.stringx.split(package.path, ";"))' -e 'os.exit()'

The ouput of the above command with older version of SILE:

SILE v0.14.9 (Lua 5.4)
{
  "./lua-libraries/?.lua",
  "./lua-libraries/?/init.lua",
  "./?.lua",
  "./?/init.lua",
  "/usr/share/lua/5.4/sile/?.lua",
  "/usr/share/lua/5.4/sile/?/init.lua",
  "/usr/lib/lua/5.4/sile/?.lua",
  "/usr/lib/lua/5.4/sile/?/init.lua",
  "./sile/?.lua",
  "./sile/?/init.lua",
  "~/.luarocks/share/lua/5.4/sile/?.lua",
  "~/.luarocks/share/lua/5.4/sile/?/init.lua",
  "/usr/lib/sile/lua-libraries/?.lua",
  "/usr/lib/sile/lua-libraries/?/init.lua",
  "/usr/lib/sile/?.lua",
  "/usr/lib/sile/?/init.lua",
  "/usr/share/sile/lua-libraries/?.lua",
  "/usr/share/sile/lua-libraries/?/init.lua",
  "/usr/share/sile/?.lua",
  "/usr/share/sile/?/init.lua",
  "/usr/share/lua/5.4/?.lua",
  "/usr/share/lua/5.4/?/init.lua",
  "/usr/lib/lua/5.4/?.lua",
  "/usr/lib/lua/5.4/?/init.lua",
  "~/.luarocks/share/lua/5.4/?.lua",
  "~/.luarocks/share/lua/5.4/?/init.lua"
}% 

"~/.luarocks/share/lua/5.4/sile/?.lua",
  "~/.luarocks/share/lua/5.4/sile/?/init.lua",

These above 2 lines maybe the culprits.

Switching to 0.14.13-1, I got:

/usr/bin/luajit: error loading module 'lua-utf8' from file '/usr/lib/lua/5.4/lua-utf8.so':
	/usr/lib/lua/5.4/lua-utf8.so: undefined symbol: lua_callk
stack traceback:
	[C]: at 0x7fe5fa702d70
	[C]: in function 'require'
	/usr/share/sile/core/sile.lua:24: in main chunk
	[C]: in function 'require'
	/usr/bin/sile:26: in main chunk
	[C]: at 0x55686a549cd0

Also which sile might suggest if you're actually running the system installed SILE package.

which sile : /usr/bin/sile

[no-vici]

Do you have anything in your user's shell RC files for using local rocks? Fore xample eval $(luarocks path) or anything along those lines?

Thank you so much, this is the ANSWER! I commented the line:
eval $(luarocks --local path) in my .zshenv out and the problem disappears.

sile --version spits: SILE v0.14.13 (LuaJIT 2.1.1700008891)

Now, SILE does not see resilient.book class. I have to find another solution for that though. Maybe install the third party modules at the system level?

Thank you so so much for helping me solve my problem.

[alerque]

You can install the Resiliant library with a matching Lua version. I'll lookup the command later but it is probably something like luarocks --lua-version 5.1 install --local, and then you can add your paths with something like eval $(luarocks --lua-version 5.1 patch --local). However I haven't tested that so the exact command might vary. Also instead of using --local for this I would actually recommend using --tree lua_modules or something like that and install the Rocks local to your project. Installing them to the system or your user makes for trouble when you want to run a project that does not use those modules because silex kind of aggressively takes over some core SILE functions. SILE is designed so that you can override its internals, but that comes at the cost of playing with fire. By only installing it locally to a project and only updating your patch to the modules when using it you'll eliminate some of that pain.

[Omikhleia]

@alerque Well probably resilient doesn't work with LuaJIT yet anyway - unchecked / unsupported as of yet.

[alerque]

@no-vici The issue here is your choice of --tree resiliant.sile. That's a poor choice that leads to confusion because this is not a place to install one rock, it is a place to install a whole collection of rocks and their potential dependencies. What you are ending up with is resiliant.sile.resiliant.sile and a bunch of other things under that namespace.

SILE comes pre-configured to look for project-local modules in a lua_modules directory, so use --tree lua_modules instead. (Also don't use --local as that is for your user space, just --tree is what you want.) Then you can skip the nonsense about loading the module yourself before the document to preload the module cache.

$ luarocks --lua-version 5.1 install --tree lua_modules resilient.sile
$ eval $(luarocks --lua-version 5.1 path --tree lua_modules)
$ echo "\\document[class=resilient.book]{test}" > test.sil
$ sile test.sil

[alerque]

@Omikhleia Given that we've been suggesting to distros for a while to use LuaJIT and at least half a dozen have done that, it might be time to officially support it. I don't think you have much to do though, I used it the other day and my impression was most stuff worked. I even submitted a PR for a few minor bits that I turned up. Also keep in mind it will be the default configuration for source installs out of the box starting with v0.15.0. I'm happy to help with any rough edges that pop up as I've been using LuaJIT it in production for all my work for a year or so now, long before I started pushing it on SILE.

[no-vici]

Thank you for your clear explanation and patience. It works like a charm now.

SILE comes pre-configured to look for project-local modules in a lua_modules directory, so use --tree lua_modules instead.

That has cleared the mist for me.

[no-vici]

Well probably resilient doesn't work with LuaJIT yet anyway - unchecked / unsupported as of yet.

It works on my system beautifully, so please keep it rolling. Your work has turned the books I'm working on look appealing ;)

[alerque]

I just checked locally and I think those commands are right. The only catch you might run into is actually an Arch Linux foible not a SILE issue per say, but in order to get luarocks --lua-version 5.1 to run correctly you might need the lua51 package installed, not just luajit.

[Omikhleia]

because silex kind of aggressively takes over some core SILE functions.

Hey, the way it's phrased makes me shiver! But nope, it's not that "aggressively" @alerque :)
It's just sile + plus earlier ports of PR submitted here - minus things that I considered were really broken.

("Agressively" in my mind is something else, which I really hope not to happen.)

[alerque]

By "aggressively" I just mean it does things like completely replacing the base typesetter and other base modules rather than just adding implementations that inherit from them. This means that when SILE upgrades or fixes or whatever roll out, anybody with silex stuff in their Lua path will still get the silex version of things without any of SILE's changes. We've intentionally let that happen so that you can do stuff like that, but the opposite of an aggressive take over would be replacing single methods rather than whole modules or creating a new module with a new name that inherits from SILE core but then overrides the things it wants to do.

At the moment I can't install a silex system package that makes it available all the time because then every project is forced to use it. I would prefer less aggressive modules that are opt-in on a per project or per sile invocation basis, but that's just me.

[alerque]

With a little sleep let me try again to clarify. By "aggressive" I don't mean that there was anything wrong or inappropriate per-se, just something that I think could be done in a less invasive way. Let me illustrate with some pseudo commands.

The way I wrote CaSILE, all the core SILE functions it changes (such as changing the line break function in the typesetter) happen by inheriting from the original, and the way the software installs itself it has to be specifically requested to run. I can pacman -S sile casile and then use both sile and casile at will. When I run sile everything is just plain vanilla SILE per the upstream project. When I run casile then some libraries get added on.

On the other hand if I install silex to the system, say with pacman -S sile sile-silex, then the fact that silex has "aggressively" taken over module paths used by sile means that any and all invocations of sile thereafter will always run the silex versions of things. I can no longer run sile and get a plain vanilla SILE, I always get the patched typesetter. Also when SILE updates, those updates don't propagate because the typesetter and other modules have been completely replaced, not just inherited from.

Again this isn't wrong per-se and SILE is intentionally designed to let the user override ANY part of the system like that, but it is also structured in a way that less aggressive tak-overs are possible. For example you could set things up so that the silex replacements are only loaded if/when certain classes are used. Rendering a document with class=book would get plain SILE, but class=resilient.book would run with the silex modifications.