Trouble getting jpm build to work on nix

[xrd]

I have janet and jpm installed on nix. I'm trying to build the example program from https://github.com/bakpakin/littleserver

I think there is an issue where the root where jpm is installed thinks it should use a janet relative to jpm, rather than the janet binary. I cannot figure out the right combination of switches to tell jpm to use janet. And, it cannot find janet.h, as is obvious from the deps/build commands below. For example, I see this as an include path, which does not exist: -I/nix/store/hfimcza6af6qd1944ngf22xqbiayrrw6-jpm-0.0.2/include/janet. I thought I could use some combination of --janet-cflags or --cflags but I'm unclear whether I use the janet one or the regular c flags. And, I wonder if there is a better way (perhaps a top level environment variable) which better configures jpm.

[nix-shell:~/Projects/janet-playground/littleserver]$ ls -la $(which jpm)
lrwxrwxrwx 1 root root 61 Jan  1  1970 /run/current-system/sw/bin/jpm -> /nix/store/hfimcza6af6qd1944ngf22xqbiayrrw6-jpm-0.0.2/bin/jpm

[nix-shell:~/Projects/janet-playground/littleserver]$ ls -la $(which janet)
lrwxrwxrwx 1 root root 66 Jan  1  1970 /run/current-system/sw/bin/janet -> /nix/store/d2d395908dfw2hl1lq7czf1ws8wky5p9-janet-1.21.2/bin/janet

$ janet -v
1.21.2-release

This command needs janet.h:

$ jpm build -l --verbose 
generating executable c source build/lserve.c from main.janet...
found native /home/chrisdawson/Projects/janet-playground/littleserver/jpm_tree/lib/circlet.so...
compiling build/lserve.c to build/build___lserve.o...
cc -c build/lserve.c -std=c99 -Wall -Wextra -I/nix/store/hfimcza6af6qd1944ngf22xqbiayrrw6-jpm-0.0.2/include/janet -I/home/chrisdawson/Projects/janet-playground/littleserver/jpm_tree/lib -O2 -o build/build___lserve.o
build/lserve.c:1:10: fatal error: janet.h: No such file or directory
    1 | #include <janet.h>
      |          ^~~~~~~~~
compilation terminated.
error: command failed with non-zero exit code 1
  in os/execute [src/core/os.c] on line 1053
  in shell [/nix/store/hfimcza6af6qd1944ngf22xqbiayrrw6-jpm-0.0.2/lib/janet/jpm/shutil.janet] on line 111, column 5
  in <anonymous> [/nix/store/hfimcza6af6qd1944ngf22xqbiayrrw6-jpm-0.0.2/lib/janet/jpm/cc.janet] on line 342, column 13
  in <anonymous> [/nix/store/hfimcza6af6qd1944ngf22xqbiayrrw6-jpm-0.0.2/lib/janet/jpm/rules.janet] on line 18, column 20
  in executor [/nix/store/hfimcza6af6qd1944ngf22xqbiayrrw6-jpm-0.0.2/lib/janet/jpm/rules.janet] (tailcall) on line 25, column 8
error: build fail
  in pdag [/nix/store/hfimcza6af6qd1944ngf22xqbiayrrw6-jpm-0.0.2/lib/janet/jpm/dagbuild.janet] (tailcall) on line 80, column 23
  in _thunk [/run/current-system/sw/bin/jpm] (tailcall) on line -1, column -1
  in cli-main [boot.janet] on line 3759, column 17

So, adding --cflags with a correct janet include path:

$ jpm deps -l --verbose  --cflags="-I/nix/store/d2d395908dfw2hl1lq7czf1ws8wky5p9-janet-1.21.2/include/janet/"
git -C /home/chrisdawson/Projects/janet-playground/littleserver/jpm_tree/lib/.cache/git__https___github.com_janet-lang_circlet.git fetch --tags origin
git -C /home/chrisdawson/Projects/janet-playground/littleserver/jpm_tree/lib/.cache/git__https___github.com_janet-lang_circlet.git fetch origin HEAD
From https://github.com/janet-lang/circlet
 * branch            HEAD       -> FETCH_HEAD
git -C /home/chrisdawson/Projects/janet-playground/littleserver/jpm_tree/lib/.cache/git__https___github.com_janet-lang_circlet.git reset --hard FETCH_HEAD
HEAD is now at 2e84f54 Merge pull request #15 from saikyun/master
git -C /home/chrisdawson/Projects/janet-playground/littleserver/jpm_tree/lib/.cache/git__https___github.com_janet-lang_circlet.git submodule update --init --recursive
cc -c circlet.c -DJANET_ENTRY_NAME=janet_module_entry_circlet -I/nix/store/d2d395908dfw2hl1lq7czf1ws8wky5p9-janet-1.21.2/include/janet/ -Wall -Wextra -I/nix/store/hfimcza6af6qd1944ngf22xqbiayrrw6-jpm-0.0.2/include/janet -I/home/chrisdawson/Projects/janet-playground/littleserver/jpm_tree/lib -O2 -o build/circlet.static.o
cc -c circlet.c -I/nix/store/d2d395908dfw2hl1lq7czf1ws8wky5p9-janet-1.21.2/include/janet/ -Wall -Wextra -I/nix/store/hfimcza6af6qd1944ngf22xqbiayrrw6-jpm-0.0.2/include/janet -I/home/chrisdawson/Projects/janet-playground/littleserver/jpm_tree/lib -O2 -fPIC -o build/circlet.o
cc -c build/circlet_lib.janet.c -I/nix/store/d2d395908dfw2hl1lq7czf1ws8wky5p9-janet-1.21.2/include/janet/ -Wall -Wextra -I/nix/store/hfimcza6af6qd1944ngf22xqbiayrrw6-jpm-0.0.2/include/janet -I/home/chrisdawson/Projects/janet-playground/littleserver/jpm_tree/lib -O2 -fPIC -o build/circlet_lib.janet.o
cc -c build/circlet_lib.janet.c -DJANET_ENTRY_NAME=janet_module_entry_circlet -I/nix/store/d2d395908dfw2hl1lq7czf1ws8wky5p9-janet-1.21.2/include/janet/ -Wall -Wextra -I/nix/store/hfimcza6af6qd1944ngf22xqbiayrrw6-jpm-0.0.2/include/janet -I/home/chrisdawson/Projects/janet-playground/littleserver/jpm_tree/lib -O2 -o build/circlet_lib.janet.static.o
circlet.c:300:12: warning: ‘is_websocket’ defined but not used [-Wunused-function]
  300 | static int is_websocket(const struct mg_connection *nc) {
      |            ^~~~~~~~~~~~
circlet.c:300:12: warning: ‘is_websocket’ defined but not used [-Wunused-function]
  300 | static int is_websocket(const struct mg_connection *nc) {
      |            ^~~~~~~~~~~~
cc -I/nix/store/d2d395908dfw2hl1lq7czf1ws8wky5p9-janet-1.21.2/include/janet/ -Wall -Wextra -I/nix/store/hfimcza6af6qd1944ngf22xqbiayrrw6-jpm-0.0.2/include/janet -I/home/chrisdawson/Projects/janet-playground/littleserver/jpm_tree/lib -O2 -o build/circlet.so build/circlet.o build/mongoose.o build/circlet_lib.janet.o -shared -pthread
ar rcs build/circlet.a build/circlet.static.o build/mongoose.static.o build/circlet_lib.janet.static.o
generating /home/chrisdawson/Projects/janet-playground/littleserver/jpm_tree/lib/.manifests/circlet.jdn...
git remote get-url origin
git rev-parse HEAD
Installed as 'circlet'.
copying build/circlet.so to /home/chrisdawson/Projects/janet-playground/littleserver/jpm_tree/lib...
cp -rf build/circlet.so /home/chrisdawson/Projects/janet-playground/littleserver/jpm_tree/lib
copying build/circlet.meta.janet to /home/chrisdawson/Projects/janet-playground/littleserver/jpm_tree/lib...
cp -rf build/circlet.meta.janet /home/chrisdawson/Projects/janet-playground/littleserver/jpm_tree/lib
copying build/circlet.a to /home/chrisdawson/Projects/janet-playground/littleserver/jpm_tree/lib...
cp -rf build/circlet.a /home/chrisdawson/Projects/janet-playground/littleserver/jpm_tree/lib

But, then jpm build does not work. I assume I need to add the correct --ldflags or --lflags but is there a simpler way to get all this?

$ jpm build -l --verbose --cflags="-I/nix/store/d2d395908dfw2hl1lq7czf1ws8wky5p9-janet-1.21.2/include/janet/"
generating executable c source build/lserve.c from main.janet...
found native /home/chrisdawson/Projects/janet-playground/littleserver/jpm_tree/lib/circlet.so...
compiling build/lserve.c to build/build___lserve.o...
cc -c build/lserve.c -I/nix/store/d2d395908dfw2hl1lq7czf1ws8wky5p9-janet-1.21.2/include/janet/ -Wall -Wextra -I/nix/store/hfimcza6af6qd1944ngf22xqbiayrrw6-jpm-0.0.2/include/janet -I/home/chrisdawson/Projects/janet-playground/littleserver/jpm_tree/lib -O2 -o build/build___lserve.o
linking build/lserve...
cc -I/nix/store/d2d395908dfw2hl1lq7czf1ws8wky5p9-janet-1.21.2/include/janet/ -Wall -Wextra -I/nix/store/hfimcza6af6qd1944ngf22xqbiayrrw6-jpm-0.0.2/include/janet -I/home/chrisdawson/Projects/janet-playground/littleserver/jpm_tree/lib -O2 -o build/lserve build/build___lserve.o /home/chrisdawson/Projects/janet-playground/littleserver/jpm_tree/lib/circlet.a /nix/store/hfimcza6af6qd1944ngf22xqbiayrrw6-jpm-0.0.2/lib/libjanet.a -lm -ldl -lrt -pthread
/nix/store/8qsl7gmnjsg7n460ll32flz5fzlk2kd7-binutils-2.38/bin/ld: cannot find /nix/store/hfimcza6af6qd1944ngf22xqbiayrrw6-jpm-0.0.2/lib/libjanet.a: No such file or directory
collect2: error: ld returned 1 exit status
error: command failed with non-zero exit code 1
  in os/execute [src/core/os.c] on line 1053
  in shell [/nix/store/hfimcza6af6qd1944ngf22xqbiayrrw6-jpm-0.0.2/lib/janet/jpm/shutil.janet] (tailcall) on line 111, column 5
  in <anonymous> [/nix/store/hfimcza6af6qd1944ngf22xqbiayrrw6-jpm-0.0.2/lib/janet/jpm/rules.janet] on line 18, column 20
  in executor [/nix/store/hfimcza6af6qd1944ngf22xqbiayrrw6-jpm-0.0.2/lib/janet/jpm/rules.janet] (tailcall) on line 25, column 8
error: build fail
  in pdag [/nix/store/hfimcza6af6qd1944ngf22xqbiayrrw6-jpm-0.0.2/lib/janet/jpm/dagbuild.janet] (tailcall) on line 80, column 23
  in _thunk [/run/current-system/sw/bin/jpm] (tailcall) on line -1, column -1
  in cli-main [boot.janet] on line 3759, column 17

[sogaiu]

I think @andrewchambers might have some insight into the situation, but I don't know how available he is recently.

[CosmicToast]

This is a common issue (I encounter it all the time on fresh installs).
The problem is that jpm looks at default-config.janet to figure out where to pull janet.h and janet.a from.
As per configs/README.md:

This directory contains some example configs for installing jpm for
various platforms. Package maintainers or people writing scripts to install jpm
may need to adopt (especially on windows!) if the default config does not work.

Specifically, if you look at the linking command, it's trying to link against /nix/store/hfimcza6af6qd1944ngf22xqbiayrrw6-jpm-0.0.2/lib/libjanet.a, which matches the nix-store path of jpm (and not that of janet).
Since nix is treating them as separate packages, the packager should make sure to modify the nix-specific config to set :headerpath, :libpath and potentially :manpath to refer to the janet version associated with the packaged jpm (or latest).

TL;DR in this specific case it's a packager issue, https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/interpreters/janet/jpm.nix already depends on the default.nix file so it should be fairly minor modifications.
I'd submit a patch but I know relatively little about nix(os) besides typical distrodev things, but hopefully I've helped narrow it down.
In the meanwhile, you can edit the default-config.janet yourself to point the above to the janet nix store path.

[sogaiu]

Looks like there are a couple of contributors to the jpm.nix file: bhankas and peterhoeg. May be if an issue is filed at the nixpkgs repository, it might be worth contacting them.

[mattchrist]

I worked around this (and a few other related items) in my project's shell.nix by including this in my shell's buildInputs.

    (pkgs.jpm.overrideAttrs (old: {
      buildInputs = old.buildInputs ++ [ pkgs.makeWrapper ];
      postInstall = "wrapProgram $out/bin/jpm --add-flags '--libpath=${pkgs.janet}/lib --ldflags=-L${pkgs.glibc}/lib --local'";
    }))

[sogaiu]

@xrd Is this still an issue for you? If so, did you try mattchrist's work-around?

[Yzupnick]

@xrd and @mattchrist ,

I just put together an initial version of janet2nix.

Its pretty rough right now, but it provides a flake with a working janet and jpm packages (some recent commit). It also provides helper functions for compiling a janet executable and including jpm packages.

@mattchrist , I based the jpm build on your suggestion here.

There are a lot of missing features, and my knowledge of both nix and janet is pretty shallow, but I would love to collaborate in some way to improve it if either of y'all are up for it.
If so, feel free to message me on the janet zulip or open an issue to start discussing something in the janet2nix repo.