custom Header not sent

[mschfh]

Lychee ignores the --header argument:

Steps to reproduce:

start local http server:

nc -lvp1234

Run lychee (tested with v0.15.1) with --header:

lychee 'http://127.0.0.1:1234' --header 'foo=bar'

Lychee should send foo: bar, but doesn't:

Listening on 0.0.0.0 1234
Connection received on localhost 36584
GET / HTTP/1.1
accept: */*
accept-encoding: gzip
host: 127.0.0.1:1234

[mre]

Maybe that has something to do with our recent move to https://docs.rs/lychee-lib/latest/lychee_lib/chain/type.RequestChain.html. @thomas-zahner fyi

[thomas-zahner]

Could not see the headers in netcat even with lychee versions as old as v0.9.0. Does the nc command really show them?

[mre]

Does the nc command really show them?

It does; at least on macOS.

 nc -lv 1234

curl -v -X GET http://localhost:1234 -H "Custom-Header: CustomValue"

Results in the following nc output:

GET / HTTP/1.1
Host: localhost:1234
User-Agent: curl/8.4.0
Accept: */*
Custom-Header: CustomValue

Might be a problem with lychee.

[mschfh]

It seems to affect the initial request, the header is only set for subsequent ones.

[mre]

Strange. I'm not aware of any special handling for the first request. Will need to investigate.

[thomas-zahner]

So I think that the --header option currently specifies the headers used for checking the actual links. When you do lychee 'http://127.0.0.1:1234' you tell lychee to check all links on the website http://127.0.0.1:1234 but lychee does not use the headers when fetching that website.

If you do echo https://localhost:1234 | lychee --header 'foo=bar' - you actually check if https://localhost:1234 is a valid link and lychee sends the headers.

Maybe we should also include the headers when fetching a websites content (first case) but maybe we should add a separate option for that. I'm not quite sure.. What are your thoughts @mre?

[mschfh]

As a user, I'd expect it to always send the specified headers on every request, including the initial one. (especially since they are used for authentication in my case)

It'd be nice to have some customizable rules in the config file to include/exclude certain headers for certain hosts, see #1298, but that is a separate issue.

[mre]

Yes, I agree with @mschfh on both points.