Docker SSL configuration doesn't enable ssl port

[semihsezer]

Making /ssl volume available for the neo4j container doesn't seem to enable ssl. I followed the instructions on Neo4j Docker docs to enable ssl and also looked into docker-entrypoint.sh. I don't think that what is mentioned in the docs is working.

https://neo4j.com/docs/operations-manual/current/docker/security/
Steps to reproduce:

$ pwd
/opt/neo4j/certs/https
$ ls
neo4j.cert  neo4j.key  private.key  public.crt
$ docker run -d -p 7473:7473 -p 7474:7474 -p 7687:7687 --rm \
    -v /opt/neo4j/certs/https/:/ssl \
    neo4j:4.0

Expected behaviour
Https is enabled for browser with https scheme and https port (7473).

Actual behaviour
Https is not enabled. Remote interface is still only available at http scheme and http port (7474).

$ docker logs <container-id>
Starting Neo4j.
2020-01-25 22:20:58.078+0000 WARN  Use of deprecated setting dbms.directories.certificates. Legacy ssl policy is no longer supported.
2020-01-25 22:20:58.090+0000 INFO  ======== Neo4j 4.0.0 ========
2020-01-25 22:20:58.094+0000 INFO  Starting...
2020-01-25 22:21:02.275+0000 INFO  Bolt enabled on 0.0.0.0:7687.
2020-01-25 22:21:02.276+0000 INFO  Started.
2020-01-25 22:21:03.468+0000 INFO  Remote interface available at http://0.0.0.0:7474/

Try curl -k https://localhost:7473 returns ssl error however curl localhost:7474 does return a response from the server.

There is also the deprecation warning about dbms.directories.certificates in the logs, which makes me think that /ssl is not working as expected in the entrypoint.

Any help would be greatly appreciated!

Neo4j Docker tag: 4.0
Docker version: 18.03.1-ce
OS: Ubuntu 16.04.5 LTS

[semihsezer]

I am able to enable ssl by setting the settings below. However simply mounting /ssl doesn't work.

$ docker run -d -p 7473:7473 -p 7474:7474 -p 7687:7687 --rm \
    -v /opt/neo4j/certs/https:/var/lib/neo4j/certificates/https \
    -e NEO4J_dbms_ssl_policy_https_enabled=true \
    -e NEO4J_dbms_ssl_policy_https_base__directory=certificates/https \
    -e NEO4J_dbms_ssl_policy_https_private__key=private.key \
    -e NEO4J_dbms_ssl_policy_https_public__certificate=public.crt \
    -e NEO4J_dbms_connector_https_enabled=true \
    -e NEO4J_dbms_connector_https_listen__address=0.0.0.0:7473 \
    neo4j:4.0

$ curl -k https://localhost:7473
{
  "bolt_routing" : "neo4j://0.0.0.0:7687",
  "transaction" : "https://localhost:7473/db/{databaseName}/tx",
  "bolt_direct" : "bolt://0.0.0.0:7687",
  "neo4j_version" : "4.0.0",
  "neo4j_edition" : "community"
}```

[Schirmi136]

I'm having a similar issue. I used to mount /ssl, which is not possible anymore with Neo4j 4.0. After changing the docker configuration to

docker run -d \
--name neo4j \
--net=host \
--volume=/opt/neo4j/certificates/https:/var/lib/neo4j/certificates/https  \
--env NEO4J_dbms_ssl_policy_https_enabled=true \
--env NEO4J_dbms_ssl_policy_https_base__directory=certificates/https \
--env NEO4J_dbms_ssl_policy_https_private__key=private.key \
--env NEO4J_dbms_ssl_policy_https_public__certificate=public.crt \
neo4j:4.0.3

I'm getting the error

 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabaseService@4a335fa8' was successfully initialized, but failed to start. Please see the attached cause exception "/var/lib/neo4j/certificates/https/private.key (Permission denied)". Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabaseService@4a335fa8' was successfully initialized, but failed to start. Please see the attached cause exception "/var/lib/neo4j/certificates/https/private.key (Permission denied)".
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabaseService@4a335fa8' was successfully initialized, but failed to start. Please see the attached cause exception "/var/lib/neo4j/certificates/https/private.key (Permission denied)".
        at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:45)
        at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:164)
        at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:114)
        at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:89)
        at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:35)

I tried changing the permissions via
sudo chmod 777 /opt/neo4j/certificates/https/*
but permissions get changed from -rwxrwxrwx to -rwx------ after running the docker run command and I get the same error.

Running docker with --user="$(id -u):$(id -g)" or --user="0:0" didn't help either. Any ideas?

[jennyowen]

You're right, our SSL documentation for docker is horribly out of date. I apologise for that and I'm going to work on fixing it.

Here's what I did to get it working, and I suppose the new documentation will be some variation on this.

HTTPS Encryption in Neo4j Docker

These instructions assume you already have the necessary ssl key and certificate.

1. Create a certificates folder, with subfolders for each encryption policy you need.

This is documented at: https://neo4j.com/docs/operations-manual/4.0/security/ssl-framework/

The key and certificate must be called private.key and public.crt respectively. For example, this is the folder structure required for https encryption:

certificates/
  |  https/
       |  private.key
       |  public.crt
       |  revoked/
            |  
       |  trusted/
            |  public.crt  (a copy of certificates/https/public.crt)

2. Start docker remembering to:

• mount certificates to /var/lib/neo4j/certificates
• run container as a user with at least READ access to the certificates folder and all its files.
• publish port 7473
• set the configurations enabling https, NEO4J_dbms_connector_https_enabled and NEO4J_dbms_ssl_policy_https_enabled, to true.

Here's an example:

docker run -it --rm \
    -p 7473:7473 -p 7474:7474 -p 7687:7687 \
    --user="$(id -u):$(id -g)" \
    -v $HOME/neo4j/certificates:/var/lib/neo4j/certificates  \
    -e NEO4J_dbms_connector_https_enabled=true \
    -e NEO4J_dbms_ssl_policy_https_enabled=true \
    neo4j:4.0.3

This should be enough to enable SSL encryption over HTTPS:

curl -k https://localhost:7473           
{
  "bolt_routing" : "neo4j://localhost:7687",
  "transaction" : "https://localhost:7473/db/{databaseName}/tx",
  "bolt_direct" : "bolt://localhost:7687",
  "neo4j_version" : "4.0.3",
  "neo4j_edition" : "community"
}

[jennyowen]

@Schirmi136 Could you check who the owning user and group is on /opt/neo4j/certificates/https? I suspect it's not the same as the user you're running as. So either neo4j won't be able to read the files there (when running with --user="$(id -u):$(id -g)" OR it can't chown the files as the neo4j user because docker daemon hasn't got the necessary permissions.

[davcamer]

I have roughly the same problem. I have a directory set up with different certificates for the bolt and https connectors:

/home/neo4j/certificates# find .
.
./bolt
./bolt/private.key
./bolt/public.crt
./https
./https/private.key
./https/public.crt

If I mount this certificates folder at the /ssl path in the container, I get an error that the configuration doesn't exist:

Directories in use:
  home:         /var/lib/neo4j
  config:       /var/lib/neo4j/conf
  logs:         /logs
  plugins:      /var/lib/neo4j/plugins
  import:       /var/lib/neo4j/import
  data:         /var/lib/neo4j/data
  certificates: /ssl
  run:          /var/lib/neo4j/run
Starting Neo4j.
2020-06-11 00:42:09.452+0000 WARN  Use of deprecated setting dbms.directories.certificates. Legacy ssl policy is no longer supported.
2020-06-11 00:42:09.462+0000 INFO  ======== Neo4j 4.0.4 ========
2020-06-11 00:42:09.466+0000 INFO  Starting...
2020-06-11 00:42:10.157+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabaseService@17aad511' was successfully initialized, but failed to start. Please see the attached cause exception "Base directory '/var/lib/neo4j/certificates/bolt' for SSL policy with name 'bolt' does not exist.". Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabaseService@17aad511' was successfully initialized, but failed to start. Please see the attached cause exception "Base directory '/var/lib/neo4j/certificates/bolt' for SSL policy with name 'bolt' does not exist.".
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabaseService@17aad511' was successfully initialized, but failed to start. Please see the attached cause exception "Base directory '/var/lib/neo4j/certificates/bolt' for SSL policy with name 'bolt' does not exist.".
	at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:45)
	at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:164)
	at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:114)
	at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:89)
	at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:35)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabaseService@17aad511' was successfully initialized, but failed to start. Please see the attached cause exception "Base directory '/var/lib/neo4j/certificates/bolt' for SSL policy with name 'bolt' does not exist.".
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:465)
	at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
	at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:157)
	... 3 more
Caused by: java.lang.IllegalArgumentException: Base directory '/var/lib/neo4j/certificates/bolt' for SSL policy with name 'bolt' does not exist.
	at org.neo4j.ssl.config.SslPolicyLoader.createSslPolicy(SslPolicyLoader.java:159)
	at org.neo4j.ssl.config.SslPolicyLoader.addPolicy(SslPolicyLoader.java:143)
	at java.base/java.util.HashMap$Values.forEach(HashMap.java:976)
	at org.neo4j.ssl.config.SslPolicyLoader.load(SslPolicyLoader.java:133)
	at org.neo4j.ssl.config.SslPolicyLoader.create(SslPolicyLoader.java:96)
	at org.neo4j.graphdb.factory.module.edition.CommunityEditionModule.<init>(CommunityEditionModule.java:110)
	at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:118)
	at org.neo4j.server.database.CommunityGraphFactory.newDatabaseManagementService(CommunityGraphFactory.java:36)
	at org.neo4j.server.database.LifecycleManagingDatabaseService.start(LifecycleManagingDatabaseService.java:88)
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
	... 5 more
2020-06-11 00:42:10.162+0000 INFO  Neo4j Server shutdown initiated by request

However, if I mount the same directory at /var/lib/neo4j/certificates, without changing anything else in the configuration, the system works.

Changed password for user 'neo4j'.
Directories in use:
  home:         /var/lib/neo4j
  config:       /var/lib/neo4j/conf
  logs:         /logs
  plugins:      /var/lib/neo4j/plugins
  import:       /var/lib/neo4j/import
  data:         /var/lib/neo4j/data
  certificates: /var/lib/neo4j/certificates
  run:          /var/lib/neo4j/run
Starting Neo4j.
2020-06-11 00:49:14.558+0000 INFO  ======== Neo4j 4.0.5 ========
2020-06-11 00:49:14.570+0000 INFO  Starting...
2020-06-11 00:49:20.469+0000 INFO  Bolt enabled on 0.0.0.0:7687.
2020-06-11 00:49:20.470+0000 INFO  Started.
2020-06-11 00:49:22.632+0000 INFO  Remote interface available at https://localhost:7473/

The permissions are the same in both situations. In both situations, I am passing a uid and gid from the host system using the --user flag to docker. That is the user that owns the certificates and directories. They have 500 permissions for directories and 400 for files on the host:

/home/neo4j/certificates# ll
total 16
dr-x------ 4 neo4j neo4j 4096 Jun 11 00:49 ./
drwxr-xr-x 8 neo4j neo4j 4096 Jun 11 00:49 ../
dr-x------ 2 neo4j neo4j 4096 Jun 11 00:49 bolt/
dr-x------ 2 neo4j neo4j 4096 Jun 11 00:49 https/
/home/neo4j/certificates# cd bolt
/home/neo4j/certificates/bolt# ll
total 16
dr-x------ 2 neo4j neo4j 4096 Jun 11 00:49 ./
dr-x------ 4 neo4j neo4j 4096 Jun 11 00:49 ../
-r-------- 1 neo4j neo4j 1707 Jun 11 00:49 private.key
-r-------- 1 neo4j neo4j 1249 Jun 11 00:49 public.crt

I'm working around this with the other mount, but I found it confusing that I need to mount the certificates under /var/lib/neo4j/, but configuration under /conf. I ran in circles trying to mount conf under /var/lib/neo4j/conf, but doing that means changes from the entrypoint script are persisted to the host.

One possible solution might be to copy the certificates instead of symlinking the directory. In other words, handle certificates the same way configuration is handled. (I am suspicious that the certificate loaders are refusing to traverse the symlink.) This would be a trade off though, as certificates couldn't be refreshed in place, without some extra logic somewhere to copy again.

I have seen this problem with both the 4.0.4 and (current) latest tagged images.

[jennyowen]

I've finally updated the documentation and included examples of how to set up https encryption. See:
https://neo4j.com/docs/operations-manual/4.0/docker/security/
Mounting the certificates folder to /ssl doesn't work any more for 4.0.0 and onwards unfortunately. The legacy ssl system that used was deprecated.

I'm going to close this issue now.
There are still problems using self signed certificates, but that is tracked separately in issue #257 .

[jennyowen]

@davcamer the default directory Neo4j 4.0 looks for certificates is /var/lib/neo4j/certificates (see: https://neo4j.com/docs/operations-manual/4.0/security/ssl-framework/).
If you mount your certificates folder to /ssl then the certificates will be stored under /ssl instead, so if you don't also configure Neo4j to look there, then it won't be able to find the certificates. This is what your error message is complaining about. When you mount your certificates to /var/lib/neo4j/certificates, Neo4j is able to find them and it all works.
Check https://neo4j.com/docs/operations-manual/4.0/docker/security/ for more help, and maybe let me know if it didn't work for you!

[davcamer]

@jennyowen In that case, should this section of the entrypoint script be removed?
https://github.com/neo4j/docker-neo4j/blob/master/docker-image-src/4.0/docker-entrypoint.sh#L315-L319

[jennyowen]

@davcamer ok you got me! In my defence, that bit of code is only in 4.0.5 and onwards.

[davcamer]

Oops! My fault for looking at the wrong code then. 😞 And my /ssl mount would probably work on the 4.0.5 image. Thanks. 🙇‍♂️