Merge branch 'acceptor-socket'

doc/Installation.md

@@ -23,10 +23,11 @@ sudo apt install python3 python3-requests python3-configargparse python3-bcrypt
 ### Developers
 …might want to additionally install
 ```
-sudo apt install python3-nosetests python3-rednose python3-nose-cov
+sudo apt install python3-nosetests python3-rednose python3-nose-cov socket
 ```
+They are required to run the tests.
 
-## Configuration
+## XMPP Server Configuration
 
 :warning: The API secret must not fall into the wrong hands!
 Anyone knowing it can authenticate as any user to the XMPP server
@@ -82,47 +83,94 @@ This fixes a bug with treating an echo of the request as the answer
 Even then, several users report problems with `lua-lpty`, [such as
 processes not dying and still occupying sockets](https://github.com/jsxc/xmpp-cloud-auth/issues/63). Please look at the [socket mode](#socket-interface) for an alternative.
 
+## Configuration as Mail Server Backend
+
+`xcauth` can also be used to provide
+- user authentication to mail servers using the *saslauthd* protocol and
+- verification of user existence using the *postfix* protocol.
+
+Administrators of small SOHO systems can thus use Nextcloud as their prime
+authentication source for
+- file storage/sharing (Nextcloud),
+- instant messaging (XMPP), and
+- email (tested with Cyrus and Postfix).
+
+### *saslauthd* authentication
+
+In an attempt to move toward Nextcloud as the main authentication source,
+`-t saslauthd` mode is supported, which allows to run services
+which can authenticate against Cyrus *saslauthd* to authenticate against
+JSXC and Nextcloud. It has been successfully tested against *Postfix*
+and *Cyrus IMAP*. More information can be found in
+[systemd/README.md (*saslauthd* mode)](../systemd/README.md#saslauthd-mode-authentication).
+The protocol is described in [doc/Protocol.md](./Protocol.md#saslauthd).
+
+### *postfix* existence tests
+
+When using virtual mailboxes (i.e., mailboxes in multiple domains, nut just
+using virtual addresses), *Postfix* needs a way to check for the existence
+of that mailbox. A *Postfix* `tcp_table` compatible interface has been
+implemented using the `-t postfix` mode, so an `xcauth` instance
+started e.g. by *systemd* can be used to provide the mailbox existence
+information, as explained in
+[systemd/README.md (*postfix* mode)](../systemd/README.md#postfix-mode-existence-check).
+Please note, that aliases or virtual users still need to be configured
+using the standard *postfix* mechanisms.
+
+The protocol is described in [doc/Protocol.md](./Protocol.md#postfix).
+
 ## Options
 ```
 $ ./xcauth.py --help
-usage: xcauth.py [-h] [-c CONFIG_FILE] -u URL -s SECRET [-l LOG]
-                 [-p PER_DOMAIN_CONFIG] [-b DOMAIN_DB] [-d] [-i]
-                 [-t {generic,prosody,ejabberd,saslauthd}] [--timeout TIMEOUT]
-                 [--cache-db CACHE_DB] [--cache-query-ttl CACHE_QUERY_TTL]
+usage: xcauth.py [-h] [--config-file CONFIG_FILE] [--domain-db DOMAIN_DB]
+                 [--auth-test USER DOMAIN PASSWORD]
+                 [--isuser-test USER DOMAIN] [--roster-test USER DOMAIN]
+                 [--update-roster] --url URL --secret SECRET [--log LOG]
+                 [--debug] [--interactive]
+                 [--type {generic,prosody,ejabberd,saslauthd,postfix}]
+                 [--timeout TIMEOUT] [--cache-db CACHE_DB]
+                 [--cache-query-ttl CACHE_QUERY_TTL]
                  [--cache-verification-ttl CACHE_VERIFICATION_TTL]
                  [--cache-unreachable-ttl CACHE_UNREACHABLE_TTL]
                  [--cache-bcrypt-rounds CACHE_BCRYPT_ROUNDS]
-                 [-A USER DOMAIN PASSWORD] [-I USER DOMAIN] [--version]
+                 [--ejabberdctl PATH] [--shared-roster-db SHARED_ROSTER_DB]
+                 [--version]
 
 XMPP server authentication against JSXC>=3.2.0 on Nextcloud. See
 https://jsxc.org or https://github.com/jsxc/xmpp-cloud-auth. Args that start
-with '--' (eg. -u) can also be set in a config file (/etc/xcauth.conf or
-/etc/external_cloud.conf or ./xcauth.conf or specified via -c). Config file
-syntax allows: key=value, flag=true, stuff=[a,b,c] (for details, see syntax at
-https://goo.gl/R74nmi). If an arg is specified in more than one place, then
-commandline values override config file values which override defaults.
+with '--' (eg. --domain-db) can also be set in a config file (/etc/xcauth.conf
+or specified via --config-file). Config file syntax allows: key=value,
+flag=true, stuff=[a,b,c] (for details, see syntax at https://goo.gl/R74nmi).
+If an arg is specified in more than one place, then commandline values
+override config file values which override defaults.
 
 optional arguments:
   -h, --help            show this help message and exit
-  -c CONFIG_FILE, --config-file CONFIG_FILE
+  --config-file CONFIG_FILE, -c CONFIG_FILE
                         config file path
-  -u URL, --url URL     base URL
-  -s SECRET, --secret SECRET
+  --domain-db DOMAIN_DB, -b DOMAIN_DB
+                        persistent domain database; manipulated with xcdbm.py
+  --auth-test USER DOMAIN PASSWORD, -A USER DOMAIN PASSWORD
+                        single, one-shot query of the user, domain, and
+                        password triple
+  --isuser-test USER DOMAIN, -I USER DOMAIN
+                        single, one-shot query of the user and domain tuple
+  --roster-test USER DOMAIN, -R USER DOMAIN
+                        single, one-shot query of the user's shared roster
+  --update-roster, -T   also try to update ejabberd shared roster; requires
+                        --ejabberdctl and --shared-roster-db
+  --url URL, -u URL     base URL
+  --secret SECRET, -s SECRET
                         secure api token
-  -l LOG, --log LOG     log directory (default: /var/log/xcauth)
-  -p PER_DOMAIN_CONFIG, --per-domain-config PER_DOMAIN_CONFIG
-                        name of file containing whitespace-separated (domain,
-                        secret, url) tuples
-  -b DOMAIN_DB, --domain-db DOMAIN_DB
-                        persistent domain database; manipulated with -G, -P,
-                        -D, -L, -U
-  -d, --debug           enable debug mode
-  -i, --interactive     log to stdout
-  -t {generic,prosody,ejabberd,saslauthd}, --type {generic,prosody,ejabberd,saslauthd}
-                        XMPP server type (prosody=generic); implies reading
-                        requests from stdin
-  --timeout TIMEOUT     Timeout for each of connection setup and request
-                        processing
+  --log LOG, -l LOG     log directory (default: /var/log/xcauth)
+  --debug, -d           enable debug mode
+  --interactive, -i     log to stderr
+  --type {generic,prosody,ejabberd,saslauthd,postfix}, -t {generic,prosody,ejabberd,saslauthd,postfix}
+                        XMPP server/query protocol type (prosody≘generic);
+                        implies reading requests from stdin. See
+                        doc/Installation.md and systemd/README.md for more
+                        information and overrides.
+  --timeout TIMEOUT     Timeout for connection setup, request processing
   --cache-db CACHE_DB   Database path for the user cache; enables cache if set
   --cache-query-ttl CACHE_QUERY_TTL
                         Maximum time between queries
@@ -133,17 +181,17 @@ optional arguments:
                         (overrides the other TTLs)
   --cache-bcrypt-rounds CACHE_BCRYPT_ROUNDS
                         Encrypt passwords with 2^ROUNDS before storing (i.e.,
-                        every increasing ROUNDS takes twice as much
+                        every increment of ROUNDS results in twice the
                         computation time)
-  -A USER DOMAIN PASSWORD, --auth-test USER DOMAIN PASSWORD
-                        single, one-shot query of the user, domain, and
-                        password triple
-  -I USER DOMAIN, --isuser-test USER DOMAIN
-                        single, one-shot query of the user and domain tuple
+  --ejabberdctl PATH    Enables shared roster updates on authentication; use
+                        ejabberdctl command at PATH to modify them
+  --shared-roster-db SHARED_ROSTER_DB
+                        Which groups a user has been added to (to ensure
+                        proper deletion)
   --version             show program's version number and exit
 
--A takes precedence over -I over -t. -A and -I imply -d. -A, -I, -G, -P, -D,
--L, and -U imply -i. The database operations require -b.
+-I, -R, and -A take precedence over -t. One of them is required. -I, -R, and
+-A imply -i and -d.
 ```
 
 Note that `-t generic` is identical to `-t prosody`. This is just to indicate
@@ -183,15 +231,6 @@ the `external_auth_command = "@localhost:23664";` option to talk over a socket t
 by *Prosody* on port 23664. [systemd/README.md](../systemd/README.md) explains how to automatically start
 such a process using *systemd*.
 
-### Experimental *saslauthd* compatibility
-
-In an attempt to move toward Nextcloud as the main authentication source,
-a new `-t saslauthd` mode is supported, which allows to run services
-which can authenticate against Cyrus *saslauthd* to authenticate against
-JSXC and Nextcloud. It has been successfully tested against *Postfix*
-and *Cyrus IMAP*. More information can be found in
-[systemd/README.md (*saslauthd* mode)](../systemd/README.md#saslauthd-mode)
-
 ### *ejabberd* shared roster support
 
 In the configuration file, set

systemd/README.md

@@ -1,20 +1,44 @@
-# *systemd* socket configuration support
+# *systemd* integration
 
-For some environments, it might be advantageous to use *xcauth* over a network socket. Here is a pair of sample *systemd* configuration files, accepting network connection to `localhost:23664`.
+`xcauth` can also be started from *systemd*. Three modes are supported:
 
-## Installation (as root)
+1. Starting in *inetd* compatibility mode: For each connection to that socket, a new `xcauth` process is started. `xcauth` reads from stdin/stdout (DEPRECATED).
+1. Using *systemd* [socket activation](http://0pointer.net/blog/projects/socket-activation.html), single protocol per configuration file: On the first connection, the single `xcauth` process is started for this protocol/port. For each incoming connection, only a thread is spawned. This is more efficient if a new connection is opened for every request (common for *saslauthd* and *postfix* modes, but depends on the requesting application).
+1. Using *systemd* socket activation, multiple protocols per configuration file: Similar to the one above, but only a single `xcauth` process is ever started. All protocols are determined by information passed by *systemd* on process start (RECOMMENDED).
+
+The following ports are used by default:
+- TCP port 23662: *ejabberd* protocol support
+- TCP port 23663: *prosody* protocol support
+- TCP port 23664: *prosody* or *ejabberd* protocol support, depending on configuration in `/etc/xcauth.conf` (DEPRECATED)
+- TCP port 23665: *postfix* protocol support
+- `/var/run/saslauthd/mux` (stream-based Unix domain socket): *saslauthd* protocol support
+
+## XMPP authentication over *systemd* socket
+
+For some environments, it might be advantageous to use *xcauth* over a network socket. Here is a set of sample *systemd* configuration files, accepting the network connections described above.
+
+### Installation (as root)
 
 1. Perform the *xcauth* installation as explained in the [parent README](../README.md) or the [installation wiki](https://github.com/jsxc/xcauth/wiki). Especially install source into `/opt/xcauth` and put the configuration in `/etc/xcauth.conf`.
-1. Copy `[email protected]` and `xcauth.socket` to `/etc/systemd/system` (if no modifications to these files are needed, you may also symlink them manually or using `systemctl link`; beware that some versions of *systemd* have problems with symlinks ([systemd#3010](https://github.com/systemd/systemd/issues/3010))
 1. Create the user `xcauth` and the directories: `sudo ../install.sh`
-1. Activate the service: `systemctl enable xcauth.socket` and `systemctl start xcauth.socket`
+1. Copy `xc*` to `/etc/systemd/system` (if no modifications to these files are needed, you may also symlink them manually or using `systemctl link`; beware that some versions of *systemd* have problems with symlinks ([systemd#3010](https://github.com/systemd/systemd/issues/3010))
+1. Activate the service:
+```sh
+systemctl enable xcauth.service
+for i in xc*.socket; do
+  systemctl start $i
+done
+systemctl start xcauth.service
+```
+
+:warning: If you do not want to replace an existing *saslauthd* on your system, do not copy or start `xcsaslauth.socket`.
 
-## Testing
+### Testing
 
-If you have set `type=generic` (equivalent to `type=prosody`) in `/etc/xcauth.conf`, then the following should work (`$` indicates the command line prompt, `<` is data received and `>` data sent):
+Trye the following (`$` indicates the command line prompt, `<` is data received and `>` data sent):
 
 ```
-$ telnet localhost 23664
+$ telnet localhost 23663
 < Trying ::1...
 < Connected to localhost.
 < Escape character is '^]'.
@@ -38,7 +62,7 @@ respective documentation for how to do this). Then, run the following
 commands to have *xcauth.py* pose as *saslauthd*:
 
 1. Install *xcauth* as described above.
-1. Copy `xcsaslauth@.service` and `xcsaslauth.socket` to `/etc/systemd/system` (see above for symlink issues)
+1. Copy `xcsaslauth.service` and `xcsaslauth.socket` to `/etc/systemd/system` (see above for symlink issues)
 1. Disable "normal" *saslauthd*: `systemctl disable saslauthd`
 1. Enable *xcauth.py* in *saslauthd* mode: `systemctl enable xcsaslauth.socket` and `systemctl start xcsaslauth.socket`
 
@@ -47,7 +71,7 @@ Note that the *xcsaslauth* service listens on the Unix domain socket
 the software configuration files might only mention `/var/run/saslauthd`,
 the `/mux` suffix is added internally by the *SASL* library.
 
-## `postfix` mode (existance check)
+## `postfix` mode (existence check)
 
 When a *Postfix* mail server serves multiple realms (=domains), it
 needs some way to know whether a
@@ -80,7 +104,7 @@ user's mailbox.
    to `/etc/postfix/main.cf`. Integrate any existing assignment to
    `virtual_mailbox_maps`.
 1. Install *xcauth* as described above.
-1. Copy `xcpostfix@.service` and `xcpostfix.socket` to `/etc/systemd/system` (see above for symlink issues)
+1. Copy `xcpostfix.service` and `xcpostfix.socket` to `/etc/systemd/system` (see above for symlink issues)
 1. Enable *xcauth.py* in *postfix* mode: `systemctl enable xcpostfix.socket` and `systemctl start xcpostfix.socket`
 
 ## Security considerations

systemd/xcauth.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=XMPP to Nextcloud+JSXC authentication bridge
+
+[Service]
+ExecStart=/opt/xmpp-cloud-auth/xcauth.py
+User=xcauth
+Sockets=xcauth.socket xcejabberd.socket xcpostfix.socket xcprosody.socket xcsaslauth.socket
+# Should be doable in `xcsaslauth.socket` according to
+# https://www.freedesktop.org/software/systemd/man/systemd.socket.html#ExecStartPre=
+# but doesn't for me in Ubuntu 18.04. So I moved it here.
+#
+# The downside of this setup: salsauth connection may not occur before
+# manual `systemctl start xcauth.service` or connection to one of the
+# other sockets.
+ExecStartPre=+/bin/chgrp sasl /var/run/saslauthd/
+
+[Install]
+WantedBy=multi-user.target

systemd/xcauth.socket

@@ -1,11 +1,10 @@
 [Unit]
 Description=XMPP to Nextcloud+JSXC authentication bridge
-PartOf=xcauth.service
 
 [Socket]
 ListenStream=[::1]:23664
 ListenStream=127.0.0.1:23664
-Accept=yes
-
-[Install]
-WantedBy=multi-user.target
+Accept=false
+# Is not one of the recognized protocols, so this name will be ignored
+# (i.e., the protocol defaults to what is passed by "-t")
+FileDescriptorName=deprecated

systemd/xcejabberd.socket

@@ -0,0 +1,9 @@
+[Unit]
+Description=XMPP to Nextcloud+JSXC authentication bridge for ejabberd
+
+[Socket]
+ListenStream=[::1]:23662
+ListenStream=127.0.0.1:23662
+Accept=false
+FileDescriptorName=ejabberd
+Service=xcauth.service

systemd/xcpostfix.socket

@@ -1,12 +1,9 @@
 [Unit]
 Description=Postfix virtual mailbox to Nextcloud+JSXC bridge
-PartOf=xcpostfix.service
 
 [Socket]
 ListenStream=[::1]:23665
 ListenStream=127.0.0.1:23665
-Accept=yes
-Accept=yes
-
-[Install]
-WantedBy=multi-user.target
+Accept=false
+FileDescriptorName=postfix
+Service=xcauth.service

systemd/xcprosody.socket

@@ -0,0 +1,9 @@
+[Unit]
+Description=XMPP to Nextcloud+JSXC authentication bridge for Prosody
+
+[Socket]
+ListenStream=[::1]:23663
+ListenStream=127.0.0.1:23663
+Accept=false
+FileDescriptorName=prosody
+Service=xcauth.service

systemd/xcsaslauth.socket

@@ -1,14 +1,15 @@
 [Unit]
 Description=SASL to Nextcloud+JSXC authentication bridge
-PartOf=xcsaslauth.service
 
 [Socket]
 ListenStream=/var/run/saslauthd/mux
 SocketUser=root
 SocketGroup=sasl
 SocketMode=660
 DirectoryMode=710
-Accept=yes
-
-[Install]
-WantedBy=multi-user.target
+# Does not work for me yet, despite the documentation in
+# https://www.freedesktop.org/software/systemd/man/systemd.socket.html#ExecStartPre=
+#ExecStartPost=/bin/chgrp sasl /var/run/saslauthd/
+Accept=false
+FileDescriptorName=saslauthd
+Service=xcauth.service

tests/Makefile

@@ -1,7 +1,18 @@
 all:	nosetests perltests
+more:	really-all
+really-all:	nosetests perltests-all
 
 nosetests:
 	(cd .. && nosetests3)
 
-perltests:
-	for i in run-online*.pl; do (cd .. && tests/$$i) || exit 1; done
+perltests: perltests-direct perltests-subprocess perltests-socket1366x
+perltests-all: perltests perltests-socket2366x
+
+perltests-direct:
+	cd .. && tests/run-online.pl
+perltests-subprocess:
+	for i in run-online-*.pl; do echo == $$i; (cd .. && tests/$$i subprocess) || exit 1; done
+perltests-socket1366x:
+	for i in run-online-*.pl; do echo == $$i; (cd .. && tests/$$i socket1366x) || exit 1; sleep 1; done
+perltests-socket2366x:
+	for i in run-online-*.pl; do echo == $$i; (cd .. && tests/$$i socket2366x) || exit 1; done