Skip to content

openresty/stream-lua-nginx-module

BranchesTags

Repository files navigation

Name

ngx_stream_lua_module - Embed the power of Lua into Nginx stream/TCP Servers.

This module is a core component of OpenResty. If you are using this module, then you are essentially using OpenResty.

This module is not distributed with the Nginx source. See the installation instructions.

Table of Contents

Status

Production ready.

Version

This document describes ngx_stream_lua v0.0.13, which was released on 21 May, 2023.

Synopsis

events {
    worker_connections 1024;
}

stream {
    # define a TCP server listening on the port 1234:
    server {
        listen 1234;

        content_by_lua_block {
            ngx.say("Hello, Lua!")
        }
    }
}

Set up as an SSL TCP server:

stream {
    server {
        listen 4343 ssl;

        ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers         AES128-SHA:AES256-SHA:RC4-SHA:DES-CBC3-SHA:RC4-MD5;
        ssl_certificate     /path/to/cert.pem;
        ssl_certificate_key /path/to/cert.key;
        ssl_session_cache   shared:SSL:10m;
        ssl_session_timeout 10m;

        content_by_lua_block {
            local sock = assert(ngx.req.socket(true))
            local data = sock:receive()  -- read a line from downstream
            if data == "thunder!" then
                ngx.say("flash!")  -- output data
            else
                ngx.say("boom!")
            end
            ngx.say("the end...")
        }
    }
}

Listening on a UNIX domain socket is also supported:

stream {
    server {
        listen unix:/tmp/nginx.sock;

        content_by_lua_block {
            ngx.say("What's up?")
            ngx.flush(true)  -- flush any pending output and wait
            ngx.sleep(3)  -- sleeping for 3 sec
            ngx.say("Bye bye...")
        }
    }
}

Back to TOC

Description

This is a port of the ngx_http_lua_module to the Nginx "stream" subsystem so as to support generic stream/TCP clients.

The available Lua APIs and Nginx directives remain the same as those of the ngx_http_lua module.

Back to TOC

Directives

The following directives are ported directly from ngx_http_lua. Please check the documentation of ngx_http_lua for more details about their usage and behavior.

The send_timeout directive in the Nginx "http" subsystem is missing in the "stream" subsystem. As such, ngx_stream_lua_module uses the lua_socket_send_timeout directive for this purpose instead.

Note: the lingering close directive that used to exist in older version of stream_lua_nginx_module has been removed and can now be simulated with the newly added tcpsock:shutdown API if necessary.

Back to TOC

preread_by_lua_block

syntax: preread_by_lua_block { lua-script }

context: stream, server

phase: preread

Acts as a preread phase handler and executes Lua code string specified in lua-script for every connection (or packet in datagram mode). The Lua code may make API calls and is executed as a new spawned coroutine in an independent global environment (i.e. a sandbox).

It is possible to acquire the raw request socket using ngx.req.socket and receive data from or send data to the client. However, keep in mind that calling the receive() method of the request socket will consume the data from the buffer and such consumed data will not be seen by handlers further down the chain.

The preread_by_lua_block code will always run at the end of the preread processing phase unless preread_by_lua_no_postpone is turned on.

This directive was first introduced in the v0.0.3 release.

Back to TOC

preread_by_lua_file

syntax: preread_by_lua_file <path-to-lua-script-file>

context: stream, server

phase: preread

Equivalent to preread_by_lua_block, except that the file specified by <path-to-lua-script-file> contains the Lua code or LuaJIT bytecode to be executed.

Nginx variables can be used in the <path-to-lua-script-file> string to provide flexibility. This however carries some risks and is not ordinarily recommended.

When a relative path like foo/bar.lua is given, it will be turned into the absolute path relative to the server prefix path determined by the -p PATH command-line option given when starting the Nginx server.

When the Lua code cache is turned on (by default), the user code is loaded once at the first connection and cached. The Nginx config must be reloaded each time the Lua source file is modified. The Lua code cache can be temporarily disabled during development by switching lua_code_cache off in nginx.conf to avoid having to reload Nginx.

This directive was first introduced in the v0.0.3 release.

Back to TOC

log_by_lua_block

syntax: log_by_lua_block { lua-script }

context: stream, server

phase: log

Runs the Lua source code specified as <lua-script> during the log request processing phase. This does not replace the current access logs, but runs before.

Yielding APIs such as ngx.req.socket, ngx.socket.*, ngx.sleep, or ngx.say are not available in this phase.

This directive was first introduced in the v0.0.3 release.

Back to TOC

log_by_lua_file

syntax: log_by_lua_file <path-to-lua-script-file>

context: stream, server

phase: log

Equivalent to log_by_lua_block, except that the file specified by <path-to-lua-script-file> contains the Lua code or LuaJIT bytecode to be executed.

Nginx variables can be used in the <path-to-lua-script-file> string to provide flexibility. This however carries some risks and is not ordinarily recommended.

When a relative path like foo/bar.lua is given, it will be turned into the absolute path relative to the server prefix path determined by the -p PATH command-line option given when starting the Nginx server.

When the Lua code cache is turned on (by default), the user code is loaded once at the first connection and cached. The Nginx config must be reloaded each time the Lua source file is modified. The Lua code cache can be temporarily disabled during development by switching lua_code_cache off in nginx.conf to avoid having to reload Nginx.

This directive was first introduced in the v0.0.3 release.

Back to TOC

lua_add_variable

syntax: lua_add_variable $var

context: stream

Add the variable $var to the "stream" subsystem and makes it changeable. If $var already exists, this directive will do nothing.

By default, variables added using this directive are considered "not found" and reading them using ngx.var will return nil. However, they could be re-assigned via the ngx.var.VARIABLE API at any time.

This directive was first introduced in the v0.0.4 release.

Back to TOC

preread_by_lua_no_postpone

syntax: preread_by_lua_no_postpone on|off

context: stream

Controls whether or not to disable postponing preread_by_lua* directives to run at the end of the preread processing phase. By default, this directive is turned off and the Lua code is postponed to run at the end of the preread phase.

This directive was first introduced in the v0.0.4 release.

Back to TOC

Nginx API for Lua

Many Lua API functions are ported from ngx_http_lua. Check out the official manual of ngx_http_lua for more details on these Lua API functions.

This module fully supports the new variable subsystem inside the Nginx stream core. You may access any built-in variables provided by the stream core or other stream modules.

Only raw request sockets are supported, for obvious reasons. The raw argument value is ignored and the raw request socket is always returned. Unlike ngx_http_lua, you can still call output API functions like ngx.say, ngx.print, and ngx.flush after acquiring the raw request socket via this function.

When the stream server is in UDP mode, reading from the downstream socket returned by the ngx.req.socket call will only return the content of a single packet. Therefore the reading call will never block and will return nil, "no more data" when all the data from the datagram has been consumed. However, you may choose to send multiple UDP packets back to the client using the downstream socket.

The raw TCP sockets returned by this module will contain the following extra method:

Back to TOC

reqsock:receiveany

syntax: data, err = reqsock:receiveany(max)

context: content_by_lua*, ngx.timer.*, ssl_certificate_by_lua*

This method is similar to tcpsock:receiveany method

This method was introduced into stream-lua-nginx-module since v0.0.8.

Back to TOC

tcpsock:shutdown

syntax: ok, err = tcpsock:shutdown("send")

context: content_by_lua*

Shuts down the write part of the request socket, prevents all further writing to the client and sends TCP FIN, while keeping the reading half open.

Currently only the "send" direction is supported. Using any parameters other than "send" will return an error.

If you called any output functions (like ngx.say) before calling this method, consider use ngx.flush(true) to make sure all busy buffers are complely flushed before shutting down the socket. If any busy buffers were detected, this method will return nil will error message "socket busy writing".

This feature is particularly useful for protocols that generate a response before actually finishing consuming all incoming data. Normally, the kernel will send RST to the client when tcpsock:close is called without emptying the receiving buffer first. Calling this method will allow you to keep reading from the receiving buffer and prevents RST from being sent.

You can also use this method to simulate lingering close similar to that provided by the ngx_http_core_module for protocols in need of such behavior. Here is an example:

local LINGERING_TIME = 30 -- 30 seconds
local LINGERING_TIMEOUT = 5000 -- 5 seconds

local ok, err = sock:shutdown("send")
if not ok then
    ngx.log(ngx.ERR, "failed to shutdown: ", err)
    return
end

local deadline = ngx.time() + LINGERING_TIME

sock:settimeouts(nil, nil, LINGERING_TIMEOUT)

repeat
    local data, _, partial = sock:receive(1024)
until (not data and not partial) or ngx.time() >= deadline

Back to TOC

reqsock:peek

syntax: ok, err = reqsock:peek(size)

context: preread_by_lua*

Peeks into the preread buffer that contains downstream data sent by the client without consuming them. That is, data returned by this API will still be forwarded upstream in later phases.

This function takes a single required argument, size, which is the number of bytes to be peeked. Repeated calls to this function always returns data from the beginning of the preread buffer.

Note that preread phase happens after the TLS handshake. If the stream server was configured with TLS enabled, the returned data will be in clear text.

If preread buffer does not have the requested amount of data, then the current Lua thread will be yielded until more data is available, preread_buffer_size has been exceeded, or preread_timeout has elapsed. Successful calls always return the requested amounts of data, that is, no partial data will be returned.

When preread_buffer_size has been exceeded, the current stream session will be terminated with the session status code 400 immediately by the stream core module, with error message "preread buffer full" that will be printed to the error log.

When preread_timeout has been exceeded, the current stream session will be terminated with the session status code 200 immediately by the stream core module.

In both cases, no further processing on the session is possible (except log_by_lua*). The connection will be closed by the stream core module automatically.

Note that this API cannot be used if consumption of client data has occurred. For example, after calling reqsock:receive. If such an attempt was made, the Lua error "attempt to peek on a consumed socket" will be thrown. Consuming client data after calling this API is allowed and safe.

Here is an example of using this API:

local sock = assert(ngx.req.socket())

local data = assert(sock:peek(1)) -- peek the first 1 byte that contains the length
data = string.byte(data)

data = assert(sock:peek(data + 1)) -- peek the length + the size byte

local payload = data:sub(2) -- trim the length byte to get actual payload

ngx.log(ngx.INFO, "payload is: ", payload)

This API was first introduced in the v0.0.6 release.

Back to TOC

Back to TOC

TODO

  • Add new directives access_by_lua_block and access_by_lua_file.
  • Add lua_postpone_output to emulate the postpone_output directive.

Back to TOC

Nginx Compatibility

The latest version of this module is compatible with the following versions of Nginx:

  • 1.25.x (last tested: 1.25.1)
  • 1.21.x (last tested: 1.21.4)
  • 1.19.x (last tested: 1.19.3)
  • 1.17.x (last tested: 1.17.8)
  • 1.15.x (last tested: 1.15.8)
  • 1.13.x (last tested: 1.13.6)

Nginx cores older than 1.13.6 (exclusive) are not tested and may or may not work. Use at your own risk!

Back to TOC

Installation

It is highly recommended to use OpenResty releases which bundle Nginx, ngx_http_lua, ngx_stream_lua, (this module), LuaJIT, as well as other powerful companion Nginx modules and Lua libraries.

It is discouraged to build this module with Nginx yourself since it is tricky to set up exactly right.

Note that Nginx, LuaJIT, and OpenSSL official releases have various limitations and long standing bugs that can cause some of this module's features to be disabled, not work properly, or run slower. Official OpenResty releases are recommended because they bundle OpenResty's optimized LuaJIT 2.1 fork and Nginx/OpenSSL patches.

Alternatively, ngx_stream_lua can be manually compiled into Nginx:

  1. LuaJIT can be downloaded from the latest release of OpenResty's LuaJIT fork. The official LuaJIT 2.x releases are also supported, although performance will be significantly lower for reasons elaborated above
  2. Download the latest version of ngx_stream_lua HERE
  3. Download the latest supported version of Nginx HERE (See Nginx Compatibility)

Build the source with this module:

wget 'https://nginx.org/download/nginx-1.13.6.tar.gz'
tar -xzvf nginx-1.13.6.tar.gz
cd nginx-1.13.6/

# tell nginx's build system where to find LuaJIT 2.1:
export LUAJIT_LIB=/path/to/luajit/lib
export LUAJIT_INC=/path/to/luajit/include/luajit-2.1

# Here we assume Nginx is to be installed under /opt/nginx/.
./configure --prefix=/opt/nginx \
        --with-ld-opt="-Wl,-rpath,/path/to/luajit-or-lua/lib" \
        --with-stream \
        --with-stream_ssl_module \
        --add-module=/path/to/stream-lua-nginx-module

# Build and install
make -j4
make install

You may use --without-http if you do not wish to use this module with the "http" subsystem. ngx_stream_lua will work perfectly fine without the "http" subsystem.

Back to TOC

Community

Back to TOC

English Mailing List

The openresty-en mailing list is for English speakers.

Back to TOC

Chinese Mailing List

The openresty mailing list is for Chinese speakers.

Back to TOC

Code Repository

The code repository of this project is hosted on GitHub at openresty/stream-lua-nginx-module.

Back to TOC

Bugs and Patches

Please submit bug reports, wishlists, or patches by

  1. creating a ticket on the GitHub Issue Tracker,
  2. or posting to the OpenResty community.

Back to TOC

Acknowledgments

We appreciate Kong Inc. for kindly sponsoring OpenResty Inc. on the following work:

  • Compatibility with Nginx core 1.13.3.
  • Development of meta-lua-nginx-module to make code sharing between this module and lua-nginx-module possible.
  • balancer_by_lua_*, preread_by_lua_*, log_by_lua_* and ssl_certby_lua* phases support.
  • reqsock:peek API support.

Back to TOC

Copyright and License

This module is licensed under the BSD license.

Copyright (C) 2009-2019, by Yichun "agentzh" Zhang (章亦春) [email protected], OpenResty Inc.

Copyright (C) 2009-2016, by Xiaozhe Wang (chaoslawful) [email protected].

All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  • Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

  • Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Back to TOC

See Also

Back to TOC

About

Embed the power of Lua into NGINX TCP/UDP servers

Resources

Readme

License

BSD-2-Clause license
Activity
Custom properties

Stars

727 stars

Watchers

85 watching

Forks

198 forks
Report repository

Releases

39 tags

Packages

No packages published

Contributors 37

+ 23 contributors

Languages