About Win32 OpenSSH and Design Details
OpenSSH, part of OpenBSD operating system, is a bunch of utility programs based on SSH protocol. These include server and client executables as well as utilities to create and manage cryptographic keys. Portable OpenSSH is derived from the OpenBSD project and ported to support a wide variety of Unix flavors. The goal of this project is to extend support to Windows family of operating systems and be able to do this in a common repository without needing a Windows specific fork.
Relevant design details in the context of this project -
OpenSSH and the portable version are single threaded applications, interacting with IO using POSIX based File Descriptors and multiplexing IO operations using [select] (https://en.wikipedia.org/wiki/Select_%28Unix%29) calls. Session isolation and privilege separation are implemented using the standard UNIX routines - fork, setuid, chroot. IPC is carried over UNIX domain sockets.
As stated earlier, the main goal is side by side Windows support in the portable version of OpenSSH. The project is currently being worked on a fork of OpenSSH7.1p1 - here after, called win32-fork code and main code respectively. The plan is get this fork to a state that could integrate into the latest version in main, with minimum impact to main sources. Obviously, we would want to reuse the main code as much as possible, whilst respecting the fundamental differences between Unix and Windows operating systems.
To prevent any regressions in main and to enable easier review of the changes coming from win32-fork, there will be no "main" code moving or refactoring in win32-fork. There are multiple places where platform abstraction makes sense (auth, console to name a few), but this wont be addressed in the fork as it would lead to significant code churn. This will be done post integration once we have stable Windows supported version with significant test coverage living in main repo. Crypto support using Windows CNG has been tested out in win32-fork but since it needed reasonable modifications to original code, relevant changes will be reverted\undone. This means that the Windows supported version potentially available mid this year will rely on OpenSSL's crypto (exception is SSP for key-based authentication that will use CNG - more details later). Coding style will follow OpenBSD guidelines.
Here, we'll discuss the design choices aimed at keeping majority of code base common between Unix and Windows, deviating where deemed necessary.
POSIX IO calls are a significant part of OpenSSH code. A POSIX IO wrapper will be implemented on top of Win32 async File IO. This wrapper strictly implements the POSIX IO needs of OpenSSH keeping the code differences, especially in the transport layer, to a minimum. Note that the wrapper implements only the calls needed by OpenSSH (and not all defined in POSIX standard). Specifically, the wrapper implements
- IO calls creating file descriptors - open, creat, socket, accept, socketpair, pipe
- operations on a single file descriptor - fd_set, FD_* macros, fcntl, read, write, recv, send, fstat, fdopen, close, dup and dup2
- operations on multiple file descriptors - select
- signal semantics on these operations - ex. select (or any blocking IO call) returning EINTR
- Apart from these, the wrapper also bridges the gap in using POSIX signals. Details below.
Design summary of POSIX wrapper
- Single threaded (not thread safe based on current needs but can be made so if needed going forward).
- Maintains file descriptor table and its state. Table stores mapping between file descriptors (int) and associated Windows IO state (handle, buffers, async call state, etc).
- fd_set implemented as a bit array, lowest available file descriptors get allocated first.
- Calls underlying Win32 APIs that provide Overlapped semantics.
- "select" and blocking IO calls go through a common "wait_for_any" function that wakes up on
- IO completions
- Signals
- Timers
- All underlying Win32 IO API calls are made asynchronous (non-blocking). Blocking semantics are implemented within the wrapper by an explicit "wait_for_any" for IO to complete.
- FD_CLOEXEC is supported, setting this flag denies inheritance of underlying Windows handle.
- Uses APCs wherever available and minimzing use of events. This simplifies code and has performance benefits.
- Maintains internal buffers to accommodate a fundamental underlying difference between POSIX and Win32 IO async models - IOReady Vs IOComplete (Ex for a Read operation, POSIX APIs signal when IO is ready - data will be subsequently explicitly read, Win32 APIs signal when IO has completed - data is already copied to a user provided buffer. Though the internal buffer and additional copy may seem to be a performance hit, a validation exercise did not show any major impact. It in fact proved beneficial in reducing kernel calls during "read"s (ex. reading a header, would fetch the entire packet in a single call).
- Maintains Interrupt queue and interrupt handler mapping table. Queue is maintained to temporarily hold interrupts that are otherwise supported in Windows but handled in a different approach typically in a different thread. Ex SIGINT, SIGWINCH. These are processed inside of "wait_for_any" in the main thread. On processing any queued interrupt, this function will return SIGINT.
- Details on different interrupts handled by OPENSSH code and how they will be handled in Windows
| Signal | Detail |
|---|---|
| SIGINT | Windows invokes its Ctrl+C handler on a different thread, that handler queues the interrupt in the internal queue and handled by any of the blocking calls (select, etc) |
| SIGWINCH | Like Ctrl+C, a console windows size change event is captured in a native handled, queued and processed in one of the blocking calls |
| SIGILL, SIGTERM, SIGQUIT, WJSIGNAL,SIGTTIN, SIGTTOU | Not generated in Windows, these may be defined by handlers will never be invoked |
| SIGHUP | TBD |
| SIGCHLD | TBD |
| SIGTSTP | TBD |
| SIGPIPE | only accepts SIG_IGN handler (that's all OpenSSH uses) |
| SIGALARM | implemented internally inside the wrapper. handler automatically called when native timer set by CreateWaitableTimer expires |
- Additional details on underlying Win32 calls used
| POSIX IO call | Underlying Win32 IO call(s) | Additional details |
|---|---|---|
| accept | AcceptEx | No APC semantics so, explicit event used for async IO |
| connect | ConnectEx | No APC semantics so, explicit event used for async IO |
| send | WSASend | |
| recv | WSARecv | |
| open | CreateFile | |
| creat | CreateFile | |
| pipe | CreateNamedPipe | A uni directional named pipe with and internal name is created, CreateFile called to connect from other end |
| read | ReadFileEx | |
| write | WriteFileEx | |
| fdopen | TBD | |
| fstat | TBD | |
| dup, dup2 | SetStdHandle | only supported on standard IO file descriptors (used for IO redirection) |
| socketpair | CreateNamedPipe | A bi directional named pipe with an internal name is created, CreateFile called to connect from other end. This does not support AF_UNIX ancilliary messages. More details later |
| setitimer | CreateWaitableTimer |
A fully functional prototype (for socket, file and pipe IO) of this wrapper is available here
There is no easy fork() equivalent in Windows. fork() is used in OpenSSH in multiple places, of those - 3 are worth mentioning
- Session isolation: Each accepted connection in sshd is handed off and processed in a forked child. This will be implemented in Windows using CreateProcess based custom logic - will need #def differentiated code between Unix and Windows
- Privilege separation: Implemented in OpenSSH by processing and parsing network data in forked and underprivileged child processes that communicate to privileged Monitor process through IPC. Monitor does the core crypto validation and authentication. Privilege downgrading is done by setuid(restricted_user). Security model in Windows will be different, running the SSHD service itself in a low privileged mode. So, the whole Privilege separation relevant code is not needed and will be disabled for Windows.
- sftp and scp: sftp and scp client side utilities invoke ssh using fork() and exec(). This logic will be disabled and substituted with CreateProcess based one.
- The rest of the places fork() is used is listed below. None of these are critical to the functionality in Windows and will be appropriately disabled for Windows.
- TBD
Unix domain sockets are used for IPC communication between processes on the same host. Apart from providing stream/datagram modes, they also support a secure way to transmit ancillary data (like file descriptors). The only place ancillary data is used in OpenSSH is in "ProxyUseFDPass" feature where a proxy command is issued by ssh client to create a connected socket, and its FD is transmitted back over IPC. This feature will be disabled on Windows. The rest of the places AF_UNIX sockets are used:
- ControlMaster - used to multiplex multiple sessions over a single SSH connection.
- SSHAgent - used to managed store keys and crypto validation based on those. SSH agent and key management for Windows are discussed later in this document.
- Local Socket Forwarding - This is forwarding traffic to AF_UNIX sockets and this feature is not applicable in Windows
- SSHD rexec - Not applicable for Windows. SSHD will be implemented as Windows service, that can be configured for auto restart.
- SSHD from inetd - Not applicable for Windows.
AF_UNIX channel will be implemented using secure bidirectional named pipes in Windows. This does not support ancillary data but is sufficient for above listed features relevant in Windows.
SSHD will be implemented as a Windows service, running in its virtual account context - NT Service\SSHD - this is a restricted account that will only be granted the following needed privileges (primarily needed to spawn off processes as client user):
- SE_ASSIGNPRIMARYTOKEN_NAME
- SE_INCREASE_QUOTA_NAME
ssh-agent will be reimplemented for Windows as a Windows service, running as LocalSystem with TCB privileges (equivalent to root on Linux). Unlike in Unix, ssh-agent will listen on a known static IPC port. This is done as a security measure to protect ssh-agent port from hijack/spoof attacks. It serves the following requests that need be processed at SYSTEM privilege level:
- Register a host key - All host keys, to be used by ssh deamon for host authentication can be securely registered with ssh-agent. The registration process will be similar to ssh-add usage in Unix. Host keys will be internally encrypted using DPAPI using OS System account.
- Register a user key - User keys, can be securely one-time registered with ssh-agent for a single sign-on experience. These keys are DPAI encrypted using user's password and ACL'ed as SYSTEM only. This ensures that malware running under user's context can never steal key material.
- Delete a host or a user key - Similar to ssh-add usage in Unix.
- signature generation and validation - using a registered key.
The above listed requests are similar to what ssh-agent serves in Unix. In addition, on Windows, ssh-agent will also serve the following operations
- Authentication: ssh-agent will currently serve Basic and Key-Based authentication. It will be responsible for generating the client/user token once authentication succeeds. This includes:
- key authentication - ensuring validity of public key mapping, validating a signed payload as part of client key based authentication and generating a Windows user token. Token generation is done using S4U for domain accounts and a custom SSP for local accounts.
- Basic authentication - done using LogonUser.
As detailed earlier, session isolation in Windows will be done using CreateProcess based custom logic (in place of fork based logic in Unix). Spawned child process will run as NT Service\SSHD too.
End result of authentication in Windows is a Windows user token (if authentication succeeds). SSH sessions that need client user capabilities are hosted in processes running under the context of client user (launched using CreateProcess(user_token)). Ex. cmd.exe for terminal session, sftp_server.exe for sftp session and scp.exe for scp session.
Shown below is a high level overview of the various SSH components and access boundaries for various resources involved:
