There may be some null-dereference bugs in dtls.c

[tluio]

In handle_handshake_msg(), the variable peer is checked in:
if (peer) {
dtls_stop_retransmission(ctx, peer);
}
This indicates that peer can be NULL.
If so, some null-dereference bugs will occur in handle_handshake_msg().
peer is also checked in line 3436:
if (peer && !peer->handshake_params)
and it is dereferenced in line 3451:
peer->state = DTLS_STATE_CLIENTHELLO;

[boaks]

On the server side, a CLIENT_HELLO is processed "stateless", so only for that case the peer may be NULL. With that, the NULL must be only considered processing such a CLIENT_HELLO. Maybe, moving the stateless part into a own function makes the code more readable and easier to ensure, that the peer is not NULL in the other cases. Or a check as:

  if (peer) {
    dtls_stop_retransmission(ctx, peer);
  } else if (data[0] != DTLS_HT_CLIENT_HELLO) {
        return dtls_alert_fatal_create(DTLS_ALERT_INTERNAL_ERROR);
  }

Just to mention:

That check is already done in handle_handshake:

  if (!peer || !peer->handshake_params) {
    /* This is the initial ClientHello */
    if (hs_header->msg_type != DTLS_HT_CLIENT_HELLO && !peer) {
      dtls_warn("If there is no peer only ClientHello is allowed\n");
      return dtls_alert_fatal_create(DTLS_ALERT_HANDSHAKE_FAILURE);
    }

[boaks]

Fixed with PR #89

[boaks]

@obgm

That may be closed.

[obgm]

Thanks, closing.