Skip to main content

Connector: Flapping Offline/Online or Unavailable

Aaron
  • Updated
In this article:
    Applicable to:
    • Twingate Component: Connector

    Overview

    The Twingate Connector is sensitive to 5 seconds of clock drift from the Twingate Controller. In some environments, clock sync may not be taking place as expected, thus the Connector can shift out of the clock drift threshold.

    Symptoms

    • Email alerts received for the Connector flapping offline/online. 
    • After several days after a system restart or container deployment, the Twingate Connector suddenly is unavailable or becomes intermittently available.
    • Non-debug level Twingate Connector Logs include the below snippet: 
      controller_t::operator(): failed to get an access token: Invalid token
      controller_t::operator(): failed to get SD: Invalid token, err code 1
    • Debug level Twingate Connector Logs include similar above Invalid token errors as seen in the below snippet:
      Jun 26 21:39:49 myconnector twingate-connector[90815]: operator(): got HTTP request 1726182465575576172 successful response
      Jun 26 21:39:49 myconnector twingate-connector[90815]: controller_t::verify_token: {"typ":"DAT","alg":"ES256","kid":"Yg4pCAUFYXMDZnogGE80WV7Tpw=="} {"nt":"AN","aid":"11111","did":"222222","jti":"b23349978d62-8c5b-8f78-4b1e-65c84959","iss":"twingate","aud":"mytenant","exp":1656283197,"iat":1656279597,"ver":"4","tid":"3333","rnw":1656279897,"rnetid":"12041"}
      Jun 26 21:39:49 myconnector twingate-connector[90815]: controller_t::parse_verify_token: token verification failed: token verification failed: token expired
      Jun 26 21:39:49 myconnector twingate-connector[90815]: http::response::from: certificate e5afa0458e4f44e01a3da24328f329d2c866fae6b51dc8761d59f36428e8df21, issuer: C=US, O=Let's Encrypt, CN=R3, subject: CN=twingate.com
      Jun 26 21:39:49 myconnector twingate-connector[90815]: http::request::handle_response: POST "https://mytenant.twingate.com/api/v4/connector/ant" 200 OK (duration 0 sec)
      Jun 26 21:39:49 myconnector twingate-connector[90815]: operator(): got HTTP request 6525086497920236858 successful response
      Jun 26 21:39:49 myconnector twingate-connector[90815]: controller_t::verify_token: {"typ":"ANT","alg":"ES256","kid":"Yg4pCAUFYXMDZnogGE80WV7Tpw=="} {"jti":"6ce9c122e296-41ac-bab1-8c9d-aab9c1f3","iss":"twingate","aud":"mytenant","exp":1656279897,"iat":1656279597,"ver":"4","tid":"3333","rnw":1656279897,"nt":"AN","aid":"11111","did":"222222"}
      Jun 26 21:39:49 myconnector twingate-connector[90815]: controller_t::parse_verify_token: token verification failed: token verification failed: token expired
      Jun 26 21:39:49 myconnector twingate-connector[90815]: controller_t::operator(): failed to fetch ANT: Invalid token
      Jun 26 21:39:49 myconnector twingate-connector[90815]: controller_t::operator(): scheduling the next attempt in 30 seconds

    Troubleshooting

    Method 1

    1. Check the Time Offset found within the Connector Details in the screenshot below.
      Further information can be found at Connector metadata.
    2. If the time is 5s or greater, see the Resolution section below. 

    Note: In some scenarios the clock on the system flaps, monitor the Time Offset to see if it is variable.

    Surfacing additional Connector metadata

     

    Method 2

    1. Enable debug level logs for the connector as per Twingate Connector Logs.
    2. For the line with verify_token: {"typ":"DAT" , copy out the system timestamp and the value for iat (issued at), such as Jun 26 21:39:49 and 1656279597
      Note: if your log output does not include timestamps, you may be missing an argument such as use of either -t, --timestamps in Docker.
    3. Take the value for iat and convert it from epoch to human readable. We can see 1656279597 converts to June 26 21:39:57.
    4. Compare the system timestamp Jun 26 21:39:49 against the iat timestamp to see what the difference is.
    5. If the difference of the controller issuing the token (iat timestamp) is greater than 5 seconds from that of the logged system timestamp reports, the Connector flapping is due to clock drift. In this example we can see there is a difference of 8 seconds from when the Controller issued the token to the system time of the Connector.

    Resolution

    • Ensure hardware clock is synced. If only utilizing ntpd, it would be suggested to install/run chronyd in tandem.
    • If the issue is exhibited by a container, you will need to perform the above action on the container host. If the host is a managed by a cloud provider, you will need to engage their support team to review host clock drift.

    Related articles