How-to Series: CredHub Login Cause "validity check failed" Error? Fix it by a surprising way!

Issue / Problem Statement

Today while rebuilding my demo Concourse cluster, which has UAA (see my PR #85) + CredHub (see my PR #111) integrated, I encountered a wired error while trying to run my scripts:

$ cat > connect-concourse-credhub.sh <<EOF
source connect-bosh-credhub.sh

A="\$( credhub get -n /bosh-lite/concourse/credhub_admin_secret -j | jq -r .value )"
B="\$( credhub get -n /bosh-lite/concourse/atc_tls -j | jq -r .value.ca )"

export CREDHUB_SERVER=https://concourse.test:8844
export CREDHUB_CLIENT=credhub-admin
export CREDHUB_SECRET=\${A}
export CREDHUB_CA_CERT=\${B}

credhub login
EOF

The idea was very simple:

1. Retrieve the credentials from BOSH CredHub;
2. Export CredHub system variables and then login

Surprisingly, I kept getting below error while trying to execute the credhub login command:

invalid_token: An I/O error occurred while reading from the JWK Set source: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed

The thing was, while checking UAA and CredHub, everything looked okay!

Tried reset CredHub by `rm -rf ~/.credhub` and redeployed Concourse couldn't help.

Solution

I had to re-look at this issue again: validity check failed.
What could cause validity checking failure? Too many possibilities!

But it was obvious that the error was NOT thrown at backend, but frontend. And since it complained about token, let's check the token first.

$ credhub --token

We would get back the token that was used for login.

Open jwt.io website and paste the retrieved token to the "Encoded" box and I could see something like this:

{
  "jti": "5697ecd271b042fcbc65831bd25264c9",
  "sub": "credhub-admin",
  "authorities": [
    "credhub.write",
    "credhub.read"
  ],
  "scope": [
    "credhub.write",
    "credhub.read"
  ],
  "client_id": "credhub-admin",
  "cid": "credhub-admin",
  "azp": "credhub-admin",
  "revocable": true,
  "grant_type": "client_credentials",
  "rev_sig": "6131f65e",
  "iat": 1546099456,
  "exp": 1546103056,
  "iss": "https://concourse.test:8443/oauth/token",
  "zid": "uaa",
  "aud": [
    "credhub-admin",
    "credhub"
  ]
}

I checked the "exp" which stands for expiry and tried it out in https://www.epochconverter.com/.

Which showed:

GMT: Saturday, December 29, 2018 5:04:16 PM
Your time zone: Sunday, December 30, 2018 1:04:16 AM GMT+08:00
Relative: A day ago

OMG, this just-issued token was expired a day ago!!!

Why? Checked out the VM time -- yeah! It was obviously wrong.

So changed it back to the right Date/Time:

# date +%Y%m%d -s "20181231"
# date +%T -s "13:00:00"

After restart of the processes, everything went well.

./connect-concourse-credhub.sh
Setting the target url: https://192.168.50.6:8844
Login Successful
Setting the target url: https://concourse.test:8844
Login Successful