How-to Series: How to `bosh scp` files into BOSH-managed VMs' Folders

Issue / Problem Statement

If we bosh scp directly to folders under /var/vcap/jobs/xxx/config, you will encounter Permission denied like:

$ bosh -e lite -d concourse4 scp concourse.yml web:/var/vcap/jobs/uaa/config/

Using environment '192.168.50.6' as client 'admin'

Using deployment 'concourse4'

Task 3687. Done
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stderr | Unauthorized use is strictly prohibited. All access and activity
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stderr | is subject to logging and monitoring.
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stderr | scp: /var/vcap/jobs/uaa/config//concourse.yml: Permission denied

Interestingly, if we bosh scp to some folders like `/tmp', it's fine:

$ bosh -e lite -d concourse4 scp concourse.yml web:/tmp/
Using environment '192.168.50.6' as client 'admin'

Using deployment 'concourse4'

Task 3689. Done
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stderr | Unauthorized use is strictly prohibited. All access and activity
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stderr | is subject to logging and monitoring.

Let's verify it:

$ bosh -e lite -d concourse4 ssh web -c 'ls -la /tmp/'
Using environment '192.168.50.6' as client 'admin'

Using deployment 'concourse4'

Task 3693. Done
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stderr | Unauthorized use is strictly prohibited. All access and activity
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stderr | is subject to logging and monitoring.
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stdout | total 2128
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stdout | drwxrwx--T  4 root                 vcap                    4096 Oct 27 13:27 .
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stdout | drwxr-xr-x 34 root                 root                    4096 Oct 25 08:55 ..
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stdout | -rw-r--r--  1 root                 root                  233394 Oct 25 08:57 ca-certificates.crt
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stdout | -rw-r--r--  1 bosh_99ef981c59fa482 bosh_99ef981c59fa482    2496 Oct 27 13:27 concourse.yml
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stdout | -rwxr-xr-x  1 root                 root                 1923450 Aug 24 05:46 garden-init
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stdout | drwxr-xr-x  2 root                 root                    4096 Oct 25 08:57 hsperfdata_root
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stdout | drwxr-x---  2 vcap                 vcap                    4096 Oct 25 08:57 hsperfdata_vcap
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stderr | Connection to 10.244.0.104 closed.

Yes, we can see that the file concourse.yml has been successfully copied into our BOSH-managed VM node web.

Now if we really want to scp file(s) to the right place under /var/vcap/jobs/xxx/config, how?

Solution

As you have seen we can copy over to /tmp, let's wrap it up with some more scripts.

$ bosh -e lite -d concourse4 scp concourse.yml web:/tmp/
$ bosh -e lite -d concourse4 ssh web -c 'sudo cp /tmp/concourse.yml /var/vcap/jobs/uaa/config/'

Let's verify it again:

$ bosh -e lite -d concourse4 ssh web -c 'sudo ls -al /var/vcap/jobs/uaa/config/'
Using environment '192.168.50.6' as client 'admin'

Using deployment 'concourse4'

Task 3699. Done
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stderr | Unauthorized use is strictly prohibited. All access and activity
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stderr | is subject to logging and monitoring.
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stdout | total 60
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stdout | drwxr-x--- 3 root vcap  4096 Oct 27 13:34 .
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stdout | drwxr-x--- 5 root vcap  4096 Oct 25 08:56 ..
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stdout | -rw-r----- 1 root vcap   420 Oct 25 08:56 bpm.yml
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stdout | -rw-r--r-- 1 root root  2496 Oct 27 13:34 concourse.yml
...
web/4de05ffc-ae08-4dfa-a52a-a751e62b624f: stderr | Connection to 10.244.0.104 closed.

Yes, we made it! The file has been copied to the desired folder, say /var/vcap/jobs/uaa/config/.

I also posted it in my gist, here.

References:

1. https://bosh.io/docs/cli-v2/#scp

How-to Series: How to Quickly Setup OpenLDAP for Testing Purposes

Issue / Problem Statements

Like it or not, LDAP still plays heavy roles in most of the organizations as user store and authentication authority.
To setup a proper LDAP Server may require some LDAP knowledge and experience. So how to setup a LDAP in a quick (say <10mins) and easy way becomes obvious requirement.

Solution:

Highlights:

• Use Docker
• OpenLDAP
• With sample data (which can be customized of course) pre-loaded
• Provide sample `ldapsearch`

Steps:

1. Prepare users and group data, the *.ldif files

$ mkdir testdata

$ cat testdata/1-users.ldif
dn: ou=people,dc=bright,dc=com
ou: people
description: All people in organisation
objectclass: organizationalunit

# admin1
dn: cn=admin1,ou=people,dc=bright,dc=com
objectClass: inetOrgPerson
sn: admin1
cn: admin1
uid: admin1
mail: admin1@bright.com
# secret
userPassword: {SSHA}RRN6AM9u0tpTEOn6oBcIt9X3BbFPKVk5

# admin2
dn: cn=admin2,ou=people,dc=bright,dc=com
objectClass: inetOrgPerson
sn: admin2
cn: admin2
uid: admin2
mail: admin2@bright.com
# secret
userPassword: {SSHA}RRN6AM9u0tpTEOn6oBcIt9X3BbFPKVk5

# developer
dn: cn=developer1,ou=people,dc=bright,dc=com
objectClass: inetOrgPerson
sn: developer1
cn: developer1
mail: developer1@bright.com
userPassword: {SSHA}RRN6AM9u0tpTEOn6oBcIt9X3BbFPKVk5

$ cat testdata/2-groups.ldif
dn: ou=groups,dc=bright,dc=com
objectClass: organizationalUnit
ou: groups

dn: cn=admins,ou=groups,dc=bright,dc=com
objectClass: groupOfNames
cn: admins
member: cn=admin1,ou=people,dc=bright,dc=com
member: cn=admin2,ou=people,dc=bright,dc=com

dn: cn=developers,ou=groups,dc=bright,dc=com
objectClass: groupOfNames
cn: developers
member: cn=admin1,ou=people,dc=bright,dc=com
member: cn=developer1,ou=people,dc=bright,dc=com

2. Start It Up

$ docker run --name my-openldap-container \
    --env LDAP_ORGANISATION="Bright Inc" \
    --env LDAP_DOMAIN="bright.com" \
    --env LDAP_ADMIN_PASSWORD="secret" \
    --volume "$(pwd)"/testdata:/container/service/slapd/assets/config/bootstrap/ldif/custom \
    -p 10389:389 \
    --detach \
    osixia/openldap:1.2.2 --copy-service --loglevel debug

Now the LDAP service is exposed from container port 389 to local port 10389.

Note: to clean it up, use below command:

$ docker stop $(docker ps -aqf "name=my-openldap-container") && \
  docker rm $(docker ps -aqf "name=my-openldap-container")

3. Test It Out

$ ldapsearch -LLL -x \

    -H ldap://localhost:10389 \
    -D "cn=admin,dc=bright,dc=com" -w secret \
    -b 'dc=bright,dc=com' \
    dn
dn: dc=bright,dc=com
dn: cn=admin,dc=bright,dc=com
dn: ou=people,dc=bright,dc=com
dn: cn=admin1,ou=people,dc=bright,dc=com
dn: cn=admin2,ou=people,dc=bright,dc=com
dn: cn=developer1,ou=people,dc=bright,dc=com
dn: ou=groups,dc=bright,dc=com
dn: cn=admins,ou=groups,dc=bright,dc=com
dn: cn=developers,ou=groups,dc=bright,dc=com

$ ldapsearch -LLL -x \
    -H ldap://localhost:10389 \
    -D "cn=admin,dc=bright,dc=com" -w secret \
    -b 'dc=bright,dc=com' \
    '(&(objectClass=groupOfNames)(member=cn=admin2,ou=people,dc=bright,dc=com))' \
    cn
dn: cn=admins,ou=groups,dc=bright,dc=com
cn: admins

That's it. Hope it helps!

How-to Series: How To Correctly Enable Self-signed Cert for PCF - NSX-T Integration

Issue Description:

While trying to enable NSX-T for Pivotal Cloud Foundry (PCF), OpsMan complains about:

Error connecting to NSX IP: The NSX server's certificate is not signed with the provided NSX CA cert. Please provide the correct NSX CA cert', type: IaasConfigurationVerifier




Issue Analysis:

As the connectivity between PCF and NSX-T Manager is secured by SSL, one has to configure the CA certificate in PEM format that authenticates to the NSX-T Manager.
Refer to below screenshot for the configuration requirements.


Now the issue is how to get back the right CA cert and what should be the right way to handle this situation.

Solution:

If you're using a simple self-signed cert, you can probably try below "quick" solution -- simply use openssl tool to get the cert back, as suggested in the doc here:

openssl s_client -showcerts \
  -connect NSX-MANAGER-ADDRESS:443 < /dev/null 2> /dev/null | openssl x509

So that you can simply get the cert and put it there.

But sometimes it won't work so I'd always recommend a "better" way and the steps are as below:

1. Generate NSX-T CSR

Log on to the NSX Manager and navigate to System|Trust, click CSRs tab and then “Generate CSR”, populate the certificate request and click Save.
Select the new CSR and click Actions|Download CSR PEM to save the exported CSR in PEM format.


2. Ask Your CA to Sign This CSR

Submit the CSR file to your CA to get it signed and get back the new certificate.

Some organization has a CA chain and use the intermediate CA to sign the real CSR. At this case, please remember to make it a full cert chain by having certs concatenated (simply copy & paste to the end of the cert, one by one in a good text editor like Visual Studio Code) as:
- Newly signed NSX Certificate
- Intermediate CA
- Root CA

But sometimes there is no official internal CA, I'd always recommend to generate one so that you can have better control and easier trust link while handling big distributed systems like PCF.
I compiled a simple two-step process for doing that:
- Gist of generate-internal-ca.md
- Gist of sign-certs-with-internal-ca.md


3. Configure NSX Manager

Now assuming we already got our CSR signed and with you there is the concatenated certs.

In NSX Manager, select the CSRs tab and click Actions | Import Certificate for CSR.
In the window, paste in the concatenated certificates from above and click save.

Now you’ll have a new certificate and CA certs listed under Certificates tab.


4. Apply the Certs

Under Certificates tab, copy the full ID of the newly added certificate (not CA).
Note: somehow the UI only shows a portion of the ID by default, click it to display the full ID and copy it to the clip board.

Launch RESTClient in Firefox or Advanced REST client in Chrome.
Make sure have below request configuration:
- URL: https://<NSX Manager IP or FQDN>api/v1/node/services/http?action=apply_certificate&certificate_id=<certificate ID copied in previous step>
- Authentication: Basic Authentication, enter the NSX Manager credentials for Username and Password
- Method: POST

click SEND button and make sure that the response status code is 200.

Refresh browser session to NSX Manager GUI to confirm new certificate is in use.


Ref: 
1. https://brianragazzi.wordpress.com/2018/03/08/replacing-the-self-signed-certificate-on-nsx-t/