Introduction
Managing PCF or other cloud platforms requires a solid Control Plane so that we can drive the platforms in an automated way.There is a reference architecture for how to build Control Plane for PCF. But frankly speaking, it's a great generic design for all BOSH-managed clusters.
What Components are in Control Plane?
If you ask a question like that to me, I'd start with "it depends".But it's common to have components like:
- A Jumpbox where you can start things from;
- A BOSH Director so all Control Plane workloads can be managed and benefited by BOSH's great capacity;
- Some Control Plane workloads like
- Concourse Cluster
- Prometheus Cluster with Grafana
- Minio
- And more
There are already BOSH releases available for all Control Plane workloads mentioned above. So deploying these is just a simply `bosh deploy`.
Here I'd like to focus on how to bootstrap our Control Plane.
Ways to Bootstrap Control Plane
There are many ways to bootstrap our Control Plane but below are two of frequently used patterns.
- Manually provision a Jumpbox -> Install necessary tools in Jumpbox -> use bosh create-env to create a BOSH Director -> use bosh deploy to create BOSH releases
- Use BOSH BootLoader's (bbl) bbl up to create a Jumpbox and BOSH Director -> use bosh deploy to create BOSH releases
As you may have seen, using BBL can simply the process of building our Control Plane. Furthermore, as it offers a series of out-of-the-box features, we can benefit a lot if we go this way.
Let's find out how.
The BBL Way
1. Prepare An "init.sh" File
It's a good practice to prepare a simple init.sh file to export some parameters even you can simple just run some export commands directly.
vSphere:
$ cat > init.sh <<EOF export BBL_IAAS=vsphere export BBL_VSPHERE_VCENTER_USER= export BBL_VSPHERE_VCENTER_PASSWORD= export BBL_VSPHERE_VCENTER_IP= export BBL_VSPHERE_VCENTER_DC= export BBL_VSPHERE_VCENTER_CLUSTER= export BBL_VSPHERE_VCENTER_RP= export BBL_VSPHERE_NETWORK= export BBL_VSPHERE_VCENTER_DS= export BBL_VSPHERE_SUBNET= export BBL_VSPHERE_VCENTER_DISKS= export BBL_VSPHERE_VCENTER_TEMPLATES= export BBL_VSPHERE_VCENTER_VMS= EOF
GCP:
cat > init.sh <<EOF export BBL_IAAS=gcp export BBL_GCP_REGION= export BBL_GCP_SERVICE_ACCOUNT_KEY= EOF
You may refer to other IaaS platform:
Now we can simply source it so that we expose the required parameters for further processing.
$ source init.sh
2. Generate a Plan: bbl plan
It's another good practice to plan it before actually executing because we may customize something.
$ bbl plan
This command will generate a series files based on the default settings:
-rw-r--r-- 1 user1 user1 587 Nov 10 20:47 bbl-state.json drwxrwxr-x 19 user1 user1 4096 Nov 10 20:47 bosh-deployment drwxrwxr-x 2 user1 user1 4096 Nov 10 20:47 cloud-config -rwxr-x--- 1 user1 user1 644 Nov 10 20:47 create-director.sh -rwxr-x--- 1 user1 user1 569 Nov 10 20:47 create-jumpbox.sh -rwxr-x--- 1 user1 user1 644 Nov 10 20:47 delete-director.sh -rwxr-x--- 1 user1 user1 569 Nov 10 20:47 delete-jumpbox.sh -rwxrwxr-x 1 user1 user1 568 Nov 5 14:17 init.sh drwxrwxr-x 7 user1 user1 4096 Nov 10 20:47 jumpbox-deployment drwxrwxr-x 3 user1 user1 4096 Nov 10 20:47 terraform drwxr----- 2 user1 user1 4096 Nov 10 20:47 vars
As you can see, it generates a couple of files and folders.
Among them, there are something to highlight here:
bosh-deployment --- this is a copy of bosh-deployment cloud-config --- there is a cloud config file which you can customize create-director.sh --- a script file which will be used to create director create-jumpbox.sh --- a script file which will be used to create jumpbox delete-director.sh --- a script file which will be used to delete director delete-jumpbox.sh --- a script file which will be used to create jumpbox init.sh --- our init file jumpbox-deployment --- this is a copy of jumpbox-deployment terraform --- a folder which contains all terraform files vars --- a folder which contains var files
It's NOT recommended to change these generated files directly if you want to customize, as another bbl plan command will replace your customization effort.
There are some conventions to customize the deployment. Below are frequently used methods:
- By adding a *-override.sh file for the sh files like create-jumpbox.sh, create-director.sh
- By adding a new yaml file as the ops files.
For example, if we want to customize the way we create director later, say adding custom dns, we can do something like this:
$ cp create-director.sh create-director-override.sh $ vi create-director-override.sh #!/bin/sh bosh create-env \ ${BBL_STATE_DIR}/bosh-deployment/bosh.yml \ --state ${BBL_STATE_DIR}/vars/bosh-state.json \ --vars-store ${BBL_STATE_DIR}/vars/director-vars-store.yml \ --vars-file ${BBL_STATE_DIR}/vars/director-vars-file.yml \ -o ${BBL_STATE_DIR}/bosh-deployment/vsphere/cpi.yml \ -o ${BBL_STATE_DIR}/bosh-deployment/jumpbox-user.yml \ -o ${BBL_STATE_DIR}/bosh-deployment/uaa.yml \ -o ${BBL_STATE_DIR}/bosh-deployment/credhub.yml \ -o ${BBL_STATE_DIR}/bosh-deployment/vsphere/resource-pool.yml \ -o ${BBL_STATE_DIR}/bosh-deployment/misc/dns.yml \ -v vcenter_user="${BBL_VSPHERE_VCENTER_USER}" \ -v vcenter_password="${BBL_VSPHERE_VCENTER_PASSWORD}" \ -v internal_dns=[10.193.239.2] \ -v internal_cidr=10.193.239.0/24 \ -v internal_ip=10.193.239.41
At this case, the bbl tool will use the -override.sh file instead of the originally generated file to execute.
Or if you want to add another ops file to customize the default network:
$ cat > ops-cloud-config-network.yml <<EOF - type: replace path: /networks/name=default/subnets/0/static? value: [11.193.239.50-10.193.239.70] - type: replace path: /networks/name=default/subnets/0/reserved? value: [11.193.239.1-10.193.239.49] EOF $ cp ops-cloud-config-network.yml cloud-config/
At this case, the newly added ops file will be automatically added and merged as another ops file.
Of course, if you really want to update further on cloud config, you can do it anytime, even after the deployment.
3. Execute It: bbl up
Once you've done the customization, you can issue the bbl up command to execute it.
$ bbl up step: terraform init step: terraform apply step: creating jumpbox Deployment manifest: '/home/user1/bbl/jumpbox-deployment/jumpbox.yml' Deployment state: '/home/user1/bbl/vars/jumpbox-state.json' Started validating Downloading release 'os-conf'... Skipped [Found in local cache] (00:00:00) Validating release 'os-conf'... Finished (00:00:00) Downloading release 'bosh-vsphere-cpi'... Skipped [Found in local cache] (00:00:00) Validating release 'bosh-vsphere-cpi'... Finished (00:00:00) Validating cpi release... Finished (00:00:00) Validating deployment manifest... Finished (00:00:00) Downloading stemcell... Skipped [Found in local cache] (00:00:00) Validating stemcell... Finished (00:00:03) Finished validating (00:00:03) Started installing CPI Compiling package 'ruby-2.4-r3/8471dec5da9ecc321686b8990a5ad2cc84529254'... Finished (00:01:44) Compiling package 'vsphere_cpi/3049e51ead9d72268c1f6dfb5b471cbc7e2d6816'... Finished (00:00:50) Compiling package 'iso9660wrap/82cd03afdce1985db8c9d7dba5e5200bcc6b5aa8'... Finished (00:00:00) Installing packages... Finished (00:00:00) Rendering job templates... Finished (00:00:00) Installing job 'vsphere_cpi'... Finished (00:00:00) Finished installing CPI (00:02:35) Starting registry... Finished (00:00:00) Uploading stemcell 'bosh-vsphere-esxi-ubuntu-trusty-go_agent/3468.17'... Finished (00:00:30) Started deploying Creating VM for instance 'jumpbox/0' from stemcell 'sc-4227b41a-f52a-4192-bfce-02f7cf802067'... Finished (00:00:21) Waiting for the agent on VM 'vm-3f9dccaf-a8e5-4214-bc6e-8413c6ff4dfb' to be ready... Finished (00:00:18) Rendering job templates... Finished (00:00:00) Updating instance 'jumpbox/0'... Finished (00:00:11) Waiting for instance 'jumpbox/0' to be running... Finished (00:00:00) Running the post-start scripts 'jumpbox/0'... Finished (00:00:01) Finished deploying (00:00:57) Stopping registry... Finished (00:00:00) Cleaning up rendered CPI jobs... Finished (00:00:00) Succeeded step: created jumpbox step: creating bosh director Deployment manifest: '/home/user1/bbl/bosh-deployment/bosh.yml' Deployment state: '/home/user1/bbl/vars/bosh-state.json' Started validating Downloading release 'bosh'... Skipped [Found in local cache] (00:00:00) Validating release 'bosh'... Finished (00:00:00) Downloading release 'bpm'... Skipped [Found in local cache] (00:00:00) Validating release 'bpm'... Finished (00:00:01) Downloading release 'bosh-vsphere-cpi'... Skipped [Found in local cache] (00:00:00) Validating release 'bosh-vsphere-cpi'... Finished (00:00:00) Downloading release 'os-conf'... Skipped [Found in local cache] (00:00:00) Validating release 'os-conf'... Finished (00:00:00) Downloading release 'uaa'... Skipped [Found in local cache] (00:00:00) Validating release 'uaa'... Finished (00:00:03) Downloading release 'credhub'... Skipped [Found in local cache] (00:00:00) Validating release 'credhub'... Finished (00:00:01) Validating cpi release... Finished (00:00:00) Validating deployment manifest... Finished (00:00:00) Downloading stemcell... Skipped [Found in local cache] (00:00:00) Validating stemcell... Finished (00:00:05) Finished validating (00:00:32) Started installing CPI Compiling package 'ruby-2.4-r4/0cdc60ed7fdb326e605479e9275346200af30a25'... Finished (00:01:46) Compiling package 'iso9660wrap/82cd03afdce1985db8c9d7dba5e5200bcc6b5aa8'... Finished (00:00:00) Compiling package 'vsphere_cpi/e1a84e5bd82eb1abfe9088a2d547e2cecf6cf315'... Finished (00:00:52) Installing packages... Finished (00:00:00) Rendering job templates... Finished (00:00:00) Installing job 'vsphere_cpi'... Finished (00:00:00) Finished installing CPI (00:02:40) Starting registry... Finished (00:00:00) Uploading stemcell 'bosh-vsphere-esxi-ubuntu-xenial-go_agent/97.12'... Finished (00:00:40) Started deploying Creating VM for instance 'bosh/0' from stemcell 'sc-dea8d9a0-e423-4b92-8cc3-b72dae27f65e'... Finished (00:00:27) Waiting for the agent on VM 'vm-c5b55b02-91ca-4986-9eab-558fc35245f7' to be ready... Finished (00:00:15) Creating disk... Finished (00:00:06) Attaching disk 'disk-2afadc6c-5d44-4681-8fea-8b1b2ceefe78' to VM 'vm-c5b55b02-91ca-4986-9eab-558fc35245f7'... Finished (00:00:22) Rendering job templates... Finished (00:00:09) Compiling package 'ruby-2.4-r4/0cdc60ed7fdb326e605479e9275346200af30a25'... Skipped [Package already compiled] (00:00:00) Compiling package 'openjdk_1.8.0/4d45452ce6bd79122873640ac63cae4d9b419ed4'... Skipped [Package already compiled] (00:00:00) Compiling package 'bpm-runc/c0b41921c5063378870a7c8867c6dc1aa84e7d85'... Skipped [Package already compiled] (00:00:00) Compiling package 'golang/27413c6b5a88ea20a24a9eed74d4b090b7b88331'... Skipped [Package already compiled] (00:00:01) Compiling package 'golang-1.9-linux/8d6c67abda8684ce454f0bc74050a213456573ff'... Skipped [Package already compiled] (00:00:01) Compiling package 'mysql/898f50dde093c366a644964ccb308a5281c226de'... Skipped [Package already compiled] (00:00:00) Compiling package 'libpq/e2414662250d0498c194c688679661e09ffaa66e'... Skipped [Package already compiled] (00:00:00) Compiling package 'ruby-2.4-r4/0cdc60ed7fdb326e605479e9275346200af30a25'... Finished (00:01:51) Compiling package 'health_monitor/2ea21f1adae7dd864b38dff926675ba4fca89ef0'... Skipped [Package already compiled] (00:00:00) Compiling package 'verify_multidigest/8fc5d654cebad7725c34bb08b3f60b912db7094a'... Skipped [Package already compiled] (00:00:00) Compiling package 'davcli/f8a86e0b88dd22cb03dec04e42bdca86b07f79c3'... Skipped [Package already compiled] (00:00:00) Compiling package 'lunaclient/b922e045db5246ec742f0c4d1496844942d6167a'... Skipped [Package already compiled] (00:00:00) Compiling package 'bosh-gcscli/fce60f2d82653ea7e08c768f077c9c4a738d0c39'... Skipped [Package already compiled] (00:00:00) Compiling package 'credhub/62f912abb406d6d9b49393be629713fd407328c7'... Skipped [Package already compiled] (00:00:00) Compiling package 'nginx/5a68865452a3bdcc233867edbbb59c1e18658f6b'... Skipped [Package already compiled] (00:00:00) Compiling package 'gonats/73ec55f11c24dd7c02288cdffa24446023678cc2'... Skipped [Package already compiled] (00:00:00) Compiling package 'bpm/d139b63561eaa3893976416be9668dea539bf17d'... Skipped [Package already compiled] (00:00:00) Compiling package 'configurator/d19e331ac9c867c132d19426007802f86070526a'... Skipped [Package already compiled] (00:00:00) Compiling package 'uaa/87da0e8d38c63e84fda7069ce77285419399623d'... Skipped [Package already compiled] (00:00:01) Compiling package 's3cli/3097f27cb9356172c9ae52de945821c4e338c87a'... Skipped [Package already compiled] (00:00:00) Compiling package 'uaa_utils/90097ea98715a560867052a2ff0916ec3460aabb'... Skipped [Package already compiled] (00:00:00) Compiling package 'postgres-9.4/52b3a31d7b0282d342aa7a0d62d8b419358c6b6b'... Skipped [Package already compiled] (00:00:00) Compiling package 'iso9660wrap/82cd03afdce1985db8c9d7dba5e5200bcc6b5aa8'... Finished (00:00:01) Compiling package 'director/81a742bcbbb4f6eabea846365b9fd491d8d2fff8'... Skipped [Package already compiled] (00:00:00) Compiling package 'vsphere_cpi/e1a84e5bd82eb1abfe9088a2d547e2cecf6cf315'... Finished (00:00:53) Updating instance 'bosh/0'... Finished (00:01:00) Waiting for instance 'bosh/0' to be running... Finished (00:01:38) Running the post-start scripts 'bosh/0'... Finished (00:00:02) Finished deploying (00:06:57) Stopping registry... Finished (00:00:00) Cleaning up rendered CPI jobs... Finished (00:00:00) Succeeded step: created bosh director step: generating cloud config step: applying cloud config step: applying runtime config
As a result, below items will be ready and deployed:
- A BOSH Director
- A Jumpbox VM
- A set of randomly generated BOSH director credentials
- A generated keypair allowing you to SSH into the BOSH Director and any instances BOSH deploys
- A copy of the manifest the BOSH Director was deployed with
- A basic cloud config
4. Verify the deployment
The easiest way to verify the environment may be as below:
$ eval "$(bbl print-env)" $ bosh vms Using environment 'https://10.193.239.41:25555' as client 'admin' Succeeded
Yes, the BOSH Director is ready to rock!
What else? Well, you can do some interesting things like ssh into Jumpbox VM:
$ bbl ssh --jumpbox ... jumpbox/0:~$
Or ssh into BOSH Director VM for troubleshooting purposes:
$ bbl ssh --director ... bosh/0:~$ sudo su - bosh/0:~# monit summary The Monit daemon 5.2.5 uptime: 10m Process 'nats' running Process 'postgres' running Process 'blobstore_nginx' running Process 'director' running Process 'worker_1' running Process 'worker_2' running Process 'worker_3' running Process 'worker_4' running Process 'director_scheduler' running Process 'director_sync_dns' running Process 'director_nginx' running Process 'health_monitor' running Process 'uaa' running Process 'credhub' running System 'system_localhost' running
What's next? Having a BOSH Director is a really a great start for deploying cool software, like Concourse, Prometheus, Mino, MySQL, Kafka or whatever.
Meanwhile, you may discover BBL more by accessing its GitHub repo, here.
Enjoy!
