Thursday, 20 October 2016

Integrated Continuous Deployment (CD) Solution with Mesos, Zookeeper, Marathon on top of CI

Overview

After development of our RESTful webservices (see post here) and CI exercises (see post here), now we eventually reach the Continuous Deployment (CD) part.
Obviously, we’re going to experiment a Docker-based CD process.

Below components will be engaged to build up our integrated CD solution:

  • The Docker image we built and published previously in our private Docker Registry;
  • Apache Mosos: The DC/OS platform for managing our CPU, memory, storage, and other compute resources within our distributed clusters;
  • Apache ZooKeeper: A highly reliable distributed coordination tool for Apache Mosos cluster;
  • Mesosphere Marathon: The container orchestration platform for Apache Mesos.

What we’re going to focus in this chapter is as highlighted as below:



Table of Contents

Update docker-compose.yml By Adding CD Components

Having the infrastructure setup for CI previously, it’s pretty easy to add more components into our Docker Compose configuration file for our CD solution which includes:
  • ZooKeeper
  • Mesos: Master, Slave
  • Marathon

ZooKeeper

~/docker-compose.yml
…
# Zookeeper
zookeeper:
  image: jplock/zookeeper:3.4.5
  ports:
    - "2181"

Mesos: mesos-master, mesos-slave

~/docker-compose.yml
…
# Mesos Master
mesos-master:
  image: mesosphere/mesos-master:0.28.1-2.0.20.ubuntu1404
  hostname: mesos-master
  links:
    - zookeeper:zookeeper
  environment:
    - MESOS_ZK=zk://zookeeper:2181/mesos
    - MESOS_QUORUM=1
    - MESOS_WORK_DIR=/var/lib/mesos
    - MESOS_LOG_DIR=/var/log
  ports:
    - "5050:5050"

# Mesos Slave
mesos-slave:
  image: mesosphere/mesos-slave:0.28.1-2.0.20.ubuntu1404
  links:
    - zookeeper:zookeeper
    - mesos-master:mesos-master
  environment:
    - MESOS_MASTER=zk://zookeeper:2181/mesos
    - MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins
    - MESOS_CONTAINERIZERS=docker,mesos
    - MESOS_ISOLATOR=cgroups/cpu,cgroups/mem
    - MESOS_LOG_DIR=/var/log
  volumes:
    - /var/run/docker.sock:/run/docker.sock
- /usr/bin/docker:/usr/bin/docker

    - /sys:/sys:ro
- mesosslace-stuff:/var/log

    - /lib/x86_64-linux-gnu/libsystemd-journal.so.0:/lib/x86_64-linux-gnu/libsystemd-journal.so.0
    - /usr/lib/x86_64-linux-gnu/libapparmor.so.1:/usr/lib/x86_64-linux-gnu/libapparmor.so.1
    - /lib/x86_64-linux-gnu/libcgmanager.so.0:/lib/x86_64-linux-gnu/libcgmanager.so.0
    - /lib/x86_64-linux-gnu/libnih.so.1:/lib/x86_64-linux-gnu/libnih.so.1
    - /lib/x86_64-linux-gnu/libnih-dbus.so.1:/lib/x86_64-linux-gnu/libnih-dbus.so.1
    - /lib/x86_64-linux-gnu/libgcrypt.so.11:/lib/x86_64-linux-gnu/libgcrypt.so.11
  expose:
    - "5051"

Marathon

~/docker-compose.yml
…
# Marathon
marathon:
  image: mesosphere/marathon
  links:
    - zookeeper:zookeeper
  ports:
    - "8080:8080"
  command: --master zk://zookeeper:2181/mesos --zk zk://zookeeper:2181/marathon

The Updated docker-compose.yml

Now the final updated ~/docker-compose.yml is as below:
# Git Server
gogs:
  image: 'gogs/gogs:latest'
  #restart: always
  hostname: '192.168.56.118'
  ports:
    - '3000:3000'
    - '1022:22'
  volumes:
    - '/data/gogs:/data'

# Jenkins CI Server
jenkins:
  image: devops/jenkinsci
#  image: jenkinsci/jenkins
#  links:
#    - marathon:marathon
  volumes:
    - /data/jenkins:/var/jenkins_home

    - /opt/jdk/java_home:/opt/jdk/java_home
    - /opt/maven:/opt/maven
    - /data/mvn_repo:/data/mvn_repo

    - /var/run/docker.sock:/var/run/docker.sock
    - /usr/bin/docker:/usr/bin/docker
    - /etc/default/docker:/etc/default/docker

    - /lib/x86_64-linux-gnu/libsystemd-journal.so.0:/lib/x86_64-linux-gnu/libsystemd-journal.so.0
    - /usr/lib/x86_64-linux-gnu/libapparmor.so.1:/usr/lib/x86_64-linux-gnu/libapparmor.so.1
    - /lib/x86_64-linux-gnu/libcgmanager.so.0:/lib/x86_64-linux-gnu/libcgmanager.so.0
    - /lib/x86_64-linux-gnu/libnih.so.1:/lib/x86_64-linux-gnu/libnih.so.1
    - /lib/x86_64-linux-gnu/libnih-dbus.so.1:/lib/x86_64-linux-gnu/libnih-dbus.so.1
    - /lib/x86_64-linux-gnu/libgcrypt.so.11:/lib/x86_64-linux-gnu/libgcrypt.so.11
  ports:
- "8081:8080"

# Private Docker Registry
docker-registry:
  image: registry
  volumes:
    - /data/registry:/var/lib/registry
  ports:
    - "5000:5000"

# Zookeeper
zookeeper:
  image: jplock/zookeeper:3.4.5
  ports:
    - "2181"

# Mesos Master
mesos-master:
  image: mesosphere/mesos-master:0.28.1-2.0.20.ubuntu1404
  hostname: mesos-master
  links:
    - zookeeper:zookeeper
  environment:
    - MESOS_ZK=zk://zookeeper:2181/mesos
    - MESOS_QUORUM=1
    - MESOS_WORK_DIR=/var/lib/mesos
    - MESOS_LOG_DIR=/var/log
  ports:
    - "5050:5050"

# Mesos Slave
mesos-slave:
  image: mesosphere/mesos-slave:0.28.1-2.0.20.ubuntu1404
  links:
    - zookeeper:zookeeper
    - mesos-master:mesos-master
  environment:
    - MESOS_MASTER=zk://zookeeper:2181/mesos
    - MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins
    - MESOS_CONTAINERIZERS=docker,mesos
    - MESOS_ISOLATOR=cgroups/cpu,cgroups/mem
    - MESOS_LOG_DIR=/var/log
  volumes:
    - /var/run/docker.sock:/run/docker.sock
     - /usr/bin/docker:/usr/bin/docker

    - /sys:/sys:ro
    - mesosslace-stuff:/var/log

    - /lib/x86_64-linux-gnu/libsystemd-journal.so.0:/lib/x86_64-linux-gnu/libsystemd-journal.so.0
    - /usr/lib/x86_64-linux-gnu/libapparmor.so.1:/usr/lib/x86_64-linux-gnu/libapparmor.so.1
    - /lib/x86_64-linux-gnu/libcgmanager.so.0:/lib/x86_64-linux-gnu/libcgmanager.so.0
    - /lib/x86_64-linux-gnu/libnih.so.1:/lib/x86_64-linux-gnu/libnih.so.1
    - /lib/x86_64-linux-gnu/libnih-dbus.so.1:/lib/x86_64-linux-gnu/libnih-dbus.so.1
    - /lib/x86_64-linux-gnu/libgcrypt.so.11:/lib/x86_64-linux-gnu/libgcrypt.so.11
  expose:
    - "5051"

# Marathon
marathon:
  image: mesosphere/marathon
  links:
    - zookeeper:zookeeper
  ports:
    - "8080:8080"
  command: --master zk://zookeeper:2181/mesos --zk zk://zookeeper:2181/marathon

Spin Up the Docker Containers

$ docker-compose up

Verify Our Components

Jenkins: http://192.168.56.118:8081/
Marathon: http://192.168.56.118:8080/

Deployment By Using Mesos + Marathon

Marathon provides RESTful APIs for managing the behaviors we're going to load into the platform, such as deleting an application, deploying an application etc.
So the idea we're going to illustrate is to:
  1. Prepare the payload, or the Marathon deployment file;
  2. Wrap the interactions with Marathon as simple deployment scripts which can eventually be "plugged" into Jenkins as one more build steps to kick off the CD pipeline.

Marathon Deployment File

Firstly we need to create the Marathon deployment file, which is in JSON format, for scheduling our application on Marathon.
Let’s name it marathon-app-springboot-jersey-swagger.json and put it under /data/jenkins/workspace/springboot-jersey-swagger (which will be eventually mapped to Jenkins container’s path of “/var/jenkins_home/workspace/springboot-jersey-swagger”, or "${WORKSPACE}/" so that it’s reachable in Jenkins).
$ sudo touch /data/jenkins/workspace/springboot-jersey-swagger/marathon-app-springboot-jersey-swagger.json
$ sudo vim /data/jenkins/workspace/springboot-jersey-swagger/marathon-app-springboot-jersey-swagger.json


/data/jenkins/workspace/springboot-jersey-swagger/marathon-app-springboot-jersey-swagger.json




{
    "id": "springboot-jersey-swagger", 
    "container": {
      "docker": {
        "image": "192.168.56.118:5000/devops/springboot-jersey-swagger:latest",
        "network": "BRIDGE",
        "portMappings": [
          {"containerPort": 8000, "servicePort": 8000}
        ]
      }
    },
    "cpus": 0.2,
    "mem": 256.0,
    "instances": 1
}













Deployment Scripts





Once we have the Marathon deployment file to publish we can compile the deployment scripts that will remove the currently running application and redeploy it using the new image. 



There are better upgrade strategies out of the box but, I won’t discuss them here.



Let’s put it under /data/jenkins/workspace/springboot-jersey-swagger (which will be eventually mapped to Jenkins container’s below path so that it’s reachable in Jenkins) and don’t forget to grant it with execution permission.








$ sudo touch /data/jenkins/workspace/springboot-jersey-swagger/deploy-app-springboot-jersey-swagger.sh
$ sudo chmod +x /data/jenkins/workspace/springboot-jersey-swagger/deploy-app-springboot-jersey-swagger.sh
$ sudo vim /data/jenkins/workspace/springboot-jersey-swagger/deploy-app-springboot-jersey-swagger.sh














#!/bin/bash

# We can make it much more flexible if we expose some of below as parameters
APP="springboot-jersey-swagger"
MARATHON="192.168.56.118:8080"
PAYLOAD="${WORKSPACE}/marathon-app-springboot-jersey-swagger.json"
CONTENT_TYPE="Content-Type: application/json"

# Delete the old application
# Note: again, this can be enhanced for a better upgrade strategy
curl -X DELETE -H "${CONTENT_TYPE}" http://${MARATHON}/v2/apps/${APP}

# Wait for a while
sleep 2

# Post the application to Marathon
curl -X POST -H "${CONTENT_TYPE}" http://${MARATHON}/v2/apps -d@${PAYLOAD}









Plug It In By Adding One More Build Step In Jenkins



Simply add one more build step after the Maven step.



It's a "Execute Shell" build step and the commands are really simple:






cd ${WORKSPACE}
./deploy-app-springboot-jersey-swagger.sh







Save it.







Trigger the Process Again





By default, any code check-in will automatically trigger the build and deployment process.



But for testing purpose, we can click the “Build Now” to manually trigger it.



The Console Output is really verbose but it can help you understand the whole process. Check it out here.











Once the process is done in Jenkins (we can monitor the progress by viewing the Console Output), we can see our application exist in “Deployments” tab and quickly move into “Application” tab of Marathon which means that the deployment is done.



Note: if an application keeps pending in deployment because of “waiting for resource offer”, it’s most likely because of no sufficient resource (e.g. Mem) set in Mesos. In this case, we may need to add more Mesos Slave or fine tune the JSON deployment file with fewer resources.





























One may notice that our application is under “Unknown”. Why? It’s because we haven’t set up our health check mechanism yet so Marathon doesn't really know whether the application is healthy or not.









Verify Our App



Once application is deployed by Mesos + Marathon, there will be new Docker container(s) spun up automatically.



But how can we access our application deployed by Mesos + Marathon? The port configuration part in Marathon may be a bit confusing and we will figure it out later. In current stage we have to see what port our app is exposed, by below command:






$ docker ps
CONTAINER ID        IMAGE                                                         COMMAND                  CREATED             STATUS              PORTS                                          NAMES
a020940ae0b5        192.168.56.118:5000/devops/springboot-jersey-swagger:latest   "java -jar /springboo"   8 minutes ago       Up 8 minutes        0.0.0.0:31489->8000/tcp                        mesos-e51e55d3-cd64-4369-948f-aaa433f12c1e-S0.b0e0d351-4da0-4af5-be1b-5e7dab876d06
...











As what I highlighted in red, the port exposed to host is 31489 so our application URL will be:



http://{the docker host ip}:31489, or http://192.168.56.118:31489 in my case.













Yes, once you reach here, our application's continuous deployment is now part of our integrated CI + CD pipeline. We did it!







Conclusion



Now it's time to recap our DevOps story.



I update a bit on the diagram which may be better for me to illustrate the process and components we have engaged.











The Development As Usual



Firstly, as usual, we develop our RESTful webservices by following Microservices architecture with components like Java, Spring Boot, Jersey, Swagger, Maven and more.



Our source code is hosted by Gogs, which is our private Git Repository.



We develop, do unit test, check in code -- everything is just as usual.







Related posts:




The Continuous Integration (CI) Comes To Help



Jenkins, acted as our CI Server, comes to help which detects changes in our Git Repository and triggers our integrated CI + CD pipeline.



For CI part, Jenkins engages Maven to run through tests, both Unit Test and SIT if any and build our typical Spring Boot artifact, an all-in-one fat Jar file. Meanwhile, with the Docker Maven plugin, docker-maven-plugin, Maven can take one step further to help build our desired Docker image where our application Jar file is being hosted.



The Docker image will be published to our private Docker Registry and our CI part ends.







Related posts:




The Final Show in Continuous Deployment (CD)



Apache Mesos, together with Marathon and ZooKeeper founds the CD world.



The trigger point is still in Jenkins -- by adding one more build step -- once CI part is done, Jenkins then executes our Shell scripts where the interactions with Marathon are wrapped.



By posting a predefined JSON payload, or the deployment file, to Marathon, it will deploy our application to Mesos platform. As a result, Docker container(s) will be spun up automatically with specific resources (e.g. CPU, RAM) allocated for our application.



Accordingly, the web services offered by our application are up and running.



That's our whole story of "from code to online services".







Future Improvements?



Yes, of course. Anyway, it's still a DevOps prototype.



Along the way, for simplicity purposes, there are some topics or concerns that haven't been properly addressed yet which include but are not limited to:




    

  • Using multi-server cluster to make it "real";
    • 

  • Introducing security and ACL to bring in necessary control;
    • 

  • Applying HAProxy setup for the service clustering;
    • 

  • Enabling auto scaling;
    • 

  • Adding health check mechanism for our application into Marathon;
    • 

  • Fine-tuning the application upgrade strategy;
    • 

  • Engaging Consul for automatic service discovery;
    • 

  • Having a centralized monitoring mechanism;
    • 

  • Trying it out with AWS or Azure;
    • 

  • And more
    • 










Final Words



Continuous Delivery (= Continuous Integration + Continuous Deployment) is a big topic and has variable solutions in the market now, be it commercial or open source.



I started working on this set-up mainly to make my hands dirty and also help developers and administrators to learn how to use a series of open source components to build up a streamlined CI+CD pipeline so that we can sense the beauty of continuous delivery.



As mentioned in previous chapter of "Future Improvements", a lot more things are still pending for exploring. But that's also the reason why architects like me love it the way that IT rocks, isn't it?







Enjoy DevOps and appreciate your comments if any!


















Posted by



at





1 comment:
  






















Labels:
,
,
,
,
,

















          

        

          

        

Monday, 3 October 2016


          

        










Integrated Continuous Integration (CI) Solution With Jenkins, Maven, docker-maven-plugin, Private Docker Registry












In previous blog, I shared how to run Jenkins + Maven in Docker container here and we could already automate the Maven-based build automatically in our Jenkins CI server, which is great to kick off for the CI process.



But, while reviewing what we're going to achieve as below diagram, we realize that having a Spring Boot-powered jar file is not what we want, we want to have a Docker image built and registered into our private Docker Registry so that we're fully ready for the coming Continuous Deployment!









Integrated Jenkins + Maven + docker-maven-plugin



So let's focus on the highlighted part on above diagram to build an integrated Jenkins + Maven + docker-maven-plugin solution, by using Docker container technology also!





Build Up Our Private Docker Registry



Docker has provided a great Docker image for building up our private Docker Registry, here.



Let's do it by simply running through below commands:






$ sudo mkdir /data/registry
$ sudo chown -R devops:docker /data/registry
$ whoami
devops
$ pwd
/home/devops
$ vim docker-compose.yml
-------------------------------
# Private Docker Registry
docker-registry:
  image: registry
  volumes:
    - /data/registry:/var/lib/registry
  ports:
    - "5000:5000"
-------------------------------







It's almost done, let's spin it up:




$ docker-compose up





And try it out:




$ curl http://127.0.0.1:5000/


"\"docker-registry server\""





Well, our private Docker Registry works but wait…the Docker client can’t push image to it yet.

We need to make some changes at Docker's configuration before pushing images:




$ sudo vim /etc/default/docker
---------------------------
DOCKER_OPTS="--insecure-registry 192.168.56.118:5000"
---------------------------





And then restart the Docker daemon:




$ sudo service docker restart





Now our private Docker Registry is ready to rock.



But how to verify whether our Docker Registry is really working as what we expected?

It must be: 1) Ready for publishing; and 2) Ready for searching.





The idea is to use our greatest public available "hello-world" image which is lightweight enough to try, 960B!






$ docker pull hello-world


$ docker images


...
hello-world    latest                     690ed74de00f        11 months ago       960 B




$ docker tag 690ed74de00f 192.168.56.118:5000/hello-world


$ docker images


...
192.168.56.118:5000/hello-world    latest  690ed74de00f        11 months ago       960 B
hello-world                        latest  690ed74de00f        11 months ago       960 B




$ docker push 192.168.56.118:5000/hello-world
The push refers to a repository [192.168.56.118:5000/hello-world]
5f70bf18a086: Image successfully pushed
b652ec3a27e7: Image succewr12334567ssfully pushed
Pushing tag for rev [690ed74de00f] on {http://192.168.56.118:5000/v1/repositories/hello-world/tags/latest}




$ curl http://192.168.56.118:5000/v1/search?q=hello
{"num_results": 1, "query": "hello", "results": 
 [
  {"description": "", "name": "library/hello-world"}
 ]
}





Sure enough, our private Docker Registry is fully ready!



Okay, so far we already have a private Docker Registry, already can do a basic Maven build in Jenkins. What should we do next?





Enable the docker-maven-plugin

It's time to enable our glue, the docker-maven-plugin, in our project.

The purposes of using this plugin are to:




    

  1. Automate the Docker image build process on top of our typical build which the output might be jar, war. Now, we want to have a Docker image equipped with our application as the deliverable!
    2. 

  2. Publish our built Docker image to our private Docker Registry where is the starting point of our Continuous Deployment!
    3. 




<Frankly speaking, there were many issues encountered along my experiment like here but to make long story short, I'll go directly to the points>





Firstly, please refer to my previous blog "Some Practices to Run Maven + Jenkins In Docker Container" here for the basic setup of the environment.

Below successful experiment would be on top of this environment.





Explicitly Specify Docker Host in /etc/default/docker



Edit the /etc/default/docker as below:






#DOCKER_OPTS="--insecure-registry 192.168.56.118:5000"
DOCKER_OPTS="--insecure-registry 192.168.56.118:5000 -H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock"







And restart the Docker daemon is required, of course:




$ sudo service docker restart







Enable the docker-maven-plugin In pom.xml

There are many Docker Maven plugins available and I simply choose spotify's one, here.

The configuration of the plugin in pom.xml is as below:




<!-- Docker image building plugin -->
<plugin>
 <groupId>com.spotify</groupId>
 <artifactId>docker-maven-plugin</artifactId>
 <version>0.4.13</version>
 <executions>
  <execution>
   <phase>package</phase>
   <goals>
    <goal>build</goal>
   </goals>
  </execution>
 </executions>
 <configuration>
  <dockerHost>${docker.host}</dockerHost>
  
  <baseImage>frolvlad/alpine-oraclejdk8:slim</baseImage>
  <imageName>${docker.registry}/devops/${project.artifactId}</imageName>
  <entryPoint>["java", "-jar", "/${project.build.finalName}.jar"]</entryPoint>
  
  <resources>
     <resource>
    <targetPath>/</targetPath>
    <directory>${project.build.directory}</directory>
    <include>${project.build.finalName}.jar</include>
     </resource>
  </resources>
  
  <forceTags>true</forceTags>
  <imageTags>
     <imageTag>${project.version}</imageTag>
  </imageTags>
 </configuration>
</plugin>



Note: For complete pom.xml file, please refer to my GitHub repository, here.





The Jenkins Job

Below screenshots are the major configuration parts of the Jenkins Job.



Source Code Management









Build Triggers









Maven Goals









Console Output

Here I also posted a typical Console Output of the Jenkins Job for reference as a Gist here.





Have A Check On What We Have



Firstly we can check whether our local repository has the cached image:






$ docker images
REPOSITORY                                             TAG                        IMAGE ID            CREATED             SIZE


...
192.168.56.118:5000/devops/springboot-jersey-swagger   1.0.0-SNAPSHOT             203fde08df84        44 minutes ago      191.9 MB


...











Great, then have a check whether this image has been registered into our private Docker Registry:






$ curl http://192.168.56.118:5000/v1/search?q=springboot-jersey-swagger
{"num_results": 1, "query": "springboot-jersey-swagger", "results": 
 [
  {"description": "", "name": "devops/springboot-jersey-swagger"}
 ]
}











Yeah! We made it!









Try Our Docker Image Out Also





To go deeper, we can also take a look on whether our application works or not:






docker run -d -p 8000:8000 192.168.56.118:5000/devops/springboot-jersey-swagger











Open the browser and key in: http://192.168.56.118:8000



















Hooray, it works perfectly!







Conclusion





From this experiment we have learned that:




    

  1. By using a customized Docker container, we can put Jenkins, Maven and docker-maven-plugin together to automate our project build process and the deliverable can be our desired Docker image;
    2. 

  2. The Docker image can be pushed automatically to our private Docker Registry within the pipeline which will enable us to make the Continuous Deployment happen soon.
    3. 










So the CI part is done and let's look forward to our CD part soon!



Enjoy your CI experiments also!
















Posted by



at





8 comments:
  






















Labels:
,
,
,
,
,
,
,

















        

      









Subscribe to:
Posts (Atom)