Monday, 20 June 2016

From Code To Online Services: My experiments of DevOps - Development of RESTful Web Services by Spring Boot, Jersey, Swagger


1. The Stack of Frameworks / Specs


Well, even any application can be built and used for this experiment, but why not build an interesting one: a real world RESTful service by saying hello to someone? Alright, that’s just another statement for a hello world application;)

Here we go:

  • Sprint Boot: The fundamental framework for the services.
  • Jersey: The RI of JAX-RS specs.
  • Swagger: The API documentation specs and tooling.
  • For testing, we’re going to use 
    • Spring Boot Test Framework (spring-boot-starter-test) with Junit, Hamcrest
    • Rest Assured

2. Git Server: Gogs


Even we can setup Git Server later, but as version control is important, please do it as early as possible.
Note: I did try GitLab but: 1) it’s too slow running in my laptop; and 2) I was facing several permission issues being thrown from embedded Nginx…so I changed my mind and had Gogs instead as it’s really “a painless self-hosted Git service” as advertised.

2.1 Spin Up the Docker Container

Login to the Ubuntu VM, our logical host and execute below commands:

1
2
3
4
5
$ cd ~
$ mkdir cd_demo
$ cd cd_demo
$ touch docker-compose.yml
$ vim docker-compose.yml

1
2
3
4
5
6
7
Gogs:
  image: 'gogs/gogs:latest'
  ports:
    - '3000:3000'
    - '1022:22'
  volumes:
    - '/data/gogs:/data'

Note: it’s a good practice to mount a local path (here is /data) for the container so all valuable data like settings, database will not be gone even we upgrade our container. 

$ docker-compose up

Now, we have spun up our first Docker container for Gogs, our Git server.

2.2 Configuration

In just a few seconds, the Gogs container should have been up and running and we can access the app by: http://192.168.56.118:3000

It would guide us for the “Install Steps For First-time Run”.
Below were my inputs, for your reference:
  • Database Settings
    • Database Type: SQLite3
    • Path: /data/db/gogs.db
  • Application General Settings
    • Application Name: Gogs: Go Git Service
    • Repository Root Path: /data/git/gogs-repositories
    • Run User: git
    • Domain: 192.168.56.118
    • SSH Port: 22
    • HTTP Port: 3000
    • Application URL: http://192.168.56.118:3000/
    • Log Path: data/log
  • Optional Settings
    • Admin Account Settings: omitted
Once done, Gogs will persistent the settings and redirect to the home page.

2.3 Create Our First Repository

All these can be done via Gogs UIs:
Gogs - New Repository
Gogs - New Repository for Spring Boot, Jersey, Swagger










































2.4 Clone the Git Repository to Kick Off Development

By command line, it’s very simple and straightforward.
Open Git Bash and key in:

$ git clone http://192.168.56.118:3000/bright/springboot-jersey-swagger.git

Or we can do it by using Eclipse if you prefer to.

3. Create Maven-based Project Structure

Since development part is my strongest part and code says more than words, let’s make it short but clear during the experiment.
Apparently, as we follow Maven’s best practices to build the project so the folder structure is exactly Maven-driven:
---------------------------------------------
/
   LICENSE
   pom.xml
   README.md
├───src
   ├───main
      ├───java
         └───bright
             └───zheng
                 └───poc
                     └───api
                            ApiApplication.java
                         ├───config
                                JerseyConfig.java
                         ├───model
                                Hello.java
                         └───resources
                                 HelloResource.java
      └───resources
             application.yml
             bootstrap.yml
             log4j.xml
          ├───static
   └───test
       └───java
           └───bright
               └───zheng
                   └───poc
                       └───api
                               ApplicationTests.java
---------------------------------------------

Meanwhile, I’ll highlight some important files with necessary comments so that we can easily understand how to develop a RESTful application by using abovementioned frameworks.

3.1 pom.xml

/pom.xml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
       <modelVersion>4.0.0</modelVersion>

       <parent>
              <groupId>org.springframework.boot</groupId>
              <artifactId>spring-boot-starter-parent</artifactId>
              <version>1.3.5.RELEASE</version>
              <relativePath />
       </parent>

       <groupId>bright.zheng.poc.api</groupId>
       <artifactId>springboot-jersey-swagger</artifactId>
       <version>1.0.0-SNAPSHOT</version>
       <packaging>jar</packaging>

       <name>springboot-jersey-swagger-docker</name>
       <description>POC - Restful API by Spring Boot, Jersey, Swagger</description>


       <properties>
              <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
              <java.version>1.8</java.version>
             
              <!-- project related properties -->
              <start-class>bright.zheng.poc.api.ApiApplication</start-class>
              <swagger-jersey2-jaxrs>1.5.3</swagger-jersey2-jaxrs>
              <rest-assured>3.0.0</rest-assured>
       </properties>

       <dependencies>
              <dependency>
                     <groupId>org.springframework.boot</groupId>
                     <artifactId>spring-boot-starter-actuator</artifactId>
              </dependency>
              <dependency>
                     <groupId>org.springframework.boot</groupId>
                     <artifactId>spring-boot-starter-jersey</artifactId>
              </dependency>
              <dependency>
                     <groupId>org.springframework.boot</groupId>
                     <artifactId>spring-boot-starter-web</artifactId>
              </dependency>
              <dependency>
                     <groupId>io.swagger</groupId>
                     <artifactId>swagger-jersey2-jaxrs</artifactId>
                     <version>${swagger-jersey2-jaxrs}</version>
              </dependency>

              <!-- testing related dependencies -->
              <dependency>
                     <groupId>org.springframework.boot</groupId>
                     <artifactId>spring-boot-starter-test</artifactId>
                     <scope>test</scope>
              </dependency>
              <dependency>
                     <groupId>io.rest-assured</groupId>
                     <artifactId>rest-assured</artifactId>
                     <version>${rest-assured}</version>
                     <scope>test</scope>
              </dependency>
       </dependencies>

       <build>
              <plugins>
                     <plugin>
                           <groupId>org.springframework.boot</groupId>
                           <artifactId>spring-boot-maven-plugin</artifactId>
                     </plugin>
              </plugins>
       </build>
</project>

3.2 ApiApplication.java


\src\main\java\bright\zheng\poc\api\ApiApplication.java
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
package bright.zheng.poc.api;


import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.boot.context.web.SpringBootServletInitializer;

/**
 * This is a typical boostrap class for Spring Boot based application
 * 
 * @author bright.zheng
 *
 */
@SpringBootApplication
public class ApiApplication
extends SpringBootServletInitializer {  
       @Override
       protected SpringApplicationBuilder configure(SpringApplicationBuilder builder) {
            return builder.sources(ApiApplication.class);
       }

       public static void main(String[] args) {
              SpringApplication.run(ApiApplication.class, args);
       }
}

3.3 HelloResource.java

\src\main\java\bright\zheng\poc\api\resources\HelloResource.java
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
package bright.zheng.poc.api.resources;

import javax.ws.rs.Consumes;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.Response.Status;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;

import bright.zheng.poc.api.model.Hello;
import io.swagger.annotations.Api;
import io.swagger.annotations.ApiOperation;
import io.swagger.annotations.ApiParam;
import io.swagger.annotations.ApiResponse;
import io.swagger.annotations.ApiResponses;

@Component
@Path("/v1/hello")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.APPLICATION_JSON)
@Api(value = "Hello API - say hello to the world", produces = "application/json")
public class HelloResource {

       private static final Logger LOGGER = LoggerFactory.getLogger(HelloResource.class);

       @GET                                     //JAX-RS Annotation
       @Path("/{name}")     //JAX-RS Annotation
       @ApiOperation(                           //Swagger Annotation
                     value = "Say hello by providing the name by URI, via standard json header",
                     response = Hello.class) 
       @ApiResponses(value = {           //Swagger Annotation
              @ApiResponse(code = 200, message = "Resource found"),
           @ApiResponse(code = 404, message = "Resource not found")
       })
       public Response sayHelloByGet(@ApiParam @PathParam("name") String name) {
              LOGGER.info("v1/hello/{} - {}", name, MediaType.APPLICATION_JSON);
              return this.constructHelloResponse(name, MediaType.APPLICATION_JSON);
       }
      
       ///////////////////////////////////////////////////////////////

       private Response constructHelloResponse(String name, String via) {
              //for testing how we handle 404
              if ("404".equals(name)) {
                     return Response.status(Status.NOT_FOUND).build();
              }
              Hello result = new Hello();
              result.setMsg(String.format("Hello %s - %s", name, via));
              return Response.status(Status.OK).entity(result).build();
       }
}



3.4 JerseyConfig.java

\src\main\java\bright\zheng\poc\api\config\JerseyConfig.java
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
package bright.zheng.poc.api.config;

import javax.annotation.PostConstruct;

import org.glassfish.jersey.server.ResourceConfig;
import org.glassfish.jersey.server.wadl.internal.WadlResource;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;

import bright.zheng.poc.api.resources.HelloResource;
import io.swagger.jaxrs.config.BeanConfig;
import io.swagger.jaxrs.listing.ApiListingResource;
import io.swagger.jaxrs.listing.SwaggerSerializers;

/**
 * Jersey Configuration
 *
 * <p>
 *
 * The endpoint will expose not only the real APIs,
 * but also two important documents:
 * <li> - swagger spec: /swagger.json</li>
 * <li> - WADL spec: /application.wadl</li>
 *
 * </p>
 *
 * @author bright.zheng
 *
 */
@Component
public class JerseyConfig extends ResourceConfig {

       @Value("${spring.jersey.application-path:/api}")
       private String apiPath;

       public JerseyConfig() {
              this.registerEndpoints();
       }
      
       @PostConstruct
       public void init() {
              this.configureSwagger();
       }

       private void registerEndpoints() {
              this.register(HelloResource.class);
              this.register(WadlResource.class);
       }

       private void configureSwagger() {
              this.register(ApiListingResource.class);
              this.register(SwaggerSerializers.class);

              BeanConfig config = new BeanConfig();
              config.setTitle("POC - Restful API by Spring Boot, Jersey, Swagger");
              config.setVersion("v1");
              config.setContact("Bright Zheng");
              config.setSchemes(new String[] { "http", "https" });
              config.setBasePath(this.apiPath);
              config.setResourcePackage("bright.zheng.poc.api.resources");
              config.setPrettyPrint(true);
              config.setScan(true);
       }

}


4. Swagger UI Integration

It would be great to have Swagger UI integration for automatic API documentation.
Simply do below steps:


5. Testing

Programmers write unit tests, of course. Here we go:

5.1 ApplicationTests.java

\src\test\java\bright\zheng\poc\api\ApplicationTests.java
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
package bright.zheng.poc.api;

import static io.restassured.RestAssured.expect;
import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.equalTo;

import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.test.SpringApplicationConfiguration;
import org.springframework.boot.test.WebIntegrationTest;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

import io.restassured.RestAssured;

/**
 * Application level testing
 *
 * @author bright.zheng
 *
 */
@RunWith(SpringJUnit4ClassRunner.class)
@SpringApplicationConfiguration(classes = ApiApplication.class)
@WebIntegrationTest({"server.port=0"})
public class ApplicationTests {
      
    @Value("${local.server.port}")
    private int port;
      
    @Before
    public void setupURL() {
        RestAssured.baseURI = "http://localhost";
        RestAssured.port = port;
        RestAssured.basePath = "/";
    }
 
    /**
     *
     * {"app":{"name":"springboot-jersey-swagger"},"build":{"version":"1.0.0-SNAPSHOT"}}
     *
     * @throws Exception
     */
    @Test
    public void springroot_info() throws Exception {
       expect().
              body("app.name", equalTo("springboot-jersey-swagger")).
              body("build.version", equalTo("1.0.0-SNAPSHOT")).
           when().
              get("/info");
    }
   
    /**
     * {"msg":"Hello Bright - application/json"}
     *
     * @throws Exception
     */
    @Test
    public void hello_get() {
       expect().
              body("msg", containsString("Hello Bright")).
        when().
              get("/api/v1/hello/Bright");
    }

}



6. Run It


6.1 Start It Up


By Maven command line in Windows, it would be:
1
2
3
> cd /d D:\development\workspaces\java\POC\springboot-jersey-swagger
> mvn package
> java -jar target/springboot-jersey-swagger-1.0.0-SNAPSHOT.jar



By Eclipse, simply open ApiApplication.java and click Run menu -> Run As -> Java Application, it would be booted up within seconds.

6.2 Take a Look at the API Documentation UI provided by Swagger

Simply open browser and key in: http://localhost:8000/





























We can even try the services. Simply key in the parameter (name here) and click the “Try it out!” button, we can see everything in the same page including: 

    • Curl command if we want to reproduce it on Curl
    • URL we just requested
    • Response Code
    • Response Body
    • Response Headers

    7. Commit and Push to Git


    Commit and push our first draft of Hello Service to our local Git Server.

    By Git Bash:
    1
    2
    3
    4
    $ git status
    $ git add –A
    $ git commit –m “first draft of Hello Service”
    $ git push
    


    Note: I also synced the code to GitHub, follow below steps:
    • Create a GitHub repository: itstarting/springboot-jersey-swagger
    • Execute commands like:
    1
    2
    3
    4
    5
    6
    7
    8
    $ cd /D/development/workspaces/java/POC/springboot-jersey-swagger
    $ git remote add github https://github.com/brightzheng100/springboot-jersey-swagger.git
    $ git remote -v
    github  https://github.com/brightzheng100/springboot-jersey-swagger.git (fetch)
    github  https://github.com/brightzheng100/springboot-jersey-swagger.git (push)
    origin  http://192.168.56.118:3000/bright/springboot-jersey-swagger.git (fetch)
    origin  http://192.168.56.118:3000/bright/springboot-jersey-swagger.git (push) 
    $ git push -u github master
    

    By Eclipse it's very simple: 
    • Right click project -> team -> Commit…
    • Choose all items in “Unstaged Changes”, right click -> Add to Index, which will add all items into “Staged Changes”
    • Key in comments in “Commit Message” and click “Commit and Push” button and eventually we have below result.


    Well, the development works are done!



    For complete source code, please visit my GitHub. Enjoy coding!


    From Code To Online Services: My experiments of DevOps - Environment

    For Development

    Well, I have a laptop on hand with Windows 7 64bit which is my development machine.
    On top of it, we need to setup some basic development related tools of choice.

    Below are mine:

    • Eclipse Java EE IDE for Web Developers: Mars.2 Release (4.5.2) with necessary plugins like m2e for Maven, Egit for Git
    • Java 8 latest
    • Apache Maven: 3.3.x, no private repository yet at this moment
    • Git: 1.9.5


    For Continuous Integration (CI) & Continuous Deployment (CD)

    OS: Ubuntu

    As I’m going to play around with Linux, I setup VirtualBox on my laptop and span up one Ubuntu 14.04 as the logical host for everything as I’m going to run them on top of Docker.
    Here is the detailed Ubuntu version:
    $ lsb_release -a
    No LSB modules are available.
    Distributor ID: Ubuntu
    Description:    Ubuntu 14.04.2 LTS
    Release:        14.04
    Codename:       trusty

    For convenient purpose, I used host-only option and had one static IP for the Ubuntu:
    $ ifconfig eth1 | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1}'
    192.168.56.118

    Note: Please refer to VirtualBox for details on how to set up virtual boxes, or let me know if you need any assistance or sharing on this. 

    Docker & Docker Compose

    Please refer to https://docs.docker.com/engine/installation/linux/ubuntulinux/ for how to install Docker as I won’t mention it here.

    Here is the Docker latest version as of writing:
    $ docker --version
    Docker version 1.11.0, build 4dc5990
    $ docker-compose --version
    docker-compose version 1.6.2, build 4d72027



    Wednesday, 15 June 2016

    From Code To Online Services: My experiments of DevOps - Overview

    While thinking of why Internet companies are using open source products even for building mission-critical business platforms, it turns out that they have no choice: it's too big in terms of scale to get COTS commercial products in even sometimes having commercial products can buy some time for service provisioning.

    So they build, they open source, they gain contributions from the community while sharing and eventually they shine. Well, that's script that the success stories happen...and keep happening.

    Traditional enterprise solutioning and architecting are not that sexy, sometimes, as they're closed and have to mark "confidential" everywhere. And the adoption process may be very long and may not happen for some areas for ever for some reason.
    Yes, we also can't survive without open source components as it's ubiquitous. But if the decision of purchasing commercial products is easier to make, sometimes the management may still go that way as consideration of risks (well, whatever they're) will always get into the mind first.

    Now the goal for me is very straightforward: keep my hands dirty by exploring a "dream" workflow of DevOps even we can't fully utilize all the components by a daily basis.

    I call it "from code to online services".
    I know it's not new for some people and yes, I have to admit that it's a big topic.

    But I'm going to walk it through with some topics like:
    • My experiments of DevOps - Environment (see post here)
    • My experiments of DevOps - Development of RESTful Web Services by Spring Boot, Jersey, Swagger (see post here)
    • My experiments of DevOps - Continuous Integration 
      • Run Maven + Jenkins In Docker Container (see post here)
      • Integrated CI Solution With Jenkins + Maven + docker-maven-plugin + Private Docker Registry, in Docker Containers! (see post here)
    • My experiments of DevOps - Continuous Deployment (see post here)

    And eventually the idea may look like this:



    Okay, let's do it!... and appreciate any comments.


    EDIT: As of today, 21 Oct 2016, this series of experiments has been completed. I may try more options for the DevOps exercises to gain more practices for sharing...