How to build a Microservice with Go, PHP/Laravel and Kafka

Microservices have recently emerged as the main architectural framework for managing cloud applications and solutions due to their ability to handle the scalability challenge.

This article will explain the notion of microservices and show you how to build one in several languages such as Go and PHP/Laravel that connect with one other over a message broker such as Kafka.

We are going to cover the following:

• What is Microservice architecture
• What is Monolithic architecture
• Comparison between microservice and monolithic
• Communication between services
• Monitoring microservices
• Setting up Project Prerequisites

To follow along, you must have a basic understanding of the following:

• PHP/Laravel
• Golang, Echo web framework
• Docker

What is a Microservice Architecture ?

A microservice architecture, often known as microservices, is a framework for decoupling a single application into a number of smaller apps or services, with each service responsible for its own domain action. The purpose of this design is for a service to be able to interface with other services simply using an API (Application Programming Interface) while also being able to be developed and deployed separately. Microservice architecture is most commonly used in complicated and massive systems like Netflix, Google, Apple, Uber, and others.

What is a Monolithic Architecture?

A monolithic architecture is one in which the frontend, backend, and database system are all developed as a single application with a single huge codebase. Most startups and organizations continue to adopt monolithic architecture because it is simple to set up and manage for small-scale applications.

Comparison between Microservice and Monolithic architecture

Monolith Architecture

Pros

• Simple to set up and deploy
• Low setup and deployment costs
• Security administration is simplified.
• Can be readily maintained by a small team

Cons

• It might grow complicated and difficult to maintain over time.
• Scaling may be an issue.
• A minor failure or defect can cause the entire system to fail.

Microservice Architecture

Pros

• It is simple to scale.
• Each service can be created in a unique language.
• A service can be independently built, deployed, and scaled.
• Downtime is decreased because a service failure does not affect the entire system.

Cons

• It is costly to manage.
• It necessitates specific knowledge, particularly in the field of distributed systems.
• Security might be difficult to manage.

Communication between Microservices

Having a collection of services in a system necessitates a unified approach to communication. The major tradeoffs between these communication patterns are as follows.

Synchronous: This is a concept that interacts with other services via HTTP(Hypertext Transfer Protocol), RPC(Remote Procedure Call), and gRPC(Remote Procedure Call). This technique sends a request and waits for a response from the receiver.

Asynchronous: Another communications pattern in which a message is sent without waiting for a response; the message is processed asynchronously by the receiver. Because the sender does not have to wait for a response, this pattern is quick and efficient. Message brokers, AMQP (Advanced Messaging Queuing Protocol), and other protocols are presented as examples.

Monitoring Microservices

Monitoring of services is a vital component of implementing this architecture to ensure that everything works as planned and to prevent failures. A variety of things can be monitored, but the most important ones are:

• Service metrics: service health check, communication latency between other services, etc.
• Key business metrics: daily active users, retention, revenue, etc.
• Host Level metrics: CPU, Memory, Disk I/O, etc
• Aggregate level metrics: performance of database tier, cache tier, etc.

Monitoring Tools (APM): Monitoring tools suitable for microservice architecture include Jaeger Tracing, Newrelic, Middleware, and others.

Setting up project

We will create a simple web page that will interface with other services for our project. The following is a breakdown of the services that will be developed and the technology that will be used to implement them.

Services Structure

Communication protocol:

Synchronous: REST API (HTTP) Asynchronous: Kafka
Data Store: MongoDB Deployment: Docker

Docker Setup

All of our services, including Nginx, PHP, MongoDB, and Kafka, are included in the project's docker-compose file. In addition, a docker file is built for each microservice to micro-manage its build and dependencies.

    version:  '3'
    services:
      frontend-service:
        build: 
          context: ./frontend-service
          dockerfile: frontend-service.dockerfile
        ports:
          - "80:8080"
      listener-service:
        build: 
          context: ./listener-service
          dockerfile: listener-service.dockerfile
        restart: unless-stopped
        environment:
          kafkaURL: kafka:9092
          topic: logger
          groupID: logger-group
      logger-service:
        build: 
          context: ./logger-service
          dockerfile: logger-service.dockerfile
        ports:
          - "3500:3500"
        environment:
          mongoURL: mongodb://mongo:27017
          dbName: demo_app
          collectionName: logger_db
      broker-service:
        build: 
          context: ./broker-service
          dockerfile: broker-service.dockerfile
        ports:
          - "8083:1323"
        environment:
          kafkaURL: kafka:9092
          topic: logger
      php:
        container_name: php
        build:
          context: ./auth-service
          dockerfile: auth-service.dockerfile
        env_file:
          - auth-service/auth.env
        volumes:
          - ./auth-service:/var/www
        ports:
          - "9000:9000"
      auth-service:
        image: nginx:alpine
        restart: unless-stopped
        ports:
          - "8082:80"
        volumes:
          - ./auth-service:/var/www
          - ./auth-service/nginx/conf.d/default.conf:/etc/nginx/conf.d/default.conf
        depends_on:
          - php
      zookeeper:
        image: wurstmeister/zookeeper
        ports:
          - "2181:2181"
      kafka:
        image: wurstmeister/kafka
        ports:
          - "9092:29092"
        environment:
          KAFKA_ADVERTISED_HOST_NAME: kafka
          KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
          KAFKA_CONNECT_BOOTSTRAP_SERVERS: localhost:9092
          KAFKA_CONNECT_REST_PORT: 8082
          KAFKA_CONNECT_REST_ADVERTISED_HOST_NAME: "localhost"
          KAFKA_CONNECT_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
          KAFKA_CONNECT_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
          KAFKA_CONNECT_KEY_CONVERTER_SCHEMAS_ENABLE: 0
          KAFKA_CONNECT_VALUE_CONVERTER_SCHEMAS_ENABLE: 0
          KAFKA_CONNECT_INTERNAL_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
          KAFKA_CONNECT_INTERNAL_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
          KAFKA_CONNECT_INTERNAL_KEY_CONVERTER_SCHEMAS_ENABLE: 0
          KAFKA_CONNECT_INTERNAL_VALUE_CONVERTER_SCHEMAS_ENABLE: 0
          KAFKA_CREATE_TOPICS: "logger:1:0"
        depends_on:
          - zookeeper
      mongo:
        image: 'mongo:4.2.16-bionic'
        ports:
          - "27017:27017"
        environment:
          MONGO_INITDB_DATABASE: logs
          MONGO_INITDB_ROOT_USERNAME: admin
          MONGO_INITDB_ROOT_PASSWORD: password
        volumes:
          - ./db-data1/mongo/:/data/db

Here is our Makefile (supported for Linux and Mac users), which is used to bootstrap our services and also aids in the creation of binary files for Go-based services.

    LOGGER_APP=loggerApp
    BROKER_APP=brokerApp
    LISTENER_APP=listenerApp
    ## up: starts all containers in the background without forcing build
    up:
        @echo "Starting Docker images..."
        docker-compose up -d
        @echo "Docker images started!"
    ## up_build: stops docker-compose (if running), builds all projects and starts docker compose
    up_build: build_logger build_broker build_listener
        @echo "Stopping docker images (if running...)"
        docker-compose down
        @echo "Building (when required) and starting docker images..."
        docker-compose up --build -d
        @echo "Docker images built and started!"
    ## down: stop docker compose
    down:
        @echo "Stopping docker compose..."
        docker-compose down
        @echo "Done!"
    ## build_broker: builds the broker binary as a linux executable
    build_logger:
        @echo "Building logger binary..."
        cd ./logger-service && env GOOS=linux CGO_ENABLED=0 go build -o ${LOGGER_APP} ./cmd/api
        @echo "Done!"
    build_broker:
        @echo "Building broker binary..."
        cd ./broker-service && env GOOS=linux CGO_ENABLED=0 go build -o ${BROKER_APP} ./cmd/api
        @echo "Done!"
    build_listener:
        @echo "Building listener binary..."
        cd ./listener-service && env GOOS=linux CGO_ENABLED=0 go build -o ${LISTENER_APP} ./cmd/api
        @echo "Done!"

Frontend (UI): The frontend is a simple page that serves as an entrance point for users to engage with our application. It will be built with Vuejs and Bootstrap UI.

The Dockerfile for the Vue configuration is available here.

FROM node:lts-alpine
# install simple http server for serving static content
RUN yarn global add http-server
RUN mkdir app/
# make the 'app' folder the current working directory
COPY . ./app
WORKDIR /app
# install project dependencies
RUN yarn install
RUN yarn build
# RUN ls
EXPOSE 8080
CMD [ "http-server", "dist/" ]

Our user interface (UI) is simple, with three (3) buttons (test auth, test log, and get all logs) and columns for payload and output.

    <template>
      <div class="container">
        <div class="py-3">
          <h3 class="title">Microservice Demo <span class="fs-6 fw-normal">(Built with Vue, Go, Laravel, Kafka and MongoDB)</span></h3>
          <div>
            <div class="row align-items-start my-2">
              <div class="col">
                <h4 class="font-bold">Payload</h4>
                <div class="py-4 px-4 payload">
                  <div>
                    <h6>Login</h6>
                    Valid details: <span class="badge rounded-pill text-bg-secondary">email: admin@gmail.com, password: verify</span> <span class="badge rounded-pill text-bg-secondary">email: user@gmail.com, password: user</span>
                    <div class="mb-3">
                      <input type="email" v-model="auth.email" class="form-control" placeholder="name@example.com">
                    </div>
                     <div class="mb-3">
                      <input type="password" v-model="auth.password" class="form-control" placeholder="password">
                    </div>
                  </div>
                  <div>
                    <div class="mb-3">
                      <label for="log" class="form-label">Enter Log sample</label>
                      <textarea class="form-control" id="log" v-model="log" rows="3">This is a log sample from frontend</textarea>
                    </div>
                  </div>
                </div>
              </div>
              <div class="col">
                <h4 class="font-bold">Output</h4>
                   <div class="w-full border overflow-auto break-words output-h">
                    <pre class="">
                    {{response}}
                    </pre>
                </div>
              </div>
            </div>
            <div class="mt-10">
              <button class=" p-3 btn btn-success rounded-md text-white" @click="handleAuth">Test Auth</button>
              <button class="mx-1 p-3 btn btn-primary rounded-md  text-white" @click="handleLog">Test Logger</button>
              <button class="p-3 btn btn-dark rounded-md text-white" @click="handleGetLogs">Get Logs</button>
            </div>
          </div>
        </div>
      </div>
    </template>
    <style scoped>
    .title {
      color:# 666;
      border-bottom: 3px solid# DDD;
    }
    .payload {
      background:# EEE;
      height:500px;
    }
    .output-h {
      height:500px;
    }
    </style>

Each button has its own event, which calls the broker service with its own payload. According to the docker-compose settings, our broker service is accessible at http://localhost:8083.

    <script>
     import axios from 'axios'
     import { ref, reactive } from 'vue'
    export default{
      setup() {
        const response = ref()
        const log = ref('')
        const auth = reactive({
          email: '',
          password: '',
        })
        const handleAuth = async () => {
          if(auth.email == '' || auth.password == '') {
            return alert("kindly enter email and password ")
          }
          log.value = ''
          const payload = {
            action: "auth",
            auth: {
              email: auth.email,
              password: auth.password
            }
          }
       
          try {
            const res = await axios.post('http://localhost:8083', payload)
            response.value = res.data
            auth.email = ''
            auth.password = ''
          }catch(error) {
            response.value = error
          }
        }
        const handleLog = async () => {
          if(log.value == '') {
            return alert('Kindly enter a log')
          }
          auth.email = ''
          auth.password = ''
          const payload = {
            action: "log",
            log: {
              name: "log",
              data: log.value,
            }
          }
          try {
            const res = await axios.post('http://localhost:8083', payload)
            response.value = res.data
            log.value = ''
          } catch(error) {
             response.value = error
          }
        }
        const handleGetLogs = async () => {
          const payload = {
            action: "logs",
            logs: {
            }
          }
          try {
            const res = await axios.post('http://localhost:8083', payload)
            response.value = res.data
          }catch(error) {
            response.value = error
          }
        }
        return {
          log,
          auth,
          response,
          handleAuth,
          handleLog,
          handleGetLogs,
        }
      },
    }
    </script>

Broker Service: This is an API gateway that stands between user requests on the frontend and our services, acting as a single entry point for all requests from the frontend and responses from our services.

It will be written in Golang.

    FROM alpine:latest
    RUN mkdir /app
    COPY brokerApp /app
    CMD ["/app/brokerApp"]

Logger Service: The Logger Service logs data to MongoDB. It will be written in Golang.

    FROM alpine:latest
    RUN mkdir /app
    COPY loggerApp /app
    CMD ["/app/loggerApp"]

Auth Service: This is a service that will be created in PHP/Laravel to handle simple user authentication. To begin, we must consider a web server, which in this case is Nginx, PHP, and a PHP extension for Kafka.

    FROM php:8.1.1-fpm-alpine
    RUN apk add shadow && usermod -u 1000 www-data && groupmod -g 1000 www-data
    RUN set -eux; apk add libzip-dev; docker-php-ext-install zip
    RUN apk add --no-cache --update --virtual buildDeps autoconf
    
    RUN docker-php-ext-install pcntl
    RUN docker-php-ext-configure pcntl --enable-pcntl
    
    ADD https://github.com/mlocati/docker-php-extension-installer/releases/latest/download/install-php-extensions /usr/local/bin/
    RUN chmod +x /usr/local/bin/install-php-extensions && \
        install-php-extensions rdkafka
    
    RUN apk update \
        && php -r "copy('https://getcomposer.org/installer', '/tmp/composer-setup.php');" \
        && php /tmp/composer-setup.php --no-ansi --install-dir=/usr/local/bin --filename=composer \
        && rm -rf /tmp/composer-setup.php
    
    WORKDIR /var/www
    COPY . .
    EXPOSE 9000
    CMD ["php-fpm"]

For our authentication service, we establish an API route to handle our login, as well as a health check endpoint in case we need to verify the service status.

When a user signs in, the AuthController performs the login action in an invokable method that also publishes a topic(logger) or an event to a listener, and the listener then calls the logger service through REST API.

    Route::post('/', AuthController::class);
    
    Route::get('/health-check', function() {
        return response()->json(['status' => 'OK']);
    });

Listener Service: This is a service that listens to and processes the queue events triggered by other services. It will be implemented in Golang

    FROM alpine:latest
     RUN mkdir /app
    COPY listenerApp /app
    CMD ["/app/listenerApp"]

Demo time

The final outcome of our project may be found here: https://github.com/iamhabbeboy/microservice-app. Make sure docker is installed before running make up_build .

Conclusion

In conclusion, there's a reason why microservice architecture is so popular and used by large corporations. I'd say it comes with a lot of benefits at a high cost, and if not managed properly, it could cause a disaster, as it's difficult to figure out when working with about 800 or more services calling each other all the time, as Netflix does. At this stage, with so much data traveling around, the monitoring visualization tool may become overwhelming.

As a result, before choosing microservice design, make sure you exhaust all other possibilities.

Thanks for reading ✌️