How we build our APIs: From Scratch

In a previous post we detailed how we build our APIs here at Polar Signals. But we're going back to basics, in this tutorial we'll teach you how to build a small App from API to implementation.

Thor Hansen
February 22, 2022
Go
Frontend
API
Tutorial
Buf
Proto
Protobuf
Golang

In my previous blog post I talked about how we build our APIs here at Polar Signals. It was a fairly high-level overview of the technologies we use to build and why we use them. However it made some assumptions that the reader would need to know about many of those technologies already to understand the concepts, and it doesn’t really create a path for someone to do the same if they are a beginner. This post is for those that are learning these technologies and that want a walkthrough on how to build an app from scratch in the same way that we do.

We’re going to build a small App from scratch that has a server and a web interface. We'll start with an API definition, we'll use this definition to generate our server handler code, as well as our frontend clients. By the end, you should have a stand-alone binary application that you can run.

Note that for this tutorial, you'll need to have git, Go, buf, and npm installed on your machine.

First, we’re going to start by defining the API that we want our server and clients to use to communicate. For this definition, we’ll use the protobuf language. Protobuf is powerful because it’s language agnostic and allows us to define an API contract and then generate server/client stubs for that contract in many different languages. So we can define the API once, and then generate code for any languages that we would like to implement.

We'll be using a compiler and dependency manager for our protobufs called buf. Buf makes handling protobufs easy. I highly recommend the buf tour if you want to learn more about using buf and protobufs. If you follow that tour, you'll build an API for a PetStore, and we'll pick up where that tour leaves off.

We'll start off by cloning the buf tour repo

git clone https://github.com/bufbuild/buf-tour.git

However since we'll be starting where that tour leaves off, we'll be using the finish part of the tour.

cd buf-tour/finish

In the buf tour, you'll have defined a protobuf API for a Pet Store located at petapis/pet/v1/pet.proto. It should look something like this:

syntax = "proto3";


package pet.v1;


option go_package = "github.com/bufbuild/buf-tour/petstore/gen/proto/go/pet/v1;petv1";


import "google/type/datetime.proto";


// PetType represents the different types of pets in the pet store.
enum PetType {
  PET_TYPE_UNSPECIFIED = 0;
  PET_TYPE_CAT = 1;
  PET_TYPE_DOG = 2;
  PET_TYPE_SNAKE = 3;
  PET_TYPE_HAMSTER = 4;
}


// Pet represents a pet in the pet store.
message Pet {
  PetType pet_type = 1;
  string pet_id = 2;
  string name = 3;
  google.type.DateTime created_at = 4;
}


message GetPetRequest {
  string pet_id = 1;
}


message GetPetResponse {
  Pet pet = 1;
}


message PutPetRequest {
  PetType pet_type = 1;
  string name = 2;
}


message PutPetResponse {
  Pet pet = 1;
}


message DeletePetRequest {
  string petID = 1;
}


message DeletePetResponse {}


service PetStore {
  rpc GetPet(GetPetRequest) returns (GetPetResponse) {}
  rpc PutPet(PutPetRequest) returns (PutPetResponse) {}
  rpc DeletePet(DeletePetRequest) returns (DeletePetResponse) {}
}

The tour should have left off with a partial implementation of this API, and a little client application to talk to the server. This is where we'll continue the tour. We'll first start by making the implementation of the SetPet and GetPet API functions a little more interesting. We'll keep an in-memory store of the Pets that we have in our store and allow users to find pets by their ID.

Update your server code found in server/main.go to look like below.

// petStoreServiceServer implements the PetStoreService API.
type petStoreServiceServer struct {
    petv1.UnimplementedPetStoreServiceServer
    pets map[string]*petv1.Pet
}


// PutPet adds the pet associated with the given request into the PetStore.
func (s *petStoreServiceServer) PutPet(ctx context.Context, req *petv1.PutPetRequest) (*petv1.PutPetResponse, error) {
    name := req.GetName()
    petType := req.GetPetType()
    log.Println("Got a request to create a", petType, "named", name)


    p := &petv1.Pet{
        PetId:   uuid.New().String(),
        PetType: petType,
        Name:    name,
    }


    // Save the pet in memory
    s.pets[p.PetId] = p


    return &petv1.PutPetResponse{
        Pet: p,
    }, nil
}


func (s *petStoreServiceServer) GetPet(ctx context.Context, req *petv1.GetPetRequest) (*petv1.GetPetResponse, error) {
    log.Println("Got a request to Get", req.PetId)


    // Get the pet from memory
    p, ok := s.pets[req.PetId]
    if !ok {
        return &petv1.GetPetResponse{}, nil
    }


    return &petv1.GetPetResponse{
        Pet: p,
    }, nil
}

This will leave us with a server that will store Pet objects in memory, and allow us to retrieve them with the GetPet API.

Now that we have a server that can actually Get and Set Pet objects, we're going to extend our protobuf definition of the PetStore to allow us to make normal HTTP requests to the server to perform these actions in addition to the gRPC implementation. We'll modify our pet.proto file to have the annotations below. These allow us to tie normal HTTP routes to a gRPC implementation. So using the example below, if we wanted to make the GetPet call, we could either send a gRPC request, or a normal HTTP GET request to /v1/pets/{pet_id}.

service PetStoreService {
  rpc GetPet(GetPetRequest) returns (GetPetResponse) {
      option (google.api.http) = {
        get: "/v1/pets/{pet_id}"
      };
  }
  rpc PutPet(PutPetRequest) returns (PutPetResponse) {
      option (google.api.http) = {
        post: "/v1/pets"
        body: "*"
      };
  }
  rpc DeletePet(DeletePetRequest) returns (DeletePetResponse) {
      option (google.api.http) = {
        delete: "/v1/pets/{pet_id}"
      };
  }
  rpc PurchasePet(PurchasePetRequest) returns (PurchasePetResponse) {
      option (google.api.http) = {
        post: "/v1/pets/{pet_id}"
        body: "*"
      };
  }
}

Now we need to tell buf that we want to generate a grpc-gateway from out API definitions. GRPC-gateway generates the proxy that will translate the HTTP request into the gRPC request. We'll update our buf.gen.yaml file to include a generation step for grpc-gateway.

plugins:
  - name: grpc-gateway
    out: gen/proto/go
    opt:
      - paths=source_relative
      - generate_unbound_methods=true

Now if we run our buf generate command again we'll see some new proto files that have been created, these are our new gateway files.

buf generate

So, what we've done so far is

  • Implement our server to actually store and retrieve pets
  • Defined HTTP paths for our API's
  • Generated the gRPC-gateway code for the HTTP requests

Next, we'll need to tell our server how to handle HTTP and gRPC requests simultaneously. We'll update our server code to look like below:

func run() error {
      port := ":3000"
  
      server := grpc.NewServer()
      petv1.RegisterPetStoreServiceServer(server, &petStoreServiceServer{
          pets: map[string]*petv1.Pet{},
      })
  
      // Register the grpc-gateway
      grpcWebMux := runtime.NewServeMux()
      petv1.RegisterPetStoreServiceHandlerFromEndpoint(
          context.Background(),
          grpcWebMux,
          port,
          []grpc.DialOption{
              grpc.WithInsecure(),
          },
      )
  
      log.Println("starting server", port)
      return http.ListenAndServe(port, grpcHandlerFunc(server, grpcWebMux))
  }
  
  func grpcHandlerFunc(grpcServer *grpc.Server, otherHandler http.Handler) http.Handler {
      return h2c.NewHandler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
          if r.ProtoMajor == 2 && strings.Contains(r.Header.Get("Content-Type"), "application/grpc") {
              grpcServer.ServeHTTP(w, r)
          } else {
              otherHandler.ServeHTTP(w, r)
          }
      }), &http2.Server{})
  }

This change does a couple of things, first, it registers our grpc-gateway code with the gRPC server using RegisterPetStoreServiceHandlerFromEndpoint. Secondly, it updates the handler function with grpcHandlerFunc which processes an HTTP request, checks if it's a gRPC request by looking at the ProtoMajor field and the Content-Type in the request headers. Based on those settings it decides if it should respond with HTTP or a gRPC response.

We can demonstrate this works by updating our client code found at client/main.go.

request := fmt.Sprintf("http://127.0.0.1:3000/v1/pets/%v", resp.Pet.PetId)
    log.Println("HTTP GET request", request)
    r, err := http.Get(request)
    if err != nil {
        return fmt.Errorf("failed to Get pet: %w", err)
    }
    defer r.Body.Close()


    b, err := ioutil.ReadAll(r.Body)
    if err != nil {
        return fmt.Errorf("failed to read response: %w", err)
    }


    petresp := &petv1.GetPetResponse{}
    if err := json.Unmarshal(b, petresp); err != nil {
        return fmt.Errorf("failed to unmarshal pet: %w", err)
    }


    fmt.Println("HTTP Get request returned: ", petresp)

If we add this code after the SetPet call we made during the buf tour, we'll be able to retrieve our Pet snake that we stored. You can run this by running the server with go run server/main.go and go run client/main.go. So now we can make requests to our server with both gRPC and HTTP.

Now we could build our front-end code here using these HTTP request paths, but at Polar Signals we use grpc-web, which allows us to make grpc requests directly from our web interface. We'll add a Javascript and a Typescript client generation step to our buf.gen.yaml file.

plugins:
  - name: js
    out: ui/client/
    opt: import_style=commonjs,binary
  - name: ts
    out: ui/client
    path: ./node_modules/.bin/protoc-gen-ts
    opt:
      - service=grpc-web

Now we can run our buf generate step again, and we should find some generated js and ts files in our repo.

buf generate

Now that we have our front-end clients it's time that we build our front-end. At Polar Signals we use Nextjs for our front-end code, but for this demo, we'll be using Svelte, because it's really quick to setup, and it's easy to understand and write. We'll be using the quick start tutorial on Svelte for this demo.

So we'll follow that guide and create a new Svelte demo in our current directory

cd ui
npx degit sveltejs/template petstore
cd petstore


npm install

We'll edit the src/App.svelte file to include the clients that we generated before by adding import statements in our script tag. After the import statements, we'll create a client that can talk to our server at localhost:3000. Once we have a client, we can define a function that will save a pet, and after saving the pet will retrieve the pet that we just saved, and set the variable name to the name of the pet that was just saved. You can see what this all looks like in the snippet below.

<script>
    import {PetStoreServiceClient} from '../../client/pet/v1/pet_pb_service';
    import {GetPetRequest, PutPetRequest} from '../../client/pet/v1/pet_pb';


    let name;
    const client = new PetStoreServiceClient('http://127.0.0.1:3000');


    // submitPet sends the pet name to the backend; and retrieves it from the backend
    function submitPet(e) {


        const formData = new FormData(e.target);
        const data = {};
        for (let field of formData) {
            const [k, v] = field;
            data[k] = v;
        }


        console.log(data.petname);


        const req = new PutPetRequest();
        req.setName(data.petname);
        client.putPet(req, {}, function(err, response) {
            const id = response.toObject().pet.petId;
            const getReq = new GetPetRequest();
            getReq.setPetId(id);
            client.getPet(getReq, {}, function(err, response) {
                    name = response.toObject().pet.name
            });
        });
    }


</script>

Now that we have this function, we can utilize it in the code. We'll update our main section to look like this

<main>
    {#if name}
        <h1>Hello {name}!</h1>
    {/if}
    <form on:submit|preventDefault={submitPet}>
        <input id="petname" name="petname">
        <button type="submit">
            Submit
        </button>
    </form>
</main>

This will get us a web interface if we run npm run dev that tries and makes grpc-web requests to the server, however, we haven't told the server how to handle these types of requests. So we're going to go back to server/main.go to update the handler to handle grpc-web requests.

We'll add two new packages to our server code

"github.com/go-chi/cors"
"github.com/improbable-eng/grpc-web/go/grpcweb"

This will allow us to add a CORS wrapper so that we can make requests to the server from a browser running on the same machine, and will also allow us to process a grpc-web request. Below are the changes to the server code that we need to make for the handler function. This will use the grpcweb package to determine if a request should be handled by grpc-web, or our normal HTTP handler.

diff --git a/finish/server/main.go b/finish/server/main.go
index d1eef8b..6a61849 100644
--- a/finish/server/main.go
+++ b/finish/server/main.go
@@ -9,8 +9,10 @@ import (
        // This import path is based on the name declaration in the go.mod,
        // and the gen/proto/go output location in the buf.gen.yaml.
        petv1 "github.com/bufbuild/buf-tour/petstore/gen/proto/go/pet/v1"
+       "github.com/go-chi/cors"
        "github.com/google/uuid"
        "github.com/grpc-ecosystem/grpc-gateway/v2/runtime"
+       "github.com/improbable-eng/grpc-web/go/grpcweb"
        "golang.org/x/net/http2"
        "golang.org/x/net/http2/h2c"
        "google.golang.org/grpc"
@@ -23,7 +25,7 @@ func main() {
 }
 
 func run() error {
-       port := ":8080"
+       port := ":3000"
 
        server := grpc.NewServer()
        petv1.RegisterPetStoreServiceServer(server, &petStoreServiceServer{
@@ -42,14 +44,27 @@ func run() error {
        )
 
        log.Println("starting server", port)
-       return http.ListenAndServe(port, grpcHandlerFunc(server, grpcWebMux))
+       return http.ListenAndServe(port, cors.AllowAll().Handler(grpcHandlerFunc(server, grpcWebMux)))
 }
 
 func grpcHandlerFunc(grpcServer *grpc.Server, otherHandler http.Handler) http.Handler {
+       wrappedGrpc := grpcweb.WrapServer(grpcServer,
+               grpcweb.WithAllowNonRootResource(true),
+               grpcweb.WithOriginFunc(func(origin string) bool {
+                       return true
+               }))
+
        return h2c.NewHandler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
                if r.ProtoMajor == 2 && strings.Contains(r.Header.Get("Content-Type"), "application/grpc") {
                        grpcServer.ServeHTTP(w, r)
                } else {
+
+                       // handle grpc web requests
+                       if wrappedGrpc.IsGrpcWebRequest(r) {
+                               wrappedGrpc.ServeHTTP(w, r)
+                               return
+                       }
+
                        otherHandler.ServeHTTP(w, r)
                }
        }), &http2.Server{})

Now that we have this implemented, you should be able to run the server go run server/main.go and run the web interface npm run dev. Navigate on a browser to http://localhost:3000 and you should be able to store pets straight from the web interface. (you may have to change the port that either the server or the web interface listens on)

So now that we have a working front-end and server, we need to tie it all together and create a single binary that runs this application. We'll start by building our frontend code.

npm run build

This should generate all the front-end files that we need in the public directory. To get these HTML and Javascript files into our go application we need to use Go's embed directive. We'll create a file right alongside our public directory called ui.go. Its contents to embed the public directory are below.

package ui


import "embed"


//go:embed public
var FS embed.FS

Now the last part we need to do is to tell our server how to handle these embedded files. The way we're going to handle this is, we'll let our handler process requests as normal, but if it can't find a route to handle the request, we'll fallback to serving the frontend UI. To do this we'll add a fall back wrapper to the end of our server/main.go file.

// wrapResponseWriter is a proxy around an http.ResponseWriter that allows you to hook into the response.
type wrapResponseWriter struct {
    http.ResponseWriter


    wroteHeader bool
    code        int
}


func (wrw *wrapResponseWriter) WriteHeader(code int) {
    if !wrw.wroteHeader {
        wrw.code = code
        if code != http.StatusNotFound {
            wrw.wroteHeader = true
            wrw.ResponseWriter.WriteHeader(code)
        }
    }
}


// Write sends bytes to wrapped response writer, in case of not found it suppresses further writes.
func (wrw *wrapResponseWriter) Write(b []byte) (int, error) {
    if wrw.notFound() {
        return len(b), nil
    }
    return wrw.ResponseWriter.Write(b)
}


func (wrw *wrapResponseWriter) notFound() bool {
    return wrw.code == http.StatusNotFound
}


// fallbackNotFound wraps the given handler with the `fallback` handle to fallback in case of not found.
func fallbackNotFound(handler, fallback http.Handler) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        frw := wrapResponseWriter{ResponseWriter: w}
        handler.ServeHTTP(&frw, r)
        if frw.notFound() {
            w.Header().Del("Content-Type")
            fallback.ServeHTTP(w, r)
        }
    }
}

This is the code that will perform the fallback handling, and next, we'll need to update our handler code to utilize this fallback. But, we'll also need to implement a handler that will actually process all of our embedded files. Go makes this really easy for us, and we can simply use the http.FileServer built-in function to serve embedded files. The final code changes for the server are below.

uifs, _ := fs.Sub(ui.FS, "public")
    uiHandler, _ := uiHandler(uifs)


    log.Println("starting server", port)
    return http.ListenAndServe(port, cors.AllowAll().Handler(grpcHandlerFunc(server, fallbackNotFound(grpcWebMux, uiHandler))))
}


func uiHandler(uifs fs.FS) (*http.ServeMux, error) {
    uiHandler := http.ServeMux{}
    uiHandler.Handle("/", http.FileServer(http.FS(uifs)))
    return &uiHandler, nil
}


func grpcHandlerFunc(grpcServer *grpc.Server, otherHandler http.Handler) http.Handler {
    wrappedGrpc := grpcweb.WrapServer(grpcServer,
        grpcweb.WithAllowNonRootResource(true),
        grpcweb.WithOriginFunc(func(origin string) bool {
            return true
        }))


    return h2c.NewHandler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if r.ProtoMajor == 2 && strings.Contains(r.Header.Get("Content-Type"), "application/grpc") {
            grpcServer.ServeHTTP(w, r)
        } else {


            // handle grpc web requests
            if wrappedGrpc.IsGrpcWebRequest(r) {
                wrappedGrpc.ServeHTTP(w, r)
                return
            }


            otherHandler.ServeHTTP(w, r)
        }
    }), &http2.Server{})
}

Once we have all of these pieces implemented, we can build our standalone binary go build server/main.go. Now we should have a binary that we can execute, navigate to http://localhost:3000 on a web browser, and see our pet store in action! If you'd like to view the final code from all that's been built in this post you can find the final code here

If you have any questions or suggestions on how we build our APIs, please come join our Discord community server. You can also star the Parca project on GitHub and contributions are welcome!

Discuss:
Sign up for the latest Polar Signals news