Ad meliora

Nov 27, 2023

Adventures in Golang (Tests)

OUTLINE

PROBLEM STATEMENT

So I had finished the first version of my code implementing a npm registry server in Go. This version has no tests. I have been testing it manually. I wanted to add filesystem locking and deploy the code through CI. So I needed to add automated tests(unit and integration) as I wanted complement my manual testing.

Here is a snippet of what I wanted to write a unit test for:

package handler

import (
    "fmt"
    "net/http"
    "net/url"

    "github.com/gorilla/mux"
    "github.com/sirupsen/logrus"

    "gosimplenpm/internal/config"
    "gosimplenpm/internal/storage"
)

func GetPackage(lg *logrus.Logger, cfg config.Config) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        escapedName := mux.Vars(r)["name"]
        packageName, _ := url.PathUnescape(escapedName)
        lg.WithFields(logrus.Fields{
            "function": "get-package",
        }).Debugf("Package name => %s\n", packageName)

        fileToServe, found, err := storage.GetIndexJsonFromStore(packageName, cfg.RepoDir, lg)
        if err != nil {
            http.Error(w, err.Error(), http.StatusInternalServerError)
            return
        }

        if !found {
            ret := fmt.Sprintf("Package not found: %s", packageName)
            http.Error(w, ret, http.StatusNotFound)
            return
        }

        // serve file
        http.ServeFile(w, r, fileToServe)
    }
}

So I wanted to test this function without also testing the storage.GetIndexJsonFromStore function. The storage.GetIndexJsonFromStore function makes calls to the filesystem. Well, coming from Javascript, I would use a mocking framework like Sinon.JS 1 to mock out the storage.GetIndexJsonFromStore function without modifying the GetPackage handler.

FIRST APPROACH

So using my limited knowledge of Go, I wrote an interface Storage and defined the method GetIndexJsonFromStore like below:

package storage

import (
    "gosimplenpm/internal/serviceidos"

    "github.com/sirupsen/logrus"
)

type Storage interface {
    GetIndexJsonFromStore(string, string, *logrus.Logger) (string, bool, error)
    // Other methods omitted for brevity
}

Then I can define a struct MockFS that would implement the method GetIndexJsonFromStore. This is because structs that implement all the methods of an interface are objects of that interface.

type MockFs struct {
    packageJson               string
    retrieved                 bool
    err                       error
    called                    bool
}

func (m *MockFs) SetError(err error) {
    m.err = err
}

func (m *MockFs) SetRetrieved(retrieved bool) {
    m.retrieved = retrieved
}

func (m *MockFs) SetFileToServe(fileToServe string) {
    m.packageJson = fileToServe
}

func (m *MockFs) GetIndexJsonFromStore(packageName string, registryPath string, lg *logrus.Logger) (string, bool, error) {
    m.called = true
    return m.packageJson, m.retrieved, m.err
}

I would then modify the previously defined GetIndexJsonFromStore method to allow for me to inject the MockFs dependency.

package handler

import (
    "fmt"
    "net/http"
    "net/url"

    "github.com/gorilla/mux"
    "github.com/sirupsen/logrus"

    "gosimplenpm/internal/config"
    "gosimplenpm/internal/storage"
)

func GetPackage(lg *logrus.Logger, cfg config.Config, stg storage.Storage) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        escapedName := mux.Vars(r)["name"]
        packageName, _ := url.PathUnescape(escapedName)
        lg.WithFields(logrus.Fields{
            "function": "get-package",
        }).Debugf("Package name => %s\n", packageName)

        fileToServe, found, err := stg.GetIndexJsonFromStore(packageName, cfg.RepoDir, lg)
        if err != nil {
            http.Error(w, err.Error(), http.StatusInternalServerError)
            return
        }

        if !found {
            ret := fmt.Sprintf("Package not found: %s", packageName)
            http.Error(w, ret, http.StatusNotFound)
            return
        }

        // serve file
        http.ServeFile(w, r, fileToServe)
    }
}

A sample test would be something like this

package handler

import (
    "bytes"
    "fmt"
    "gosimplenpm/internal/config"
    "gosimplenpm/internal/storage"
    "io"
    "net/http"
    "net/http/httptest"
    "os"
    "testing"

    "github.com/gorilla/mux"
    "github.com/sirupsen/logrus"
    "github.com/stretchr/testify/assert"
)

func TestGet(t *testing.T) {
    t.Run("return `Not Found` error if package is not found", func(t *testing.T) {
        req := httptest.NewRequest(http.MethodGet, "/{name}", nil)
        wrt := httptest.NewRecorder()

        log := &logrus.Logger{
            Out: os.Stdout,
            Formatter: &logrus.TextFormatter{
                FullTimestamp:   true,
                TimestampFormat: "2009-01-02 15:15:15",
            },
        }

        cfg := config.Config{
            RepoDir: "",
        }

        mfs := &storage.MockFs{}
        mfs.SetRetrieved(false)


        //Hack to try to fake gorilla/mux vars
        vars := map[string]string{
            "name": "test-package",
        }

        req = mux.SetURLVars(req, vars)

        GetPackage(log, cfg, mfs)(wrt, req)

        rs := wrt.Result()

        assert.Equal(t, rs.StatusCode, http.StatusNotFound)

        defer rs.Body.Close()
        body, err := io.ReadAll(rs.Body)
        if err != nil {
            t.Fatal(err)
        }
        bytes.TrimSpace(body)

        assert.Equal(t, string(body), "Package not found: test-package\n")
    })
}

This approach fails if I wanted to test another function stg.GetOtherJsonFromNet of the interface like below:

package handler

import (
    "fmt"
    "net/http"
    "net/url"

    "github.com/gorilla/mux"
    "github.com/sirupsen/logrus"

    "gosimplenpm/internal/config"
    "gosimplenpm/internal/storage"
)

func GetPackage(lg *logrus.Logger, cfg config.Config, stg storage.Storage) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        escapedName := mux.Vars(r)["name"]
        packageName, _ := url.PathUnescape(escapedName)
        lg.WithFields(logrus.Fields{
            "function": "get-package",
        }).Debugf("Package name => %s\n", packageName)

        fileToServe, found, err := stg.GetIndexJsonFromStore(packageName, cfg.RepoDir, lg)
        if err != nil {
            http.Error(w, err.Error(), http.StatusInternalServerError)
            return
        }
        if !found {
            fileToServe, err = stg.GetOtherJsonFromNet(packageName, lg)
            if err != nil {
                http.Error(w, err.Error(), http.StatusInternalServerError)
                return
            }
        }
        //...code omitted for brevity
    }
}

SECOND APPROACH

To handle this new scenario, that is, to test the other function GetOtherJsonFromNet as well as GetIndexJsonFromStore, I would modify the struct and its implementations to something like this:

type MockFs struct {
    GetIndexJsonFromStoreFunc func(packageName string, registryPath string, lg *logrus.Logger) (string, bool, error)
    GetOtherJsonFromNetFunc func(packageName string, lg *logrus.Logger) (string, error)
}

func (m *MockFs) GetIndexJsonFromStore(packageName string, registryPath string, lg *logrus.Logger) (string, bool, error) {
    return m.GetIndexJsonFromStoreFunc(packageName, registryPath, lg)
}

func (m *MockFs) GetOtherJsonFromNet(packageName string, lg *logrus.Logger) (string, error) {
    return m.GetOtherJsonFromNetFunc(packageName, lg)
}

Basically since I cannot reassign a function to an already defined struct (as I could in Javascript with objects), I would instead define function variables as part of the MockFS struct. That way, I can redefine those function variables for each test I want to run. An example of a test that uses this is shown below:

package handler

import (
    "bytes"
    "fmt"
    "gosimplenpm/internal/config"
    "gosimplenpm/internal/storage"
    "io"
    "net/http"
    "net/http/httptest"
    "os"
    "testing"

    "github.com/gorilla/mux"
    "github.com/sirupsen/logrus"
    "github.com/stretchr/testify/assert"
)

func TestGet(t *testing.T) {
    t.Run("return `Internal Server` error if package cannot be retrieved either from disk or from the internet", func(t *testing.T) {
        req := httptest.NewRequest(http.MethodGet, "/{name}", nil)
        wrt := httptest.NewRecorder()

        log := &logrus.Logger{
            Out: os.Stdout,
            Formatter: &logrus.TextFormatter{
                FullTimestamp:   true,
                TimestampFormat: "2009-01-02 15:15:15",
            },
        }

        cfg := config.Config{
            RepoDir: "",
        }

        mfs := &storage.MockFs{}
        mfs.GetIndexJsonFromStore = func (packageName string, registryPath string, lg *logrus.Logger) (string, bool, error) {
            return "", false, fmt.Errorf("Failure to retrieve disk")
        }
        mfs.GetOtherJsonFromNet = func (packageName string, lg *logrus.Logger) (string, error) {
            return "", false, fmt.Errorf("Failure to retrieve from internet")
        }

        //Hack to try to fake gorilla/mux vars
        vars := map[string]string{
            "name": "test-package",
        }

        req = mux.SetURLVars(req, vars)

        GetPackage(log, cfg, mfs)(wrt, req)

        rs := wrt.Result()

        assert.Equal(t, rs.StatusCode, http.StatusInternalServerErro)

        defer rs.Body.Close()
        body, err := io.ReadAll(rs.Body)
        if err != nil {
            t.Fatal(err)
        }
        bytes.TrimSpace(body)

        assert.Equal(t, string(body), "Failure to retrieve from internet\n")
    })
}

This approach avoids the usage of struct-level variables, making it easier to easily fine-tune your tests.

Now, I had already moved all my tests to this approach before I found out about GoMock2. In the future, I may move to GoMock2 instead.

  1. Sinon.JS ↩

  2. GoMock ↩↩

posted at 22:04  ·   ·  Things I learned

Nov 16, 2023

Adventures in Golang (Package registry)

OUTLINE

INTRODUCTION

I wanted to learn more about golang. I had used it in my previous job roles and I have been fascinated by the language. So I decided to deepen my knowledge of the language through doing projects.

One such project is a package registry. That every language that I have used so far (java, js, golang, ruby, python) have a place where you can get extra third-party libs, other than what it is normally included in the standard library of the language. That place is called a package registry. So in looking for one to implement, I stumbled on the npm registry. The api is here.

A package registry is basically a storage server with extra sauce (code). So basically a storage server provides clients (requesters), the means for uploading data to it and downloading data from it. So the npm client (the npm command) is all about interacting with the public npm registry or any registry that implements a subset or all of the methods of the api. For a simple private npm registry, we will support these commands when run by the client:

(a) npm install which does these actions when run on the client;

  • getting the package json for a particular package from the registry and
  • installing all the dependencies (first-party and third party dependencies from the pulled package.json)

(b) npm view. A sample run of the command is shown below
npm_view_command

This command just retrieves the package.json from the registry and produces the output shown above.

(c) npm tag. A tag is an alias for a version of a package. This comamnd adds alias (tags) to a published version of a package. Basically, the non client sends these tags to the npm registry for those tags to be added to the package.json. This can be viewed by the npm view command.

(d) npm publish. This command actually uploads a version of a package to the registry after which the npm install command can be used to download the package.

(e) npm unpublish. This command deletes the package from the registry.

So what is a package according to npm? A package is described here and is:

1) a folder containing a program described by a package.json file
2) a gzipped tarball containing (1)
3) a url that resolves to (2)
4) a <name>@<version> that is published on the registry with (3)
5) a <name>@<tag> that points to (4)
6) a <name> that has a "latest" tag satisfying (5)
7) a <git remote url> that resolves to (1)

So for the private npm registry, we use @scope that is defined in the .npmrc file. The @scope is defined in the .npmrc as @scope:registry. This tells the npm client to resolve the url for the package to install to the value of the @scope:registry as shown below: 

# in the .npmrc file  
@scope:registry=<registry-url>

# npm command
npm install @scope/<package>

# is resovled by the npm client to
npm install <registry-url>/<package>

NOTES ON CREATING SERVERS IN GOLANG

In Go, an HTTP server relies on several interacting components: handlers, middleware, and a multiplexer( router ). When it includes all these parts, we call this server a web service.

First, the server’s multiplexer receives the client’s request. The multiplexer determines the destination for the request, then passes it along to the object capable of handling it. We call this object a handler. (The multiplexer itself is a handler that routes requests to the most appropriate handler.) Before the handler receives the request, the request may pass through one or more functions called middle-ware. Middleware changes the handlers’ behavior or performs auxiliary tasks, such as logging, authentication, or access control.

The mulitplexer here is an an object of type ServeMux defined in the http package. This ServeMux object contains, among other fields, a map data structure containing a mapping of the paths that you want your server to handle and the corresponding handler function.

A handler function must be of type func(http.ResponseWriter, *http.Request) ,where http.ResponseWriter and http.Request are two struct types defined in the net/http package. The object of type http.Request represents the incoming HTTP request and the http.ResponseWriter object is used to write back the response to the client making the request. The following is an example of a handler function:

func apiHandler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello World")
}

Anything you write to the ResponseWriter object, w , is sent back as a response to the client. Once you have written your handler function, the next step is to register it with a ServeMux object as shown below:

mux := http.NewServeMux()
mux.HandleFunc("/api", apiHandler)

An HTTP handler function accepts two parameters: a value of type http.ResponseWriter and a pointer to a value of type http.Request . The pointer object of http.Request type (defined in the net/http package) describes the incoming request. This type is also used to define an outgoing HTTP request.

A middleware function is defined as 

func middleware(next http.Handler) http.Handler {
    return func(w http.ResponseWriter, r *http.Request) {
        // TODO: Execute our middleware logic here...
        next.ServeHTTP(w, r)
    }
}

NOTES ON CREATING THIS PROJECT

In creating this project, I learned certain things:

(a) In trying to organize my files, I came across this link and this link. These links describe how to organize your golang project, which is something like this

.
├── .git
├── cmd
|   └── myapp
|       └── main.go
├── examples
|   └── example1
|       └── main.go
├── internal
|   ├── config
|   |   └── config.go
|   └── store
|       └── store.go
├── pkg
|   ├── public1
|   |   └── public1.go
|   └── public2
|       └── public2.go
├── go.mod
└── go.sum

I had initially wanted to have the structure but it proved too daunting initially. So I had this folder structure instead

.
├── LICENSE.txt
├── Makefile
├── README.md
├── app.go
├── go.mod
├── go.sum
├── handler
│   ├── get.go
│   ├── publish.go
│   ├── tagdelete.go
│   ├── tagget.go
│   ├── tagput.go
│   └── tar.go
├── main.go
└── storage
    └── fs.go

(b) I wanted to use the http.ServeMux but switched to gorilla/mux instead based on the following reasons (and based on the following flowchart):
- it does not support variables in Url paths - it is very similar to http.ServeMux - it is very well supported and archived.

(c) When testing the handlers, you do have to set an authToken like this

npm config set //<registry-url-without-the-protocol>/:_authToken <token>

Because if you do not set the authToken, the npm client asks you to run npm adduser which sets an authToken.

(d) If you want to pass something like config or a struct to the middleware or to the handlers,

# middleware
func middleware(s someStruct) func (http.HandlerFunc) http.HandlerFunc {
    return func(next http.HandlerFunc) http.HandlerFunc {
        return func(w http.ResponseWriter, r *http.Request) {
            // TODO: Execute our middleware logic here...
            next.ServeHTTP(w, r)
        }
    }
}

# handler
func apiHandler(s someStruct) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintf(w, "Hello World")
    }
}
posted at 22:04  ·   ·  golang