Off The Point

human ⚯ engineer ⚯ musician

Migrating a Standalone Go Project from Dep to Go Mod

This is a much shorter article than I though it would be when I started migrating our, admittedly small, projects to go mod.

I’ve been migrating some test infrastructure to Kubernetes at work lately and, at some stage during the process, it became clear that we needed a tiny Microservice (~300LOC with tests) to hand out unique tokens to all pods that make up one run of load testing. This is somewhat bad practice but was not avoidable due to the way our load test application worked internally. I won’t discuss the details of the service here but the important part was the I had initially decided to use Go and go dep for dependency management. It worked surprisingly well.

One thing that really bothered me about Go was the whole $GOPATH mess. I had a hard time understanding the reasoning behind enforcing this structure and it seemed a bit outdated in 2018 (when I started these projects). Versioning was a pain and all external packages of all dependencies were kept in the same folder (and I’ve seen project check those into their version control – yuck).

The good news is that go mod allows projects to live outside $GOPATH and it improves package management a lot. For simple projects like this one, it completely eliminates package management since it scans the source files for imports and automatically downloads them.

Another little downside before go mod was CI. Our build was pretty simple, but it still depended on dep which had to be installed on each build (because I was lazy and didn’t invest more time into setting things up) before it could pull dependencies and run tests. This is now gone as well.

Prerequisites

First of all, go mod was introduced as an experimental feature with go 1.11, so make sure to grab at least that version. I’m using

--- projects/go-dep-mod ‹master› » go version
go version go1.12.5 darwin/amd64

for this small demonstration.

Getting rid of dep

Being an external dependency, the only thing to do here is to remove the Gopkg files. Starting from the project like this

--- projects/go-dep-mod » lt
Permissions Size User  Date Modified Name
drwxr-xr-x     - dsere 17 May 11:01  .
drwxr-xr-x     - dsere 17 May 11:01  ├── counters
.rw-r--r--  1.5k dsere 17 May 11:01  │  ├── counters.go
.rw-r--r--  2.2k dsere 17 May 11:01  │  └── counters_test.go
.rw-r--r--  2.0k dsere 17 May 11:01  ├── Gopkg.lock
.rw-r--r--   741 dsere 17 May 11:01  ├── Gopkg.toml
.rw-r--r--   851 dsere 17 May 11:01  ├── main.go
.rw-r--r--  1.6k dsere 17 May 11:01  ├── main_test.go
.rw-r--r--   893 dsere 17 May 11:01  └── README.md

We can go ahead and

rm Gopkg*

Initializing a Go Module

Next, we tell go to initialize a module

go mod init github.com/you/go-dep-mod

This will create 2 new files

--- projects/go-dep-mod » lt
Permissions Size User  Date Modified Name
drwxr-xr-x     - dsere 17 May 11:03  .
drwxr-xr-x     - dsere 17 May 11:02  ├── counters
.rw-r--r--  1.7k dsere 17 May 11:02  │  ├── counters.go
.rw-r--r--  2.6k dsere 17 May 11:02  │  └── counters_test.go
.rw-r--r--   570 dsere 17 May 11:03  ├── Dockerfile
.rw-r--r--   680 dsere 17 May 11:03  ├── go.mod
.rw-r--r--  2.1k dsere 17 May 11:03  ├── go.sum
.rw-r--r--  1.1k dsere 17 May 11:03  ├── main.go
.rw-r--r--  2.0k dsere 17 May 11:03  ├── main_test.go
.rw-r--r--  1.9k dsere 17 May 11:03  └── README.md

Let’s have a look at go.mod

--- projects/go-dep-mod » bat go.mod
───────┬──────────────────────────────────────────────────────────
       │ File: go.mod
───────┼──────────────────────────────────────────────────────────
   1   │ module github.com/you/go-dep-mod
   2   │
   3   │ require (
   4   │     github.com/davecgh/go-spew v1.1.0 // indirect
   5   │     github.com/labstack/echo v3.3.5+incompatible
   6   │     github.com/labstack/gommon v0.0.0-20180613044413-d6898124de91 // indirect
   7   │     github.com/mattn/go-colorable v0.0.9 // indirect
   8   │     github.com/mattn/go-isatty v0.0.4 // indirect
   9   │     github.com/pmezard/go-difflib v1.0.0 // indirect
  10   │     github.com/stretchr/testify v1.2.2
  11   │     github.com/valyala/bytebufferpool v1.0.0 // indirect
  12   │     github.com/valyala/fasttemplate v0.0.0-20170224212429-dcecefd839c4 // indirect
  13   │     golang.org/x/crypto v0.0.0-20180608092829-8ac0e0d97ce4 // indirect
  14   │     golang.org/x/sys v0.0.0-20181122145206-62eef0e2fa9b // indirect
  15   │ )
───────┴──────────────────────────────────────────────────────────

and go.sum

--- projects/go-dep-mod » bat go.sum
───────┬──────────────────────────────────────────────────────────
       │ File: go.sum
───────┼──────────────────────────────────────────────────────────
   1   │ github.com/davecgh/go-spew v1.1.0 h1:ZD<truncated>8=
   2   │ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7<truncated>38=
   3   │ github.com/labstack/echo v3.3.5+incompatible h1:9P<truncated>O4=
   4   │ github.com/labstack/echo v3.3.5+incompatible/go.mod h1:0I<truncated>1s=
   5   │ github.com/labstack/gommon v0.0.0-20180613044413-d6898124de91 h1:6R2om+NMGEk=
   6   │ github.com/labstack/gommon v0.0.0-20180613044413-d6898124de91/go.mod h1:/tj9cNJ4=
   7   │ github.com/mattn/go-colorable v0.0.9 h1:UVL<truncated>4=
   8   │ github.com/mattn/go-colorable v0.0.9/go.mod h1:9v<truncated>ZU=
   9   │ github.com/mattn/go-isatty v0.0.4 h1:bnP<truncated>DYs=
  10   │ github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRX<truncated>i4=
  11   │ github.com/pmezard/go-difflib v1.0.0 h1:4DB<truncated>M=
  12   │ github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH<truncated>YZ/4=
  13   │ github.com/stretchr/testify v1.2.2 h1:bSDN<truncated>oJ1w=
  14   │ github.com/stretchr/testify v1.2.2/go.mod h1:a8<truncated>DkUVs=
  15   │ github.com/valyala/bytebufferpool v1.0.0 h1:GqA<truncated>Pw=
  16   │ github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6b<truncated>sc=
  17   │ github.com/valyala/fasttemplate v0.0.0-20170224212429-dcecefd839c4 h1:gK<truncated>=
  18   │ github.com/valyala/fasttemplate v0.0.0-20170224212429-dcecefd839c4/go.mod h1:fasdw=
  19   │ golang.org/x/crypto v0.0.0-20180608092829-8ac0e0d97ce4 h1:wvi<truncated>I=
  20   │ golang.org/x/crypto v0.0.0-20180608092829-8ac0e0d97ce4/go.mod h1:6S<truncated>4=
  21   │ golang.org/x/sys v0.0.0-20181122145206-62eef0e2fa9b h1:MQE+LT/ABUuuvEZ+mAkup74op4=
  22   │ golang.org/x/sys v0.0.0-20181122145206-62eef0e2fa9b/go.mod h1:STP8<truncated>hY=
───────┴──────────────────────────────────────────────────────────

So these have, in effect, the same contents as the Gopkg files that dep used before: a list of dependencies and a file containing version and checksum information. It’s nice that go mod identifies indirect dependencies as such in the go.mod file.

Getting Dependencies and Building

Now for the beautiful parts of this process: run go mod download to install the dependencies (go mod vendor you want your dependencies into <your_dir>/vendor, i.e. not in the default cache location which is $GOPATH/pkg/mod).

--- projects/go-dep-mod » go mod vendor
--- projects/go-dep-mod » lt
Permissions Size User  Date Modified Name
drwxr-xr-x     - dsere 17 May 11:43  .
drwxr-xr-x     - dsere 17 May 11:02  ├── counters
.rw-r--r--  1.7k dsere 17 May 11:02  │  ├── counters.go
.rw-r--r--  2.6k dsere 17 May 11:02  │  └── counters_test.go
.rw-r--r--   570 dsere 17 May 11:03  ├── Dockerfile
.rw-r--r--   680 dsere 17 May 11:03  ├── go.mod
.rw-r--r--  2.1k dsere 17 May 11:03  ├── go.sum
.rw-r--r--  1.1k dsere 17 May 11:03  ├── main.go
.rw-r--r--  2.0k dsere 17 May 11:03  ├── main_test.go
.rw-r--r--  1.9k dsere 17 May 11:03  ├── README.md
drwxr-xr-x     - dsere 17 May 11:43  └── vendor
drwxr-xr-x     - dsere 17 May 11:43     ├── github.com
drwxr-xr-x     - dsere 17 May 11:43     │  ├── davecgh
drwxr-xr-x     - dsere 17 May 11:43     │  │  └── go-spew
.rw-r--r--   763 dsere 17 May 11:43     │  │     ├── LICENSE
drwxr-xr-x     - dsere 17 May 11:43     │  │     └── spew
.rw-r--r--  5.8k dsere 17 May 11:43     │  │        ├── bypass.go
.rw-r--r--  1.7k dsere 17 May 11:43     │  │        ├── bypasssafe.go
.rw-r--r--   10k dsere 17 May 11:43     │  │        ├── common.go
.rw-r--r--   12k dsere 17 May 11:43     │  │        ├── config.go
.rw-r--r--  8.5k dsere 17 May 11:43     │  │        ├── doc.go
.rw-r--r--   13k dsere 17 May 11:43     │  │        ├── dump.go
.rw-r--r--   11k dsere 17 May 11:43     │  │        ├── format.go
.rw-r--r--  6.0k dsere 17 May 11:43     │  │        └── spew.go

[and many more lines]

One thing that made me very, very happy was that dependency download can be implicit, as in, Go will notice it doesn’t have dependencies and instead of complaining, just go out and get them.

--- projects/go-dep-mod » rm -r vendor/
--- projects/go-dep-mod » go test
go: finding github.com/mattn/go-isatty v0.0.4
go: finding github.com/mattn/go-colorable v0.0.9
go: finding github.com/labstack/gommon v0.0.0-20180613044413-d6898124de91
go: finding github.com/labstack/echo v3.3.5+incompatible
go: finding github.com/valyala/bytebufferpool v1.0.0
go: finding github.com/valyala/fasttemplate v0.0.0-20170224212429-dcecefd839c4
go: finding github.com/davecgh/go-spew v1.1.0
go: finding github.com/stretchr/testify v1.2.2
go: finding golang.org/x/crypto v0.0.0-20180608092829-8ac0e0d97ce4
go: finding golang.org/x/sys v0.0.0-20181122145206-62eef0e2fa9b
go: finding github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/labstack/echo v3.3.5+incompatible
go: downloading github.com/stretchr/testify v1.2.2
go: extracting github.com/stretchr/testify v1.2.2
go: extracting github.com/labstack/echo v3.3.5+incompatible
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/davecgh/go-spew v1.1.0
go: extracting github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/labstack/gommon v0.0.0-20180613044413-d6898124de91
go: downloading golang.org/x/crypto v0.0.0-20180608092829-8ac0e0d97ce4
go: extracting github.com/davecgh/go-spew v1.1.0
go: extracting github.com/labstack/gommon v0.0.0-20180613044413-d6898124de91
go: downloading github.com/mattn/go-isatty v0.0.4
go: downloading github.com/mattn/go-colorable v0.0.9
go: downloading github.com/valyala/fasttemplate v0.0.0-20170224212429-dcecefd839c4
go: extracting github.com/mattn/go-colorable v0.0.9
go: extracting github.com/mattn/go-isatty v0.0.4
go: extracting github.com/valyala/fasttemplate v0.0.0-20170224212429-dcecefd839c4
go: downloading github.com/valyala/bytebufferpool v1.0.0
go: extracting github.com/valyala/bytebufferpool v1.0.0
go: extracting golang.org/x/crypto v0.0.0-20180608092829-8ac0e0d97ce4
PASS
ok  	github.com/you/go-dep-mod	0.023s

Magical, isn’t it?

Next Post

Leave a Reply

© 2019 Off The Point

Theme by Anders Norén