Go-storage Source Code Reading: Code Generation

Generating code seems to be idiomatic when programming in golang. And go-storage relys heavily on code generation.

I learned code generation recently and this simple post may help those who are new to code generation (like me).

golang Code Generation Mechanism

Tool: go generate

First, go generate helps integrate code generator into the build process. It does nothing other than simply running a command. It does nothing that couldn’t be done with Make, but it comes with the go tool—no extra installation required—and fits nicely into the Go ecosystem [^1].

When you add the comment //go:generate cmd in your code, then go generate will run cmd. So simple!

cmd can be any program, but usually cmd is a code generator, which is a program that outputs another program.

Library: text/template

You can use a template to generate textual output, and thus generate code—code is just plain text.

text/template is a template engine, with which you can feed golang data structures to a template. Its input format uses “{{” and “}}” to represent data evaluations and control structures.

How does go-storage generate code

From an end-to-end perspective, when developing go-storage, we write definitions in service.toml, then go generate will output generated.go containing corresponding golang definitions. For example:

# in service.toml

type = "byte_array"
description = "specifies the customer-provided encryption key for Amazon S3 to use to encrypt/decrypt the source object. It must be 32-byte AES-256 key."
// in generated.go

// WithServerSideEncryptionCustomerKey will apply server_side_encryption_customer_key value to Options.
// ServerSideEncryptionCustomerKey specifies the customer-provided encryption key for Amazon S3 to use to encrypt/decrypt the source object. It must be 32-byte AES-256 key.
func WithServerSideEncryptionCustomerKey(v []byte) Pair {
	return Pair{
		Key:   pairServerSideEncryptionCustomerKey,
		Value: v,

// ...

What happened internally?

go-storage has a code generator go-storage/cmd/definitions, which uses text/template to generate code. The templates are *.tmpl files in go-storage/cmd/definitions/tmpl. The input to the template is the Data struct in definitions/type.go.

// Data is the biggest container for all definitions.
type Data struct {
	Pairs      map[string]*Pair
	Infos      []*Info
	InfosMap   map[string][]*Info
	ObjectMeta []*Info
	Service    *Service

	Interfaces    []*Interface
	interfacesMap map[string]*Interface

	// Store all specs for encoding
	pairSpec       specs.Pairs
	infoSpec       specs.Infos
	operationsSpec specs.Operations
	serviceSpec    specs.Service

So the *.toml will be parsed into Data, then fed into *.tmpl to create generated.go. Interestingly, we can notice that there are both Pair and specs.Pair in Data. Let’s take a look at their definitions:

// definitions.Pair
type Pair struct {
	Name string

	ptype string

	// Runtime generated
	Global      bool
	Description string

// specs.Pair
type Pair struct {
	Name        string
	Type        string
	Description string

We can view specs.Pair as a relatively direct translation of TOML definitions. But to fill in the template, we may need to do some pre-processing, e.g., convert a string-represented type (like byte-array) into a golang type like []byte. That’s why definitions re-defines types like definitions.Pair.

In my view, specs.Pair is TOML-oriented, while definitions.Pair is template-oriented, and there is a conversion between them performed in definitions.


  • specs specifies how to parse TOML files: what sections and keys are valid in a TOML input file and exposes the parsed data types. This doc talks about definitions in detail.
  • go-storage/definitions uses specs to parse TOML files, pre-process the parsed data into Data, and then feed it to *.tmpl using text/template to generate code.

[^1]: The Go Blog: Generating code


Next episode (if there is one): what work is done in go-storage/cmd/definitions :thinking:


This post helps me a lot in understanding code generation in go-storage, thanks!
Looking forward to the next explaining go-storage/cmd/definitions. :grinning:

1 Like

In The next generation of go-storage definitions, we are introduing the new engine powered by GitHub - Xuanwo/gg: General Golang Code Generator :star_struck: