2018-01-16
What does a distro provide? The most popular docker base container image is either busybox, or scratch. This is driven by a movement that is equal parts puritanical and pragmatic. The puritan asks “Why do I need to run init(1) just to run my process?” The pragmatist asks “Why do I need a 700 meg […]
2018-01-08
This is an article about compiler directives; or as they are commonly known, pragmas. It’s derived from a talk of a similar name that I gave last year at GopherChina in Shanghai. But first, a history lesson Before we talk about Go, let’s talk a little about pragmas, and their history. Many languages have the notion […]
2017-12-31

System diagram
I needed to create a simple framework to provide my endpoint devices ( doesn’t matter which platform they run on ) the option to send and receive messages from my backend.
I require those messages to be managed by a message broker so that they can be processed in an asynchronous way.
The system contains 4 main layers, this article section is mainly about the first one:
1. TCP servers - Needs to maintain as many TCP sockets in synch with the endpoints as possible. All of the endpoints messages will be processed on a different layer by the message broker. This keeps the TCP servers layer very thin and effective. I also want to keep as many concurrent connection as possible, and Go is a good choice for this ( see this article)
2. Message broker - Responsible for delivering the messages between the TCP servers layer and the workers layer. I chose Apache Kafka for that purpose.
3. Workers layer - will process the messages through services exposed in the backend layer.
4. Backed services layer - An encapsulation of services required by your application such as DB, Authentication, Logging, external APIs and more.

So, this Go Server:
1. communicates with its endpoint clients by TCP sockets.
2. queues the messages in the message broker.
3. receives back messages from the broker after they were processed and sends response acknowledgment and/or errors to the TCP clients.

The full source code is available in : https://github.com/orrchen/go-messaging
I have also included a Dockerfile and a build script to push the image to your Docker repository.
Special thanks to the great go Kafka sarama library from Shopify.

The article is divided to sections representing the components of the system. Each component should be decoupled from the others in a way that allows you to read about a single component in a straight forward manner.

TCP Client

Its role is to represent a TCP client communicating with our TCP server.

type Client struct {
	Uid  string /* client is responsible of generating a unique uid for each request,   
	it will be sent in the response from the server so that client will know what request generated this response */
	DeviceUid string /* a unique id generated from the client itself */
	conn net.Conn
	onConnectionEvent func(c *Client, eventType ConnectionEventType, e error) /* function for handling new connections */
	onDataEvent func(c *Client, data []byte) /* function for handling new date events */
}

Please notice that onConnectionEvent and onDataEvent are callbacks for the Struct that will obtain and manage Clients.

Our client will listen permanently using the listen() function and response to new connections, new data received and connections terminations.

Kafka Consumer

Its role is to consume messages from our Kafka broker, and to broadcast them back to relevant clients by their uids.
In this example we are consuming from multiple topics using the cluster implementation of sarama.

Let’s define our Consumer struct:

type Consumer struct {
	consumer *cluster.Consumer
	callbacks ConsumerCallbacks
}

The constructor receives the callbacks and relevant details to connect to the topic:

func NewConsumer(callbacks ConsumerCallbacks,brokerList []string, groupId string, topics []string) *Consumer {
	consumer := Consumer{callbacks:callbacks}

	config := cluster.NewConfig()
	config.ClientID = uuid.NewV4().String()
	config.Consumer.Offsets.Initial = sarama.OffsetNewest
	saramaConsumer, err := cluster.NewConsumer(brokerList, groupId, topics, config)
	if err != nil {
		panic(err)
	}
	consumer.consumer = saramaConsumer
	return &consumer

}

It will consume permanently on a new goroutine inside the Consume() function.
It reads from the Messages() channel for new messages and the Notifications() channel for events.

Kafka Producer

Its role is to produce messages to our Kafka broker.
In this example we are producing to a single topic.
This section is mainly inspired from the example in https://github.com/Shopify/sarama/blob/master/examples/http_server/http_server.go

Let’s define our Producer Struct:

type Producer struct {
	asyncProducer sarama.AsyncProducer
	callbacks     ProducerCallbacks
	topic         string
}

Producer is constructed with the callbacks for error, and the details to connect to the Kafka broker including optional ssl configurations that are created with createTLSConfiguration:

func NewProducer(callbacks ProducerCallbacks,brokerList []string,topic string,certFile *string,keyFile *string,caFile *string,verifySsl *bool ) *Producer {
	producer := Producer{ callbacks: callbacks, topic: topic}

	config := sarama.NewConfig()
	tlsConfig := createTLSConfiguration(certFile,keyFile,caFile,verifySsl)
	if tlsConfig != nil {
		config.Net.TLS.Enable = true
		config.Net.TLS.Config = tlsConfig
	}
	config.Producer.RequiredAcks = sarama.WaitForLocal       // Only wait for the leader to ack
	config.Producer.Compression = sarama.CompressionSnappy   // Compress messages
	config.Producer.Flush.Frequency = 500 * time.Millisecond // Flush batches every 500ms

	saramaProducer, err := sarama.NewAsyncProducer(brokerList, config)
	if err != nil {
		log.Fatalln("Failed to start Sarama producer:", err)
		panic(err)
	}
	go func() {
		for err := range saramaProducer.Errors() {
			if producer.callbacks.OnError!=nil {
				producer.callbacks.OnError(err)
			}
		}
	}()
	producer.asyncProducer = saramaProducer
	return &producer
}

I decided to produce messages that are encoded to JSON and to ensure it before sending them:

type message struct {
	value interface{}
	encoded []byte
	err     error
}

func (ale *message) ensureEncoded() {
	if ale.encoded == nil && ale.err == nil {
		ale.encoded, ale.err = json.Marshal(ale.value)
		if ale.err!=nil {
			log.Println(ale.err)
		}
	}
}

func (ale *message) Length() int {
	ale.ensureEncoded()
	return len(ale.encoded)
}

func (ale *message) Encode() ([]byte, error) {
	ale.ensureEncoded()
	return ale.encoded, ale.err
}

And finally, we provide the functions to produce the message and close the producer:

func (p *Producer) Produce(payload interface{}) {
	value := message{
		value: payload,
	}
	value.ensureEncoded()
	log.Println("producing: ", string(value.encoded))
	p.asyncProducer.Input() <- &sarama.ProducerMessage{
		Topic: p.topic,
		Value: &value,
	}
}  
func (p *Producer) Close() error{
	log.Println("Producer.Close()")
	if err := p.asyncProducer.Close(); err != nil {
		return err
	}
	return nil
}

TCP Server

Its role is to obtain and manage a set of Client, and send and receive messages from them.

type TcpServer struct {
	address                  string // Address to open connection: localhost:9999
	connLock sync.RWMutex
	connections map[string]*Client
	callbacks Callbacks
	listener net.Listener
}

It is constructed simply with an address to bind to and the callbacks to send:

// Creates new tcp Server instance
func NewServer(address string, callbacks Callbacks ) *TcpServer {
	log.Println("Creating Server with address", address)
	s := &TcpServer{
		address: address,
		callbacks: callbacks,
	}
	s.connections = make(map[string]*Client)
	return s
}

When a connection event occurs we process it and handle it, if it’s a new event we attach a new UID to the client.
If connection is terminated we delete this client.
In both cases we send the callbacks to notify about those events.

TcpServer will listen permanently for new connections and new data with Listen(), and support a graceful shutdown with Close().

We provide 2 options ot send data to our clients, by their device uid ( generated from the client side) with SendDataByDeviceUidor by the client id which is generated in our system with SendDataByClientId.

API

We need to create structs for the API that the tcp clients use, and the API for the messages sent to/from the messages broker.
For the TCP clients:
* DeviceRequest * DeviceResponse

For the message broker:
* ServerRequest
* ServerResponse

Main function - putting it all together

Obtains and manages all the other components in this system. It will include the TCP server that holds an array of TCP clients, and a connection to the Kafka broker for consuming and sending messages to it. Here are the main parts of main.go file:

var tcpServer *lib.TcpServer
var producer *messages.Producer
var consumer *messages.Consumer

func main() {
    callbacks := lib.Callbacks{
		OnDataReceived: onDataReceived,
		OnConnectionTerminated: onConnectionTerminated,
		OnNewConnection: onNewConnection,
	}
	tcpServer = lib.NewServer(":3000", callbacks)
	producerCallbacks := messages.ProducerCallbacks{
		OnError: onProducerError,
	}
	f := false
	producer = messages.NewProducer(producerCallbacks,configuration.BrokersList,configuration.ProducerTopic,nil,nil,nil,&f)

	consumerCallbacks := messages.ConsumerCallbacks{
		OnDataReceived: onDataConsumed,
		OnError: onConsumerError,
	}
	consumer = messages.NewConsumer(consumerCallbacks,configuration.BrokersList,consumerGroupId,configuration.ConsumerTopics)
	consumer.Consume()

	go func(){
		http.HandleFunc("/", handler)
		http.ListenAndServe(":8080", nil)
	}()

	tcpServer.Listen()


}

func cleanup(){
	tcpServer.Close()
	producer.Close()
	consumer.Close()
	os.Exit(0)
}

Build, run and deploy to Docker image

To build:

go build main.go

To run:

go run main.go -config=config/config.yml

To build and run with Docker I first set this Dockerfile:

FROM debian
MAINTAINER "Orr Chen"
WORKDIR /app
ADD app/tcp-server.linux /app/
ADD config /app/
EXPOSE 8080 3000
CMD ["./tcp-server.linux","-config=config.yml"]

And build and push to my Docker repository with the build.sh script.

build.sh --tag <tag>  --name <name> --repository <repository>

Future improvements

Of course this is just a base framework, it lacks a few things mandatory for production environments which are mainly authentication, better logging, recovery from errors and input checking.
But I believe this might be a very useful start point for many developers who need this kind of a service, just like I needed it before implementing it :)
I will be very happy to read your thoughts and comments, happy holidays to all!

About the author:

Hi, my name is Orr Chen, a software engineer and a gopher for the past 3 years. My first experience with Go was migrating the entire backend of my startup PushApps from Rails to Golang. Since then I am a big fun of the language!
Github: OrrChen
Twitter: OrrChen
LinkedIn: orrchen

3rd parties libraries used:

2017-12-30

Table of contents

Introduction

Go 1.10 is the first major release after the announcement of the plans towards Go 2.0 at GopherCon 2017.

There are a number of exciting changes which I’ll cover below as well as some changes in the behavior of either tools or Go APIs which might result in an unexpected behavior compared to the previous change. I chose to flag these changes as “breaking change” in order to make it easier to identify them.

I’ve also tried to flag the CL that brought in the change as the discussions on them as well as the related changes in other CLs are a great source from learning how Go is organized, created, how features are reviewed, and hopefully inspire you to contribute to the language itself by either participating in reviews or issues or fixing issues (like Needs Investigation or Help Wanted).


Language changes

Let’s start with the language changes that Go 1.10 brings. There are only a couple of rather small changes, none of them significant.

First, you’ll be able to use an untyped constant as the index of an expression x[1.0 << s] where s is the untyped constant CL 60230

The second change is that you can now use method expressions like this struct{io.Reader}.Read, even if this is a rather unusual way to do so CL 73233


Operating systems support

Moving on to the operating system support, Go 1.10 will be the last Go version to run on OS X 10.8 Mountain Lion, OS X 10.9 Mavericks, or on OpenBSD 6.0.

FreeBSD 10.3 is now required to run Go, up from FreeBSD 9.3 CL 64910

NetBSD is once again supported, but only in the unreleased version 8 and on 386 and amd64. arm support for NetBSD 8 was still broken at the time of writing, see Issue 23073

On 32-bit MIPS systems you can now choose if you want emulation for floating point instructions or not via a new environment variable settings GOMIPS=hardfloat (the default) and GOMIPS=softfloat CL 37954


Tooling

The bigger changes in Go 1.10 come from the tooling improvements it brings. They dramatically improve the quality of life for testing in large and very large projects and pave the road towards Go 2.0.


Environment variables

Since Go 1.9,GOROOT is inferred from the location of the go tool binary by default. However, if your application relied on that value at runtime via runtime.GOROOT() there was a bug which prevented it have the correct location. As of Go 1.10+ this bug was fixed and you can now use it as expected CL 61310

A couple of new environment variables were added to the go command. GOTMPDIR will allow you to configure where the temporary files created by Go during compilation of your applications are stored. The default path is the same as in the previous Go versions, the operating system’s temporary files directory. The other new environment variable is GOCACHE allows you to control where the go command will store the cached information that’s reused in future builds CL 75475


go build

go build can now detect changes in files on a source code level rather than rely on timestamps, which means that it will be more accurate in rebuilding the packages that have changed. In turn, this means that you will now be able to drop the usage of -a flag, which was previously used to force Go to rebuild packages in certain conditions.

Changes are coming to the -asmflags, -gcflags, -gccgoflags, and -ldflags flags. They will not be applied automatically to the list of all packages as before but only to the direct package that’s being specified in the build command. You can still achieve the same functionality as before using the new special syntax for these flags, -ldflags=pattern=flags such as:
go build -ldflags=cmd/gofmt=-X=main.version=1.2.3 cmd/.... This command allows you to build all the /cmd/... packages but it will apply the -ldflags=-X=main.version=1.2.3 flag only to the cmd/gofmt package CL 76551

To further speed up the builds, go build -i will not be necessary since now the build tool will have its own cache for build steps that do not do the install step, such as go install or go get. This means that you’ll be able to switch between branches or experiment a lot more with the code without having to invoke go install or go build -i but just go build.

Are you a Windows user? Now you can use c-shared as a target for your libraries, thanks to CL 69091.


go install

go install also received some changes. Now it will install only the explicitly mentioned packages but not their dependencies. To restore the previous behaviour, you’ll need to use this command as go install -i CL 75850

This change as well as upcoming changes is significant if your tools depend on the packages to be installed in $GOPATH/pkg and always be fresh, with additional changes being required in order to restore the old behaviour.


go test

Speaking of caching, go test has seen a lot of changes as well. One of the most important is that go test will now cache the results of the tests if they meet certain criteria such as:

  • the test executable and command line match a previous run
  • the files and environment variables used by that run have not changed
  • the results are successful
  • the go test command received a list of packages to test, go test ./... for example
  • the test command line uses a subset of the test flags, -cpu, -list, -parallel, -run, -short, and -v

When the above conditions are met, the first run will produce the output as expected then subsequent runs will simply reuse that output. The run time of the tests will also notify that the cached output is used, by displaying (cached) instead of the original test run time.

You can always force the tests to run by specifying the flag -count=1, this being considered the idiomatic way to handle this requirement. As a recap, the -count flag allows you to specify how many times a test or a benchmark runs.

This change is covered in CL 75631.

The second important change to go test is that now a subset of go vet checks will run before the tests run in order to detect issues with your code. These checks will be treated as build failures in case any of them will produce any result. Only very high accuracy checks are included in this step. To disable this new behavior you’ll need to provide the -vet=off flag to the go test command CL 74356

The coverage profile of tests can now be created when running the tests against multiple packages, which was a highly requested feature. Combined with the new way to use the -coverpkg flag, it means you’ll be able to get the coverage for all the packages tested packages as well as their dependencies when multiple packages are being tested by running go test -coverpkg=all -coverprofile cover.out ./... CL 76875 and CL 76876

Test binaries will now always write to stdout when invoked via go test whereas before stderr could have been used sometimes CL 76871

Tests running in parallel will now be better delimited by having PAUSE and CONT as status update lines when running with -v flag. This change allows tooling to better interpret the start and stop of parallel tests. The -failfast flag now will stop testing immediately on the first failure, with the caveat that parallel tests are still allowed to continue until they finish CL 74450

Finally, go test -json will now output the format of the tests in json so that tooling such as IDEs can better present the results of the test. There is also a new command, go tool test2json that will produce convert the test output to json CL 76873 and CL 76872


gofmt

Another set of tooling changes in Go 1.10 come from gofmt as it received a few updates. First, three index slice expressions containing complex expressions are now always formatted as slice[start+1 : stop : capacity] CL 67633

The second change is that single-method interface literals written on a single line, which are sometimes used in type assertions, are no longer split onto multiple lines CL 66130

The third one is that if a composite literal would include a comment and only comments, then the comment(s) will now be indented CL 74232

If you use gofmt in your CI environment, you will see some failures because of these changes. The official position is that gofmt is not covered by the same set of compatibility promises as Go 1 itself, so these are not “breaking changes” but it’s rather a constant evolving specification, which can suffer changes on each new Go release. The recommendation is not to have gofmt enforced in the CI or have everyone use the same binary version for the application that formats your source code as well as checks it in the CI system.

A good news is that now all flags supported by gofmt are supported by go fmt as well.


go fix

go fix now replaces imports from golang.org/x/net/context with context which will help you migrate your code to a Go 1.9+ compatible code by running go tool fix -r context your/package CL 58590


pprof

Go 1.10 will also bring an update to the pprof tool. This brings a host of improvements, among which an updated UI featuring a flame graph representation of the profiling data CL 75870


Runtime

Go’s runtime received a few updates, with the first one I’ll cover being done on how the LockOSThread and UnlockOSThread mechanism works. If before, in nested calls, UnlockOSThread would need to be called only once to unlock the thread, now it will need to be called as many times as LockOSThread was called CL 45752

You may have noticed the <autogenerated> frame (line) in the stack traces before. This is now hidden, unless a panic or other issue happens in it. This also means that if your code would call runtime.Caller with a certain number of skip frames, then this change will be a “breaking change” in its behaviour as the <autogenerated> frames will not be counted there either CL 45412

Another important change in the Go runtime is the introduction of soft and hard goals (limits) for garbage collection CL 59970

The soft limit is the current value of the GOGC while the hard limit is 10% higher than the soft limit. Heavy GC reliant applications (so far only benchmarks) shows that there’s an increase in the heap size of the application.


CGO support

CGO support has also received updates, with C typedefs such as typedef X Y now which means you’ll be able to use C.X and C.Y interchangeably in Go now, as if they would be Go aliases, type X = Y CL 62670

Another welcomed change when working with C and Go is that you can now pass Go strings directly to C. This is done by declaring a C function in a Go file with a parameter type of the special type name _GoString_. To access the string length, you’ll need to call size_t _GoStringLen(_GoString_ s) an in order to get the pointer to the string contents, you’ll need use const char *_GoStringPtr(_GoString_ s) CL 70890

Some C types that were previously mapped to a pointer type in Go are now mapped to uintptr type. A couple of these types are CFTypeRef in Darwin’s CoreFoundation framework and the jobject in Java’s JNI interface. You’ll need to initialize the values for the affected types with 0 instead of nil This is a breaking change but thankfully you can fix this quickly and automatically by running go tool fix -r cftype your/package or go tool fix -r jni your/package CL 66332 and CL 81876


Debugging

Debugging support has also been improved in the latest release which should make your debugging experience via Delve even better than before. And as a reminder, Delve is most likely integrated with your favorite code editor.


Assembly support

Assembly support got better as well, with a host of new instructions being added. Most important changes are under amd64 platform with 359 new instructions including the full AVX, AVX2, BMI, BMI2, F16C, FMA3, SSE2, SSE3, SSSE3, SSE4.1, and SSE4.2 extension sets.


Packages

Changes in the various standard library packages:

  • bufio - the new Reader.Size and Writer.Size and methods report the Reader or Writer’s underlying buffer size CL 75150

  • bytes - the Fields, FieldsFunc, Split, and SplitAfter each already returned slices pointing into the same underlying array as its input. Go 1.10 changes each of the returned subslices to have capacity equal to its length, so that appending to a subslice will not overwrite adjacent data in the original input. This is also a “breaking change” in the behavior of these functions and you might need to update your code.

  • crypto/tls - the TLS server now advertises support for SHA-512 signatures when using TLS 1.2. The server already supported the signatures, but some clients would not select them unless explicitly advertised CL 74950

  • crypto/x509 - leaf certificate validation now enforces the name constraints for all names contained in the certificate, not just the one name that a client has asked about. Extended key usage restrictions are similarly now checked all at once. As a result, after a certificate has been validated, now it can be trusted in its entirety. It is no longer necessary to revalidate the certificate for each additional name or key usage CL 62693

  • database/sql/driver - drivers that want to construct a sql.DB for their clients can now implement the Connector interface and call the new sql.OpenDB function, instead of needing to encode all configuration into a string passed to sql.Open. Drivers that want to parse the configuration string only once per sql.DB instead of once per sql.Conn, or that want access to each sql.Conn’s underlying context, can make their Driver implementations also implement DriverContext’s new OpenConnector method. Drivers that implement ExecerContext no longer need to implement Execer; similarly, drivers that implement QueryerContext no longer need to implement Queryer. Previously, even if the context-based interfaces were implemented they were ignored unless the non-context-based interfaces were also implemented. To allow drivers to better isolate different clients using a cached driver connection in succession, if a Conn implements the new SessionResetter interface, database/sql will now call ResetSession before reusing the Conn for a new client

  • encoding/json - the Decoder adds a new method DisallowUnknownFields that causes it to report inputs with unknown JSON fields as a decoding error. The default behavior has always been to discard unknown fields. CL 27231 Unmarshal can no longer decode into fields inside embedded pointers to unexported struct types, because it cannot initialize the unexported embedded pointer to point at fresh storage. Unmarshal now returns an error in this case. This means you may need to update your code or a “breaking change” will happen, which could be hidden if the code is not properly handling errors CL 76851

  • text/template and html/template - the new actions {{break}} and {{continue}} break out of the innermost {{range ...}} loop, like the corresponding Go statements CL 66410

  • math/rand - the new math/rand.Shuffle function and corresponding math/rand.*Rand.Shuffle method shuffle an input sequence CL 51891

  • math - the new functions Round and RoundToEven round their arguments to the nearest floating-point integer; Round rounds a half-integer to its larger integer neighbor (away from zero) while RoundToEven rounds a half-integer to its even integer neighbor CL 43652 and CL 61211

  • net - the Conn and Listener implementations in this package now guarantee that when Close returns, the underlying file descriptor has been closed. In earlier releases, if the Close stopped pending I/O in other goroutines, the closing of the file descriptor could happen in one of those goroutines shortly after Close returned. TCPListener and UnixListener now implement syscall.Conn, to allow setting options on the underlying file descriptor using syscall.RawConn.Control. The Conn implementations returned by Pipe now support setting read and write deadlines. The IPConn.ReadMsgIP, IPConn.WriteMsgIP, UDPConn.ReadMsgUDP, and UDPConn.WriteMsgUDP, methods are now implemented on Windows

  • net/http - on the client side, an HTTP proxy, most commonly configured by ProxyFromEnvironment, can now be specified as an https:// URL, meaning that the client connects to the proxy over HTTPS before issuing a standard, proxied HTTP request. Previously, HTTP proxy URLs were required to begin with http:// or socks5://. On the server side, FileServer and its single-file equivalent ServeFile now apply If-Range checks to HEAD requests. FileServer also now reports directory read failures to the Server’s ErrorLog. The content-serving handlers also now omit the Content-Type header when serving zero-length content. ResponseWriter’s WriteHeader method now panics if passed an invalid (non-3-digit) status code. Redirect now sets the Content-Type header before writing its HTTP response

  • net/url - ResolveReference now preserves multiple leading slashes in the target URL. Previously it rewrote multiple leading slashes to a single slash, which resulted in the http.Client following certain redirects incorrectly

  • os - File adds new methods SetDeadline, SetReadDeadline, and SetWriteDeadline that allow setting I/O deadlines when the underlying file descriptor supports non-blocking I/O operations CL 71770 The definition of these methods matches those in net.Conn. Also matching net.Conn, File’s Close method now guarantee that when Close returns, the underlying file descriptor has been closed

  • strings - a new type Builder is a replacement for bytes.Buffer for the use case of accumulating text into a string result. The Builder’s API is a restricted subset of bytes.Buffer’s that allows it to safely avoid making a duplicate copy of the data during the String method CL 74931

  • unicode - the unicode package and associated support throughout the system has been upgraded from version 9.0 to Unicode 10.0, which adds 8,518 new characters, including four new scripts, one new property, a Bitcoin currency symbol, and 56 new emoji

A lot more packages have received changes but I’ve tried to keep the list to a minimum. To view the full list of changes, you can read this the Draft Release Notes.


Closing notes

Due to the vast amount of changes both in compiler and in runtime, some workloads are expected to perform better as of Go 1.10. However, I highly recommend that you grab the latest Go 1.10 release, 1.10 Beta 1 and test it on your workloads, run the tests against the new Go version and help the Go team identify issues before Go 1.10 lands in February. There are a few weeks where even just running the test / benchmark suite could make the difference. And Go 1.10 Beta 1 is also available as a Docker container so you can minimize the impact it has on your system.

If you want to stay up to date with the developments of Go, I recommend following @golang_cls Twitter account which provides a curated list of interesting commits as they are added to Go.

If you want to talk more about Go, its evolution, and what’s next for Go, tweet me, or meet me at Gophercon Iceland!

Finally, I would like to thank the Go Team and all contributors that helped Go reach 1.10 and I look forward to what the future holds.

A big thank you goes to Russ Cox and the team that created the initial draft documentation which this article uses / reuses a lot.

Errata:

The initial version of this article incorrectly mentioned a change about how GOROOT is handled. Thank you to Dominik Honnef for reporting this.

2017-12-29

One of the most hotly debated topics in the world of the Go programming language, is the lack of generics. Generics are considered a key feature in other statically typed programming languages like Java or C#. However, the makers of Go resisted the demand to add it to the language so far. A key reason … Continue reading "The empty interface in the Go programming language"

The post The empty interface in the Go programming language appeared first on Mina Andrawos.

2017-12-29

Create a Slack bot with golang

Introduction

In this post we’ll look at how to set up a quick Slack bot that receives messages (either direct or from channel) and replies to the user. I’ve been an IRC user for many years and always loved setting up bots, whether for sports scores, weather, or something else entirely. Recently I’ve actually had an opportunity to implement my first Slack bot and figured I would document the process for others! You can find all of the code for this post listed here, and PRs are certainly welcome :D

For this assignment we’ll need a few things, not all of which are covered in this post. I invite the reader to take a look at the installation practices for the other software dependencies based on their specific environment needs. Here I’ll be using Fedora 26 (4.14.6-200.fc26.x86_64) along with these tools:

  1. ngrok for Slack API replies – https://ngrok.com/docs#expose
  2. NHL statsapi to collect hockey scores – https://statsapi.web.nhl.com/api/v1/schedule
  3. the excellent golang slack library from nlopes – https://github.com/nlopes/slack

You’ll either need to set up an ngrok listener for your chosen localhost port, or develop on a server that it externally routable (e.g. DigitalOcean droplet). In my case here I’m developing on my laptop but would deploy permanently on a droplet.

The Slack API

Initial Configuration

The Slack API is well flushed out and spells out what specific payloads to anticipate for any particular object. There are a number of calls you can develop your bot to address, but in our case here we’ll look at using the Real Time Messaging API (RTM) and specifically the chat.postMessage and chat.postEphemeral methods.

Before any of our code is working we’ll need to set up an app within slack itself. Navigate to the app registration tool to create a new application within your workspace. Here I’ve created the NHL Scores app within my workspace.

Create App

Once done you’ll be presented with a number of options for your new application. Here we’ll need to create a Bot User that will act as our listener within the workspace. My example is called nhlslackbot and will be visible to all users within the workspace once mounted.

Bot User

We’ll need to generate an OAuth token for our user in order to actually connect with the Slack API. To do so click on the OAuth & Permissions section to Install App to Workspace which will prompt you to authorize access and generate the tokens you’ll use. You’ll need to copy the Bot User OAuth Access Token somewhere local, but always make sure this is not shared anywhere! This token is secret and should be treated like your password!

Authorize

Lastly we’ll need to set up the Interative Components of our application and specify the ngrok (or other) endpoint that the API will send responses to. In my case, I’ve added a custom ngrok value here called https://sebtest.ngrok.io/. This endpoint is where we’ll receive all correspondence from Slack itself, and this is how we’ll be able to process any incoming messages from the channels.

Interactive

With that all sorted, we can finally dig into the code!

Code components

The crux of the code is how we handle receiving messages from the slack connection. Using the Bot User OAuth Access Token to establish the initial connection, we must continuously poll the system for incoming messages. The API gives us the ability to trigger off of a number of event types, such as:

  1. Hello Events
  2. Connected Events
  3. Presence Change Events
  4. Message Events
  5. and many more

The beauty of this verbosity is that we can trigger messages on a number of different use-cases, really giving us the ability to tailor the bot to our specific needs. For this example, we’ll look at using the *slack.MessageEvent type to support both indirect (within channel using @) or direct messages. From the library, The primary poll for message events leverages the websocket handler and just loops over events until we’ve received one that we want:

func (s *Slack) run(ctx context.Context) {
    slack.SetLogger(s.Logger)

    rtm := s.Client.NewRTM()
    go rtm.ManageConnection()

    s.Logger.Printf("[INFO]  now listening for incoming messages...")
    for msg := range rtm.IncomingEvents {
        switch ev := msg.Data.(type) {
        case *slack.MessageEvent:
            if len(ev.User) == 0 {
                continue
            }

            // check if we have a DM, or standard channel post
            direct := strings.HasPrefix(ev.Msg.Channel, "D")

            if !direct && !strings.Contains(ev.Msg.Text, "@"+s.UserID) {
                // msg not for us!
                continue
            }

            user, err := s.Client.GetUserInfo(ev.User)
            if err != nil {
                s.Logger.Printf("[WARN]  could not grab user information: %s", ev.User)
                continue
            }

            s.Logger.Printf("[DEBUG] received message from %s (%s)\n", user.Profile.RealName, ev.User)

            err = s.askIntent(ev)
            if err != nil {
                s.Logger.Printf("[ERROR] posting ephemeral reply to user (%s): %+v\n", ev.User, err)
            }
        case *slack.RTMError:
            s.Logger.Printf("[ERROR] %s\n", ev.Error())
        }
    }
}

Once we confirm that the message is indeed directed to us, we pass the event handler along to our askIntent function. Remember that this is a contrived example that’s just going to send back NHL game scores to the user, iff they acknowledge that specific intent. We could build up an entire workflow around this user interaction that would send different paths depending on user choices to our prompts, or have no prompts at all! Those different cases are outside the scope of this introductory post, so for now we just want to send back a quick Yes v No prompt and handle accordingly.

To do precisely that, our handler askIntent will process the message and genreate an chat.postEphemeral message to send back to the event user (aka the person asking for details). The “ephemeral” post is one that’s directed only to the requester. Though other users will see the initial request to the bot if within the same channel, the subsequent interaction with the bot will only be done between the user and the bot. From the docs:

This method posts an ephemeral message, which is visible only to the assigned user in a specific public channel, private channel, or private conversation.

With that in mind, we set up the initial response payload using the attachments spec from the API, defining a set of actions that the user is able to choose. For this part of the conversation the user must reply Yes or No for whether they’d like us to retrieve the most recent scores. If No, we reply with a basic note and continue listening; if Yes then let’s retrieve the scores!

// askIntent is the initial request back to user if they'd like to see
// the scores from the most recent slate of games
//
// NOTE: This is a contrived example of the functionality, but ideally here
// we would ask users to specify a date, or maybe a team, or even
// a specific game which we could present back
func (s *Slack) askIntent(ev *slack.MessageEvent) error {
    params := slack.NewPostEphemeralParameters()
    attachment := slack.Attachment{
        Text:       "Would you like to see the most recent scores?",
        CallbackID: fmt.Sprintf("ask_%s", ev.User),
        Color:      "#666666",
        Actions: []slack.AttachmentAction{
            slack.AttachmentAction{
                Name:  "action",
                Text:  "No thanks!",
                Type:  "button",
                Value: "no",
            },
            slack.AttachmentAction{
                Name:  "action",
                Text:  "Yes, please!",
                Type:  "button",
                Value: "yes",
            },
        },
    }

    params.Attachments = []slack.Attachment{attachment}
    params.User = ev.User
    params.AsUser = true

    _, err := s.Client.PostEphemeral(
        ev.Channel,
        ev.User,
        slack.MsgOptionAttachments(params.Attachments...),
        slack.MsgOptionPostEphemeralParameters(params),
    )
    if err != nil {
        return err
    }

    return nil
}

The attachments in the snippet above present the user with the following dialog:

Options

If the user selects No, thanks! then we reply with a basic message:

Choose No

This part of the interaction is precisely where the ngrok endpoint comes into play. The user’s interaction is not directly with our code, but instead with slack itself. The message and interaction is passed through slack and on to us at the redirect URL we specified earlier, in my case https://sebtest.ngrok.io which routes to our internal localhost:9191 interface, and from there to our postHandler as defined in our webapp router.

The tricky part here is to process the payload portion of the JSON response from the API. The POST that slack returns back to our URL is a payload form that contains a bevy of information for our interaction. In this case, the user’s response (either Yes or No) as well as a callbackID which we actually passed in our original mesage prompt to the user! This is incredibly useful, especially as you have more and more users interacting with your bot as you can specify unique actions based on the trigger. For example, if the user selects Yes we could send subsequent ephemeral messages to ask for a specific date, or maybe a certain team? We could even define the callback value as a key to a function map that would then trigger some kind of other workflow altogether (like posting to a blog resource, or checking DB credentials, etc). The options are indeed endless, but for the scope of this contrived example we just stick to the scores from last night.

func postHandler(w http.ResponseWriter, r *http.Request) {
    if r.URL.Path != "/" {
        w.WriteHeader(http.StatusNotFound)
        w.Write([]byte(fmt.Sprintf("incorrect path: %s", r.URL.Path)))
        return
    }

    if r.Body == nil {
        w.WriteHeader(http.StatusNotAcceptable)
        w.Write([]byte("empty body"))
        return
    }
    defer r.Body.Close()

    err := r.ParseForm()
    if err != nil {
        w.WriteHeader(http.StatusGone)
        w.Write([]byte("could not parse body"))
        return
    }

    // slack API calls the data POST a 'payload'
    reply := r.PostFormValue("payload")
    if len(reply) == 0 {
        w.WriteHeader(http.StatusNoContent)
        w.Write([]byte("could not find payload"))
        return
    }

    var payload slack.AttachmentActionCallback
    err = json.NewDecoder(strings.NewReader(reply)).Decode(&payload)
    if err != nil {
        w.WriteHeader(http.StatusGone)
        w.Write([]byte("could not process payload"))
        return
    }

    action := payload.Actions[0].Value
    switch action {
    case "yes":
        grabStats(w, r)
    case "no":
        w.Write([]byte("No worries, let me know later on if you do!"))
    default:
        w.WriteHeader(http.StatusNotAcceptable)
        w.Write([]byte(fmt.Sprintf("could not process callback: %s", action)))
        return
    }

    w.WriteHeader(http.StatusOK)
}

A key component to note here is the http response code; if you do not specify the http.StatusOK value in your prompt back to the API, the error message you may want to convey to the user gets eaten by the system. The default slackbot will absorb that message and reply to you (with an ephemeral message no less) with the status code, but not the messages. Long story short, whatever message you’d like to actually send back to the requester should have an http.StatusOK header.

Lastly, if our user has selected the Yes option we call out to our NHL stats api and process the results for the user!

// grabStats will process the information from the API and return the data to
// our user!
func grabStats(w http.ResponseWriter, r *http.Request) {
    n := fetch.New()

    buf, err := n.GetSchedule()
    if err != nil {
        w.WriteHeader(http.StatusNoContent)
        w.Write([]byte(fmt.Sprintf("error processing schedule; %v", err)))
        return
    }

    w.WriteHeader(http.StatusOK)
    w.Write(buf)
}

// GetSchedule calls out to the NHL API listed at APIURL
// and returns a formatted JSON blob of stats
//
// This function calls the 'schedule' endpoint which
// returns the most recent games by default
// TODO: add options to provide date range
func (n *NHL) GetSchedule() ([]byte, error) {
    var buf bytes.Buffer

    r, err := http.Get(fmt.Sprintf("%s/schedule", APIURL))
    if err != nil {
        return buf.Bytes(), err
    }
    defer r.Body.Close()

    err = json.NewDecoder(r.Body).Decode(&n.Schedule)
    if err != nil {
        return buf.Bytes(), fmt.Errorf("error parsing body: %+v", err)
    }

    for _, x := range n.Schedule.Dates {
        for idx, y := range x.Games {
            buf.WriteString(fmt.Sprintf("Game %d: %s\n", idx+1, y.Venue.Name))
            buf.WriteString(fmt.Sprintf("Home: %s -- %d\n", y.Teams.Home.Team.Name, y.Teams.Home.Score))
            buf.WriteString(fmt.Sprintf("Away: %s -- %d\n\n", y.Teams.Away.Team.Name, y.Teams.Away.Score))
        }
    }

    return buf.Bytes(), nil
}

Sample output below…

Sample Output

Congratulations, you’ve now delivered an ephemeral payload to your slack user’s request!


About The Author

Sebastian Borza is a golang, C, and python developer based in Chicago, IL.

Source Handle
freenode sborza
efnet sebito91
github sebito91
twitter @sebito91
keybase sborza
GPG E4110D3E
2017-12-28

This year I helped organize several online security challenges, one of which is Blacklight. Among the things I was asked to do, was creating a POC for a specific challenge, to prove that it’s possible to solve in a reasonable time. That challenge was one I face occasionally in my everyday life, not always with success: break a captcha.

The task that requires breaking the captcha is disabling a security camera, to break into a room, without the security camera capturing your face. Here is how it looked before:

A frame from the camera's capture

The provided information was the saved model used for captcha recognition in the binary ProtoBuf format, and a link to the camera control panel.

An input of a TensorFlow model requires doing some TensorFlow!

A Few words about TensorFlow

TensorFlow is an open-source software for Machine Intelligence, used mainly for machine learning applications such as neural networks.

TensorFlow runs computations involving tensors, and there are many sources to understand what a Tensor is. This article is definitely not a sufficient one, and it only holds the bare minimum to make sense of what the code does. Tensors are awesome and complex mathematical objects, and I encourage you to take the time to learn more about them.

For our purposes, here is the explanation from the TensorFlow website:

A tensor is a generalization of vectors and matrices to potentially higher dimensions. Internally, TensorFlow represents tensors as n-dimensional arrays of base datatypes.

A tensor is defined by the data type of the value(s) it holds, and its shape, which is the number of dimensions, and number of values per dimension.

The flow part in TensorFlow comes to describe that essentially the graph (model) is a set of nodes (operations), and the data (tensors) “flow” through those nodes, undergoing mathematical manipulation. You can look at, and evaluate, any node of the graph.

A Few words about TensorFlow+Go

On the official TensorFlow website, you can find a page dedicated to Go, where it says “TensorFlow provides APIs that are particularly well-suited to loading models created in Python and executing them within a Go application.” It also warns that the TensorFlow Go API is not covered by the TensorFlow API stability guarantees. To the date of this post, it is still working as expected.

When going to the package page, there are 2 warnings: 1) The API defined in this package is not stable and can change without notice. 2) The package path is awkward: github.com/tensorflow/tensorflow/tensorflow/go.

In theory, the Go APIs for TensorFlow are powerful enough to do anything you can do from the Python APIs, including training. Here is an example of training a model in Go using a graph written in Python. In practice, some of tasks, particularly those for model construction are very low level and certainly not as convenient as doing them in Python. For now, it generally makes sense to define the model in TensorFlow for Python, export it, and then use the Go APIs for inference or training that model.[1] So while Go might not be your first choice for working with TensorFlow, they do play nice together when using existing models.

Let’s break into this page

The parts of the page I was facing seemed pretty familiar to your regular captcha-protected form:

  • PIN Code - brute-force
  • Captcha - use the model

So my TO DOs were:

  • 1. Build a captcha reader
  • 2. While not logged in:
    • 2.1 Generate the next PIN code
    • 2.2 Get captcha text for current captcha image
    • 2.3 Try to log in

SavedModel CLI

From the website: SavedModel is the universal serialization format for TensorFlow models.

So our first step would be figuring out the input and output nodes of the prediction workflow. SavedModel CLI is an inspector for doing this. Here’s the command and its output:

$ saved_model_cli show --dir <PATH> --all


MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['serving_default']:
The given SavedModel SignatureDef contains the following input(s):
inputs['input'] tensor_info:
    dtype: DT_STRING
    shape: unknown_rank
    name: CAPTCHA/input_image_as_bytes:0
The given SavedModel SignatureDef contains the following output(s):
outputs['output'] tensor_info:
    dtype: DT_STRING
    shape: unknown_rank
    name: CAPTCHA/prediction:0
Method name is: tensorflow/serving/predict

What we learn from this are the node names.

Input node: CAPTCHA/input_image_as_bytes,

Output node: CAPTCHA/prediction.

Captcha

Now let’s load the model, using func LoadSavedModel(exportDir string, tags []string, options *SessionOptions) (*SavedModel, error). The function takes 3 arguments: path, tags and seesion options. Explaining tags and options can easily take the entire post and will shift the focus, so for our purpose I used the convention {"serve"}, and provided no session options.

	savedModel, err := tf.LoadSavedModel("./tensorflow_savedmodel_captcha", []string{"serve"}, nil)
	if err != nil {
		log.Println("failed to load model", err)
		return
	}

Then get the captcha from the web page, and run it through the model. First, define the output of an operation in the graph (model+node) and its index.

	feedsOutput := tf.Output{
		Op:    savedModel.Graph.Operation("CAPTCHA/input_image_as_bytes"),
		Index: 0,
	}

Create a new tensor. The input can be a scalar, slices, or array. As we want to predict a captcha, we’ll need 1 dimension with 1 element, of type string.

	feedsTensor, err := tf.NewTensor(string(buf.String()))
	if err != nil {
		log.Fatal(err)
	}

Set a map from the operation we will apply to the input it will be applied on.

	feeds := map[tf.Output]*tf.Tensor{feedsOutput: feedsTensor}

Get the output from the prediction operation into this output struct.

	fetches := []tf.Output{
		{
			Op:    savedModel.Graph.Operation("CAPTCHA/prediction"),
			Index: 0,
		},
	}

Run the data through the graph and receive the output - the captcha prediction.

	captchaText, err := savedModel.Session.Run(feeds, fetches, nil)
	if err != nil {
		log.Fatal(err)
	}
	captchaString := captchaText[0].Value().(string)

Here is how this looks like:

The captcha screenshot

Generate a PIN code

The PIN code is made of 4 digits, so we’ll go over all the combinations. Additionally, in each iteration the saved model is required for the prediction operation, and of course some logs.

for x := 0; x < 10000; x++ {
		logIntoSite(fmt.Sprintf("%0.4d", x), savedModel, *printLogs)
	}

Try to log in

Once all values are there - the current value of the PIN code in the loop and the captcha prediction - let’s POST that request to the login page.

	params := url.Values{}
	params.Set("pin", pinAttempt)
	params.Set("captcha", captchaString)

	res, err := client.PostForm(string(siteUrl+"/disable"), params)
	if err != nil {
		log.Fatal(err)
	}

	defer res.Body.Close()
	buf = new(bytes.Buffer)
	buf.ReadFrom(res.Body)
	response := buf.String()

If the captcha prediction failed, run the prediction again, and retry with the same PIN code.

	if parseResponse(response, pinAttempt, captchaString, printLogs) == badCaptcha {
		logIntoSite(savedModel, printLogs)
	}

The parseResponse function checks and reports whether the website response is a success or one of the failure messages, which I found by manually trying combinations of guessing a PIN code and correct and wrong captcha translations.

func parseResponse(response, pinAttempt, captchaString string, printLogs bool) string {
	message := "something happened"
	if strings.Contains(response, badPIN) {
		message = badPIN
	} else if strings.Contains(response, badCaptcha) {
		message = badCaptcha
	}

	logResponse(printLogs, message, pinAttempt, captchaString, response)
	return message
}

The rest of the code

To complete this code, let’s add everyones favorites: cookies and logging. Generating the captcha starts a new session, and in order to use the predicted captcha in the same session, we will open a cookie jar. Even though it’s the first time I am writing about cookies publicly, I will spare cookie jokes, as part of the Christmas spirit.

	jar, err := cookiejar.New(nil)
	if err != nil {
		log.Fatal(err)
	}
	client := &http.Client{
		Jar: jar,
	}

And here is how it looks when it’s all composed together.

To wrap this up

TensorFlow has many great models which can be used with Go. Here is a great list of those.

Online challenges can be an awesome way to learn, whether it’s coding, security or sports. The combination of putting in practice your knowledge and having a mission creates a fun environment where you can work on improving your skills. Consider joining such a challenge as your new year’s resolution.

Thanks a lot to Ed for reviewing this PR. Also thanks to Asim Ahankar from the TensorFlow team for pointing out it is possible to train models with Go, as updated in [1]. We will collaborate further to make the documentation around this more accessible.

If you want to chat more about this, tweet me, or meet me at Gophercon Iceland!

2017-12-27
It is real struggle to work with a new language, especially if the type doesn’t resemble what you have previously seen. I have been there with Go and lost my interest in the language when it first came out due to the reason I was pretending it is something I already knew. Go is considered as an object-oriented language even though it lacks type hierarchy. It has an unconventional type system.
2017-12-27

Go templates are a powerful method to customize output however you want, whether you’re creating a web page, sending an e-mail, working with Buffalo, Go-Hugo, or just using some CLI such as kubectl.

There’re two packages operating with templates — text/template and html/template. Both provide the same interface, however the html/template package is used to generate HTML output safe against code injection.

In this article we’re going to take a quick look on how to use the package, as well as how to integrate them with your application.

Actions

Before we learn how to implement it, let’s take a look at template’s syntax. Templates are provided to the appropriate functions either as string or as “raw string”. Actions represents the data evaluations, functions or control loops. They’re delimited by {{ }}. Other, non delimited parts are left untouched.

Data evaluations

Usually, when using templates, you’ll bind them to some data structure (e.g. struct) from which you’ll obtain data. To obtain data from a struct, you can use the {{ .FieldName }} action, which will replace it with FieldName value of given struct, on parse time. The struct is given to the Execute function, which we’ll cover later.

There’s also the {{.}} action that you can use to refer to a value of non-struct types.

Conditions

You can also use if loops in templates. For example, you can check if FieldName non-empty, and if it is, print its value: {{if .FieldName}} Value of FieldName is {{ .FieldName }} {{end}}.

else and else if are also supported: {{if .FieldName}} // action {{ else }} // action 2 {{ end }}.

Loops

Using the range action you can loop through a slice. A range actions is defined using the {{range .Member}} ... {{end}} template.

If your slice is a non-struct type, you can refer to the value using the {{ . }} action. In case of structs, you can refer to the value using the {{ .Member }} action, as already explained.

Functions, Pipelines and Variables

Actions have several built-in functions that’re used along with pipelines to additionally parse output. Pipelines are annotated with | and default behavior is sending data from left side to the function on right side.

Functions are used to escape the action’s result. There’re several functions available by default such as, html which returns HTML escaped output, safe against code injection or js which returns JavaScript escaped output.

Using the with action, you can define variables that’re available in that with block: {{ with $x := <^>result-of-some-action<^> }} {{ $x }} {{ end }}.

Throughput the article, we’re going to cover more complex actions, such as reading from an array instead of struct.

Parsing Templates

The three most important and most frequently used functions are:

  • New — allocates new, undefined template,
  • Parse — parses given template string and return parsed template,
  • Execute — applies parsed template to the data structure and writes result to the given writer.

The following code shows above-mentioned functions in the action:

package main

import (
	"os"
	"text/template"
)

type Todo struct {
	Name        string
	Description string
}

func main() {
	td := Todo{"Test templates", "Let's test a template to see the magic."}

  t, err := template.New("todos").Parse("You have a task named \"{{ .Name}}\" with description: \"{{ .Description}}\"")
	if err != nil {
		panic(err)
	}
	err = t.Execute(os.Stdout, td)
	if err != nil {
		panic(err)
	}
}

The result is the following message printed in your terminal:

You have a task named "Test templates" with description: "Let's test a template to see the magic."

You can reuse the same template, without needing to create or parse it again by providing the struct you want to use to the Execute function again:

// code omitted beacuse of brevity
...

tdNew := Todo{"Go", "Contribute to any Go project"}
err = t.Execute(os.Stdout, tdNew)
}

The result is like the previous one, just with new data:

You have a task named "Go" with description: "Contribute to any Go project"

As you can see, templates provide a powerful way to customize textual output. Beside manipulating textual output, you can also manipulate HTML output using the html/template package.

Verifying Templates

template packages provide the Must functions, used to verify that a template is valid during parsing. The Must function provides the same result as if we manually checked for the error, like in the previous example.

This approach saves you typing, but if you encounter an error, your application will panic. For advanced error handling, it’s easier to use above solution instead of Must function.

The Must function takes a template and error as arguments. It’s common to provide New function as an argument to it:

t := template.Must(template.New("todos").Parse("You have task named \"{{ .Name}}\" with description: \"{{ .Description}}\""))

Throughput the article we’re going to use this function so we can omit explicit error checking.

Once we know how what Template interface provides, we can use it in our application. Next section of the article will cover some practical use cases, such as creating web pages, sending e-mails or implementing it with your CLI.

Implementing Templates

In this part of the article we’re going to take a look how can you use the magic of templates. Let’s create a simple HTML page, containing an to-do list.

Creating Web Pages using Templates

The html/template package allows you to provide template file, e.g. in the form of an HTML file, to make implementing both the front-end and back-end easier.

The following data structure represents a To-Do list. The root element has the user’s name and list, which is represented as an array of struct containing the tasks’ name and status.

type entry struct {
  Name string
  Done bool
}

type ToDo struct {
  User string
  List []entry
}

This simple HTML page will be used to display user’s name and its To-Do list. For this example, we’re going to use range action to loop through tasks slice, with action to easier get data from slice and an condition checking is task already done. In case task is done, Yes will be written in the appropriate field, otherwise No will be written.

<!DOCTYPE html>
<html>
  <head>
    <title>Go To-Do list</title>
  </head>
  <body>
    <p>
      To-Do list for user: {{ .User }} 
    </p>
    <table>
      	<tr>
          <td>Task</td>
          <td>Done</td>
    	</tr>
      	{{ with .List }}
			{{ range . }}
      			<tr>
              		<td>{{ .Name }}</td>
              		<td>{{ if .Done }}Yes{{ else }}No{{ end }}</td>
      			</tr>
			{{ end }} 
      	{{ end }}
    </table>
  </body>
</html>

Just like earlier, we’re going to parse the template and then apply it to the struct containing our data. Instead of the Parse function, the ParseFile is going to be used. Also, for code brevity, we’ll write parsed data to standard output (your terminal) instead to an HTTP Writer.

package main

import (
	"html/template"
	"os"
)

type entry struct {
	Name string
	Done bool
}

type ToDo struct {
	User string
	List []entry
}

func main() {
	// Parse data -- omitted for brevity

	// Files are provided as a slice of strings.
	paths := []string{
		"todo.tmpl",
	}

    t := template.Must(template.New("html-tmpl").ParseFiles(paths...))
	err = t.Execute(os.Stdout, todos)
	if err != nil {
		panic(err)
	}
}

This time, we’re using html/template instead of text/template, but as they provide the same interface, we’re using the same functions to parse the template. That output would be same even if you used text/template, but this output is safe against code injection.

This code generates code such as the below one:

<!DOCTYPE html>
<html>
  <head>
    <title>Go To-Do list</title>
  </head>
  <body>
    <p>
      To-Do list for user: gopher 
    </p>
    <table>
      	<tr>
          <td>Task</td>
          <td>Done</td>
    	</tr>
      			<tr>
              		<td>GopherAcademy Article</td>
              		<td>Yes</td>
      			</tr>			
      			<tr>
              		<td>Merge PRs</td>
              		<td>No</td>
      			</tr>
    </table>
  </body>
</html>
Parsing Multiple Files

Sometimes, this approach is not suitable if you have many files, or you’re dynamically adding new ones and removing old ones.

Beside the ParseFiles function, there’s also the ParseGlob function which takes glob as an argument and than parses all files that matches the glob.

// ...
t := template.Must(template.New("html-tmpl").ParseGlob("*.tmpl"))
err = t.Execute(os.Stdout, todos)
if err != nil {
	panic(err)
}
// ...
Use cases
  • You can use this approach to generate a web page that obtains data using Go API.
  • You can generate and send e-mails.
  • You can create wonderful web sites using Go Hugo templating.

Customizing Command’s Output

You can incorporate templates in your CLI allowing users to customize command’s output. This is commonly done by providing flags for inputing templates. For example, Kubernetes CLI—kubectl provides two flags, --template for providing a template as a string and --template-file used to provide a template in the form of a file.

The following snippet parses the template provided via one of the two flags—template and template-file.

package main

import (
	"flag"
	"os"
)

func main() {
  	// data parsing...
  
	var template, templateFile string
	flag.StringVar(&template, "template", "", "a template")
	flag.StringVar(&templateFile, "template-file", "", "a template file path")
	flag.Parse()

	if templateFile != "" {
		path := []string{templateFile}
		t := template.Must(template.New("html-tmpl").ParseFiles(path...))
		err = t.Execute(os.Stdout, todos)
		if err != nil {
			panic(err)
		}
	} else if template != "" {
		path := []string{templateFile}
		t := template.Must(template.New("html-tmpl").Parse(template))
		err = t.Execute(os.Stdout, todos)
		if err != nil {
			panic(err)
		}
	} else {
		// non-template data logic...
	}

}

Similar could be done using spf13/cobra. This code snippet omits data parsing logic because of brevity. Users can use this to customize output using intuitive template language, without need to use tools such as sed, awk or grep.

Conclusion

In this article we showed how to use basic templating functions to manipulate data along with several use cases. This article is meant to be a quick reference to Go’s template packages. You can also check out the official text/template and html/template if you’re interested in more complex use cases.

If you have any questions, feel free to contact me! You can find me as xmudrii on Gophers Slack, Twitter and GitHub.