2018-03-23
Introduction I’ve been seeing a lot of question about interfaces lately on Slack. Most of the time the answers are technical and focus on implementation details. Implementation is important to help with debugging, but implementation doesn’t help with design. When it comes to designing code with interfaces, behavior has to be the main focus. In this post, I hope to provide a different way to think about interfaces and how to design code with them.
2018-02-20
Introduction I was guided for many years to write functions that are generalized and to create layers upon layers of abstraction so things don’t break as business requirements change. That the cost of breaking a function signature, for example, is expensive and something that should be avoided. Therefore, write functions that take more generic parameters or hide things in a receiver or context to be less prone to breakage.
2018-01-22
Prelude It will be helpful to read this four-part series first on escape analysis and data semantics. Details on how to read an escape analysis report and pprof output have been outlined here. https://www.ardanlabs.com/blog/2017/05/language-mechanics-on-stacks-and-pointers.html Introduction Even after working with Go for 4 years, I am continually amazed by the language. Thanks to the static code analysis the compiler performs, the compiler can apply interesting optimizations to the code it produces.
2018-01-16
What does a distro provide? The most popular docker base container image is either busybox, or scratch. This is driven by a movement that is equal parts puritanical and pragmatic. The puritan asks “Why do I need to run init(1) just to run my process?” The pragmatist asks “Why do I need a 700 meg […]
2018-01-08
This is an article about compiler directives; or as they are commonly known, pragmas. It’s derived from a talk of a similar name that I gave last year at GopherChina in Shanghai. But first, a history lesson Before we talk about Go, let’s talk a little about pragmas, and their history. Many languages have the notion […]
2017-12-31

System diagram
I needed to create a simple framework to provide my endpoint devices ( doesn’t matter which platform they run on ) the option to send and receive messages from my backend.
I require those messages to be managed by a message broker so that they can be processed in an asynchronous way.
The system contains 4 main layers, this article section is mainly about the first one:
1. TCP servers - Needs to maintain as many TCP sockets in synch with the endpoints as possible. All of the endpoints messages will be processed on a different layer by the message broker. This keeps the TCP servers layer very thin and effective. I also want to keep as many concurrent connection as possible, and Go is a good choice for this ( see this article)
2. Message broker - Responsible for delivering the messages between the TCP servers layer and the workers layer. I chose Apache Kafka for that purpose.
3. Workers layer - will process the messages through services exposed in the backend layer.
4. Backed services layer - An encapsulation of services required by your application such as DB, Authentication, Logging, external APIs and more.

So, this Go Server:
1. communicates with its endpoint clients by TCP sockets.
2. queues the messages in the message broker.
3. receives back messages from the broker after they were processed and sends response acknowledgment and/or errors to the TCP clients.

The full source code is available in : https://github.com/orrchen/go-messaging
I have also included a Dockerfile and a build script to push the image to your Docker repository.
Special thanks to the great go Kafka sarama library from Shopify.

The article is divided to sections representing the components of the system. Each component should be decoupled from the others in a way that allows you to read about a single component in a straight forward manner.

TCP Client

Its role is to represent a TCP client communicating with our TCP server.

type Client struct {
	Uid  string /* client is responsible of generating a unique uid for each request,   
	it will be sent in the response from the server so that client will know what request generated this response */
	DeviceUid string /* a unique id generated from the client itself */
	conn net.Conn
	onConnectionEvent func(c *Client, eventType ConnectionEventType, e error) /* function for handling new connections */
	onDataEvent func(c *Client, data []byte) /* function for handling new date events */
}

Please notice that onConnectionEvent and onDataEvent are callbacks for the Struct that will obtain and manage Clients.

Our client will listen permanently using the listen() function and response to new connections, new data received and connections terminations.

Kafka Consumer

Its role is to consume messages from our Kafka broker, and to broadcast them back to relevant clients by their uids.
In this example we are consuming from multiple topics using the cluster implementation of sarama.

Let’s define our Consumer struct:

type Consumer struct {
	consumer *cluster.Consumer
	callbacks ConsumerCallbacks
}

The constructor receives the callbacks and relevant details to connect to the topic:

func NewConsumer(callbacks ConsumerCallbacks,brokerList []string, groupId string, topics []string) *Consumer {
	consumer := Consumer{callbacks:callbacks}

	config := cluster.NewConfig()
	config.ClientID = uuid.NewV4().String()
	config.Consumer.Offsets.Initial = sarama.OffsetNewest
	saramaConsumer, err := cluster.NewConsumer(brokerList, groupId, topics, config)
	if err != nil {
		panic(err)
	}
	consumer.consumer = saramaConsumer
	return &consumer

}

It will consume permanently on a new goroutine inside the Consume() function.
It reads from the Messages() channel for new messages and the Notifications() channel for events.

Kafka Producer

Its role is to produce messages to our Kafka broker.
In this example we are producing to a single topic.
This section is mainly inspired from the example in https://github.com/Shopify/sarama/blob/master/examples/http_server/http_server.go

Let’s define our Producer Struct:

type Producer struct {
	asyncProducer sarama.AsyncProducer
	callbacks     ProducerCallbacks
	topic         string
}

Producer is constructed with the callbacks for error, and the details to connect to the Kafka broker including optional ssl configurations that are created with createTLSConfiguration:

func NewProducer(callbacks ProducerCallbacks,brokerList []string,topic string,certFile *string,keyFile *string,caFile *string,verifySsl *bool ) *Producer {
	producer := Producer{ callbacks: callbacks, topic: topic}

	config := sarama.NewConfig()
	tlsConfig := createTLSConfiguration(certFile,keyFile,caFile,verifySsl)
	if tlsConfig != nil {
		config.Net.TLS.Enable = true
		config.Net.TLS.Config = tlsConfig
	}
	config.Producer.RequiredAcks = sarama.WaitForLocal       // Only wait for the leader to ack
	config.Producer.Compression = sarama.CompressionSnappy   // Compress messages
	config.Producer.Flush.Frequency = 500 * time.Millisecond // Flush batches every 500ms

	saramaProducer, err := sarama.NewAsyncProducer(brokerList, config)
	if err != nil {
		log.Fatalln("Failed to start Sarama producer:", err)
		panic(err)
	}
	go func() {
		for err := range saramaProducer.Errors() {
			if producer.callbacks.OnError!=nil {
				producer.callbacks.OnError(err)
			}
		}
	}()
	producer.asyncProducer = saramaProducer
	return &producer
}

I decided to produce messages that are encoded to JSON and to ensure it before sending them:

type message struct {
	value interface{}
	encoded []byte
	err     error
}

func (ale *message) ensureEncoded() {
	if ale.encoded == nil && ale.err == nil {
		ale.encoded, ale.err = json.Marshal(ale.value)
		if ale.err!=nil {
			log.Println(ale.err)
		}
	}
}

func (ale *message) Length() int {
	ale.ensureEncoded()
	return len(ale.encoded)
}

func (ale *message) Encode() ([]byte, error) {
	ale.ensureEncoded()
	return ale.encoded, ale.err
}

And finally, we provide the functions to produce the message and close the producer:

func (p *Producer) Produce(payload interface{}) {
	value := message{
		value: payload,
	}
	value.ensureEncoded()
	log.Println("producing: ", string(value.encoded))
	p.asyncProducer.Input() <- &sarama.ProducerMessage{
		Topic: p.topic,
		Value: &value,
	}
}  
func (p *Producer) Close() error{
	log.Println("Producer.Close()")
	if err := p.asyncProducer.Close(); err != nil {
		return err
	}
	return nil
}

TCP Server

Its role is to obtain and manage a set of Client, and send and receive messages from them.

type TcpServer struct {
	address                  string // Address to open connection: localhost:9999
	connLock sync.RWMutex
	connections map[string]*Client
	callbacks Callbacks
	listener net.Listener
}

It is constructed simply with an address to bind to and the callbacks to send:

// Creates new tcp Server instance
func NewServer(address string, callbacks Callbacks ) *TcpServer {
	log.Println("Creating Server with address", address)
	s := &TcpServer{
		address: address,
		callbacks: callbacks,
	}
	s.connections = make(map[string]*Client)
	return s
}

When a connection event occurs we process it and handle it, if it’s a new event we attach a new UID to the client.
If connection is terminated we delete this client.
In both cases we send the callbacks to notify about those events.

TcpServer will listen permanently for new connections and new data with Listen(), and support a graceful shutdown with Close().

We provide 2 options ot send data to our clients, by their device uid ( generated from the client side) with SendDataByDeviceUidor by the client id which is generated in our system with SendDataByClientId.

API

We need to create structs for the API that the tcp clients use, and the API for the messages sent to/from the messages broker.
For the TCP clients:
* DeviceRequest * DeviceResponse

For the message broker:
* ServerRequest
* ServerResponse

Main function - putting it all together

Obtains and manages all the other components in this system. It will include the TCP server that holds an array of TCP clients, and a connection to the Kafka broker for consuming and sending messages to it. Here are the main parts of main.go file:

var tcpServer *lib.TcpServer
var producer *messages.Producer
var consumer *messages.Consumer

func main() {
    callbacks := lib.Callbacks{
		OnDataReceived: onDataReceived,
		OnConnectionTerminated: onConnectionTerminated,
		OnNewConnection: onNewConnection,
	}
	tcpServer = lib.NewServer(":3000", callbacks)
	producerCallbacks := messages.ProducerCallbacks{
		OnError: onProducerError,
	}
	f := false
	producer = messages.NewProducer(producerCallbacks,configuration.BrokersList,configuration.ProducerTopic,nil,nil,nil,&f)

	consumerCallbacks := messages.ConsumerCallbacks{
		OnDataReceived: onDataConsumed,
		OnError: onConsumerError,
	}
	consumer = messages.NewConsumer(consumerCallbacks,configuration.BrokersList,consumerGroupId,configuration.ConsumerTopics)
	consumer.Consume()

	go func(){
		http.HandleFunc("/", handler)
		http.ListenAndServe(":8080", nil)
	}()

	tcpServer.Listen()


}

func cleanup(){
	tcpServer.Close()
	producer.Close()
	consumer.Close()
	os.Exit(0)
}

Build, run and deploy to Docker image

To build:

go build main.go

To run:

go run main.go -config=config/config.yml

To build and run with Docker I first set this Dockerfile:

FROM debian
MAINTAINER "Orr Chen"
WORKDIR /app
ADD app/tcp-server.linux /app/
ADD config /app/
EXPOSE 8080 3000
CMD ["./tcp-server.linux","-config=config.yml"]

And build and push to my Docker repository with the build.sh script.

build.sh --tag <tag>  --name <name> --repository <repository>

Future improvements

Of course this is just a base framework, it lacks a few things mandatory for production environments which are mainly authentication, better logging, recovery from errors and input checking.
But I believe this might be a very useful start point for many developers who need this kind of a service, just like I needed it before implementing it :)
I will be very happy to read your thoughts and comments, happy holidays to all!

About the author:

Hi, my name is Orr Chen, a software engineer and a gopher for the past 3 years. My first experience with Go was migrating the entire backend of my startup PushApps from Rails to Golang. Since then I am a big fun of the language!
Github: OrrChen
Twitter: OrrChen
LinkedIn: orrchen

3rd parties libraries used:

2017-12-30

Table of contents

Introduction

Go 1.10 is the first major release after the announcement of the plans towards Go 2.0 at GopherCon 2017.

There are a number of exciting changes which I’ll cover below as well as some changes in the behavior of either tools or Go APIs which might result in an unexpected behavior compared to the previous change. I chose to flag these changes as “breaking change” in order to make it easier to identify them.

I’ve also tried to flag the CL that brought in the change as the discussions on them as well as the related changes in other CLs are a great source from learning how Go is organized, created, how features are reviewed, and hopefully inspire you to contribute to the language itself by either participating in reviews or issues or fixing issues (like Needs Investigation or Help Wanted).


Language changes

Let’s start with the language changes that Go 1.10 brings. There are only a couple of rather small changes, none of them significant.

First, you’ll be able to use an untyped constant as the index of an expression x[1.0 << s] where s is the untyped constant CL 60230

The second change is that you can now use method expressions like this struct{io.Reader}.Read, even if this is a rather unusual way to do so CL 73233


Operating systems support

Moving on to the operating system support, Go 1.10 will be the last Go version to run on OS X 10.8 Mountain Lion, OS X 10.9 Mavericks, or on OpenBSD 6.0.

FreeBSD 10.3 is now required to run Go, up from FreeBSD 9.3 CL 64910

NetBSD is once again supported, but only in the unreleased version 8 and on 386 and amd64. arm support for NetBSD 8 was still broken at the time of writing, see Issue 23073

On 32-bit MIPS systems you can now choose if you want emulation for floating point instructions or not via a new environment variable settings GOMIPS=hardfloat (the default) and GOMIPS=softfloat CL 37954


Tooling

The bigger changes in Go 1.10 come from the tooling improvements it brings. They dramatically improve the quality of life for testing in large and very large projects and pave the road towards Go 2.0.


Environment variables

Since Go 1.9,GOROOT is inferred from the location of the go tool binary by default. However, if your application relied on that value at runtime via runtime.GOROOT() there was a bug which prevented it have the correct location. As of Go 1.10+ this bug was fixed and you can now use it as expected CL 61310

A couple of new environment variables were added to the go command. GOTMPDIR will allow you to configure where the temporary files created by Go during compilation of your applications are stored. The default path is the same as in the previous Go versions, the operating system’s temporary files directory. The other new environment variable is GOCACHE allows you to control where the go command will store the cached information that’s reused in future builds CL 75475


go build

go build can now detect changes in files on a source code level rather than rely on timestamps, which means that it will be more accurate in rebuilding the packages that have changed. In turn, this means that you will now be able to drop the usage of -a flag, which was previously used to force Go to rebuild packages in certain conditions.

Changes are coming to the -asmflags, -gcflags, -gccgoflags, and -ldflags flags. They will not be applied automatically to the list of all packages as before but only to the direct package that’s being specified in the build command. You can still achieve the same functionality as before using the new special syntax for these flags, -ldflags=pattern=flags such as:
go build -ldflags=cmd/gofmt=-X=main.version=1.2.3 cmd/.... This command allows you to build all the /cmd/... packages but it will apply the -ldflags=-X=main.version=1.2.3 flag only to the cmd/gofmt package CL 76551

To further speed up the builds, go build -i will not be necessary since now the build tool will have its own cache for build steps that do not do the install step, such as go install or go get. This means that you’ll be able to switch between branches or experiment a lot more with the code without having to invoke go install or go build -i but just go build.

Are you a Windows user? Now you can use c-shared as a target for your libraries, thanks to CL 69091.


go install

go install also received some changes. Now it will install only the explicitly mentioned packages but not their dependencies. To restore the previous behaviour, you’ll need to use this command as go install -i CL 75850

This change as well as upcoming changes is significant if your tools depend on the packages to be installed in $GOPATH/pkg and always be fresh, with additional changes being required in order to restore the old behaviour.


go test

Speaking of caching, go test has seen a lot of changes as well. One of the most important is that go test will now cache the results of the tests if they meet certain criteria such as:

  • the test executable and command line match a previous run
  • the files and environment variables used by that run have not changed
  • the results are successful
  • the go test command received a list of packages to test, go test ./... for example
  • the test command line uses a subset of the test flags, -cpu, -list, -parallel, -run, -short, and -v

When the above conditions are met, the first run will produce the output as expected then subsequent runs will simply reuse that output. The run time of the tests will also notify that the cached output is used, by displaying (cached) instead of the original test run time.

You can always force the tests to run by specifying the flag -count=1, this being considered the idiomatic way to handle this requirement. As a recap, the -count flag allows you to specify how many times a test or a benchmark runs.

This change is covered in CL 75631.

The second important change to go test is that now a subset of go vet checks will run before the tests run in order to detect issues with your code. These checks will be treated as build failures in case any of them will produce any result. Only very high accuracy checks are included in this step. To disable this new behavior you’ll need to provide the -vet=off flag to the go test command CL 74356

The coverage profile of tests can now be created when running the tests against multiple packages, which was a highly requested feature. Combined with the new way to use the -coverpkg flag, it means you’ll be able to get the coverage for all the packages tested packages as well as their dependencies when multiple packages are being tested by running go test -coverpkg=all -coverprofile cover.out ./... CL 76875 and CL 76876

Test binaries will now always write to stdout when invoked via go test whereas before stderr could have been used sometimes CL 76871

Tests running in parallel will now be better delimited by having PAUSE and CONT as status update lines when running with -v flag. This change allows tooling to better interpret the start and stop of parallel tests. The -failfast flag now will stop testing immediately on the first failure, with the caveat that parallel tests are still allowed to continue until they finish CL 74450

Finally, go test -json will now output the format of the tests in json so that tooling such as IDEs can better present the results of the test. There is also a new command, go tool test2json that will produce convert the test output to json CL 76873 and CL 76872


gofmt

Another set of tooling changes in Go 1.10 come from gofmt as it received a few updates. First, three index slice expressions containing complex expressions are now always formatted as slice[start+1 : stop : capacity] CL 67633

The second change is that single-method interface literals written on a single line, which are sometimes used in type assertions, are no longer split onto multiple lines CL 66130

The third one is that if a composite literal would include a comment and only comments, then the comment(s) will now be indented CL 74232

If you use gofmt in your CI environment, you will see some failures because of these changes. The official position is that gofmt is not covered by the same set of compatibility promises as Go 1 itself, so these are not “breaking changes” but it’s rather a constant evolving specification, which can suffer changes on each new Go release. The recommendation is not to have gofmt enforced in the CI or have everyone use the same binary version for the application that formats your source code as well as checks it in the CI system.

A good news is that now all flags supported by gofmt are supported by go fmt as well.


go fix

go fix now replaces imports from golang.org/x/net/context with context which will help you migrate your code to a Go 1.9+ compatible code by running go tool fix -r context your/package CL 58590


pprof

Go 1.10 will also bring an update to the pprof tool. This brings a host of improvements, among which an updated UI featuring a flame graph representation of the profiling data CL 75870


Runtime

Go’s runtime received a few updates, with the first one I’ll cover being done on how the LockOSThread and UnlockOSThread mechanism works. If before, in nested calls, UnlockOSThread would need to be called only once to unlock the thread, now it will need to be called as many times as LockOSThread was called CL 45752

You may have noticed the <autogenerated> frame (line) in the stack traces before. This is now hidden, unless a panic or other issue happens in it. This also means that if your code would call runtime.Caller with a certain number of skip frames, then this change will be a “breaking change” in its behaviour as the <autogenerated> frames will not be counted there either CL 45412

Another important change in the Go runtime is the introduction of soft and hard goals (limits) for garbage collection CL 59970

The soft limit is the current value of the GOGC while the hard limit is 10% higher than the soft limit. Heavy GC reliant applications (so far only benchmarks) shows that there’s an increase in the heap size of the application.


CGO support

CGO support has also received updates, with C typedefs such as typedef X Y now which means you’ll be able to use C.X and C.Y interchangeably in Go now, as if they would be Go aliases, type X = Y CL 62670

Another welcomed change when working with C and Go is that you can now pass Go strings directly to C. This is done by declaring a C function in a Go file with a parameter type of the special type name _GoString_. To access the string length, you’ll need to call size_t _GoStringLen(_GoString_ s) an in order to get the pointer to the string contents, you’ll need use const char *_GoStringPtr(_GoString_ s) CL 70890

Some C types that were previously mapped to a pointer type in Go are now mapped to uintptr type. A couple of these types are CFTypeRef in Darwin’s CoreFoundation framework and the jobject in Java’s JNI interface. You’ll need to initialize the values for the affected types with 0 instead of nil This is a breaking change but thankfully you can fix this quickly and automatically by running go tool fix -r cftype your/package or go tool fix -r jni your/package CL 66332 and CL 81876


Debugging

Debugging support has also been improved in the latest release which should make your debugging experience via Delve even better than before. And as a reminder, Delve is most likely integrated with your favorite code editor.


Assembly support

Assembly support got better as well, with a host of new instructions being added. Most important changes are under amd64 platform with 359 new instructions including the full AVX, AVX2, BMI, BMI2, F16C, FMA3, SSE2, SSE3, SSSE3, SSE4.1, and SSE4.2 extension sets.


Packages

Changes in the various standard library packages:

  • bufio - the new Reader.Size and Writer.Size and methods report the Reader or Writer’s underlying buffer size CL 75150

  • bytes - the Fields, FieldsFunc, Split, and SplitAfter each already returned slices pointing into the same underlying array as its input. Go 1.10 changes each of the returned subslices to have capacity equal to its length, so that appending to a subslice will not overwrite adjacent data in the original input. This is also a “breaking change” in the behavior of these functions and you might need to update your code.

  • crypto/tls - the TLS server now advertises support for SHA-512 signatures when using TLS 1.2. The server already supported the signatures, but some clients would not select them unless explicitly advertised CL 74950

  • crypto/x509 - leaf certificate validation now enforces the name constraints for all names contained in the certificate, not just the one name that a client has asked about. Extended key usage restrictions are similarly now checked all at once. As a result, after a certificate has been validated, now it can be trusted in its entirety. It is no longer necessary to revalidate the certificate for each additional name or key usage CL 62693

  • database/sql/driver - drivers that want to construct a sql.DB for their clients can now implement the Connector interface and call the new sql.OpenDB function, instead of needing to encode all configuration into a string passed to sql.Open. Drivers that want to parse the configuration string only once per sql.DB instead of once per sql.Conn, or that want access to each sql.Conn’s underlying context, can make their Driver implementations also implement DriverContext’s new OpenConnector method. Drivers that implement ExecerContext no longer need to implement Execer; similarly, drivers that implement QueryerContext no longer need to implement Queryer. Previously, even if the context-based interfaces were implemented they were ignored unless the non-context-based interfaces were also implemented. To allow drivers to better isolate different clients using a cached driver connection in succession, if a Conn implements the new SessionResetter interface, database/sql will now call ResetSession before reusing the Conn for a new client

  • encoding/json - the Decoder adds a new method DisallowUnknownFields that causes it to report inputs with unknown JSON fields as a decoding error. The default behavior has always been to discard unknown fields. CL 27231 Unmarshal can no longer decode into fields inside embedded pointers to unexported struct types, because it cannot initialize the unexported embedded pointer to point at fresh storage. Unmarshal now returns an error in this case. This means you may need to update your code or a “breaking change” will happen, which could be hidden if the code is not properly handling errors CL 76851

  • text/template and html/template - the new actions {{break}} and {{continue}} break out of the innermost {{range ...}} loop, like the corresponding Go statements CL 66410

  • math/rand - the new math/rand.Shuffle function and corresponding math/rand.*Rand.Shuffle method shuffle an input sequence CL 51891

  • math - the new functions Round and RoundToEven round their arguments to the nearest floating-point integer; Round rounds a half-integer to its larger integer neighbor (away from zero) while RoundToEven rounds a half-integer to its even integer neighbor CL 43652 and CL 61211

  • net - the Conn and Listener implementations in this package now guarantee that when Close returns, the underlying file descriptor has been closed. In earlier releases, if the Close stopped pending I/O in other goroutines, the closing of the file descriptor could happen in one of those goroutines shortly after Close returned. TCPListener and UnixListener now implement syscall.Conn, to allow setting options on the underlying file descriptor using syscall.RawConn.Control. The Conn implementations returned by Pipe now support setting read and write deadlines. The IPConn.ReadMsgIP, IPConn.WriteMsgIP, UDPConn.ReadMsgUDP, and UDPConn.WriteMsgUDP, methods are now implemented on Windows

  • net/http - on the client side, an HTTP proxy, most commonly configured by ProxyFromEnvironment, can now be specified as an https:// URL, meaning that the client connects to the proxy over HTTPS before issuing a standard, proxied HTTP request. Previously, HTTP proxy URLs were required to begin with http:// or socks5://. On the server side, FileServer and its single-file equivalent ServeFile now apply If-Range checks to HEAD requests. FileServer also now reports directory read failures to the Server’s ErrorLog. The content-serving handlers also now omit the Content-Type header when serving zero-length content. ResponseWriter’s WriteHeader method now panics if passed an invalid (non-3-digit) status code. Redirect now sets the Content-Type header before writing its HTTP response

  • net/url - ResolveReference now preserves multiple leading slashes in the target URL. Previously it rewrote multiple leading slashes to a single slash, which resulted in the http.Client following certain redirects incorrectly

  • os - File adds new methods SetDeadline, SetReadDeadline, and SetWriteDeadline that allow setting I/O deadlines when the underlying file descriptor supports non-blocking I/O operations CL 71770 The definition of these methods matches those in net.Conn. Also matching net.Conn, File’s Close method now guarantee that when Close returns, the underlying file descriptor has been closed

  • strings - a new type Builder is a replacement for bytes.Buffer for the use case of accumulating text into a string result. The Builder’s API is a restricted subset of bytes.Buffer’s that allows it to safely avoid making a duplicate copy of the data during the String method CL 74931

  • unicode - the unicode package and associated support throughout the system has been upgraded from version 9.0 to Unicode 10.0, which adds 8,518 new characters, including four new scripts, one new property, a Bitcoin currency symbol, and 56 new emoji

A lot more packages have received changes but I’ve tried to keep the list to a minimum. To view the full list of changes, you can read this the Draft Release Notes.


Closing notes

Due to the vast amount of changes both in compiler and in runtime, some workloads are expected to perform better as of Go 1.10. However, I highly recommend that you grab the latest Go 1.10 release, 1.10 Beta 1 and test it on your workloads, run the tests against the new Go version and help the Go team identify issues before Go 1.10 lands in February. There are a few weeks where even just running the test / benchmark suite could make the difference. And Go 1.10 Beta 1 is also available as a Docker container so you can minimize the impact it has on your system.

If you want to stay up to date with the developments of Go, I recommend following @golang_cls Twitter account which provides a curated list of interesting commits as they are added to Go.

If you want to talk more about Go, its evolution, and what’s next for Go, tweet me, or meet me at Gophercon Iceland!

Finally, I would like to thank the Go Team and all contributors that helped Go reach 1.10 and I look forward to what the future holds.

A big thank you goes to Russ Cox and the team that created the initial draft documentation which this article uses / reuses a lot.

Errata:

The initial version of this article incorrectly mentioned a change about how GOROOT is handled. Thank you to Dominik Honnef for reporting this.

2017-12-29

One of the most hotly debated topics in the world of the Go programming language, is the lack of generics. Generics are considered a key feature in other statically typed programming languages like Java or C#. However, the makers of Go resisted the demand to add it to the language so far. A key reason … Continue reading "The empty interface in the Go programming language"

The post The empty interface in the Go programming language appeared first on Mina Andrawos.

2017-12-29

Create a Slack bot with golang

Introduction

In this post we’ll look at how to set up a quick Slack bot that receives messages (either direct or from channel) and replies to the user. I’ve been an IRC user for many years and always loved setting up bots, whether for sports scores, weather, or something else entirely. Recently I’ve actually had an opportunity to implement my first Slack bot and figured I would document the process for others! You can find all of the code for this post listed here, and PRs are certainly welcome :D

For this assignment we’ll need a few things, not all of which are covered in this post. I invite the reader to take a look at the installation practices for the other software dependencies based on their specific environment needs. Here I’ll be using Fedora 26 (4.14.6-200.fc26.x86_64) along with these tools:

  1. ngrok for Slack API replies – https://ngrok.com/docs#expose
  2. NHL statsapi to collect hockey scores – https://statsapi.web.nhl.com/api/v1/schedule
  3. the excellent golang slack library from nlopes – https://github.com/nlopes/slack

You’ll either need to set up an ngrok listener for your chosen localhost port, or develop on a server that it externally routable (e.g. DigitalOcean droplet). In my case here I’m developing on my laptop but would deploy permanently on a droplet.

The Slack API

Initial Configuration

The Slack API is well flushed out and spells out what specific payloads to anticipate for any particular object. There are a number of calls you can develop your bot to address, but in our case here we’ll look at using the Real Time Messaging API (RTM) and specifically the chat.postMessage and chat.postEphemeral methods.

Before any of our code is working we’ll need to set up an app within slack itself. Navigate to the app registration tool to create a new application within your workspace. Here I’ve created the NHL Scores app within my workspace.

Create App

Once done you’ll be presented with a number of options for your new application. Here we’ll need to create a Bot User that will act as our listener within the workspace. My example is called nhlslackbot and will be visible to all users within the workspace once mounted.

Bot User

We’ll need to generate an OAuth token for our user in order to actually connect with the Slack API. To do so click on the OAuth & Permissions section to Install App to Workspace which will prompt you to authorize access and generate the tokens you’ll use. You’ll need to copy the Bot User OAuth Access Token somewhere local, but always make sure this is not shared anywhere! This token is secret and should be treated like your password!

Authorize

Lastly we’ll need to set up the Interative Components of our application and specify the ngrok (or other) endpoint that the API will send responses to. In my case, I’ve added a custom ngrok value here called https://sebtest.ngrok.io/. This endpoint is where we’ll receive all correspondence from Slack itself, and this is how we’ll be able to process any incoming messages from the channels.

Interactive

With that all sorted, we can finally dig into the code!

Code components

The crux of the code is how we handle receiving messages from the slack connection. Using the Bot User OAuth Access Token to establish the initial connection, we must continuously poll the system for incoming messages. The API gives us the ability to trigger off of a number of event types, such as:

  1. Hello Events
  2. Connected Events
  3. Presence Change Events
  4. Message Events
  5. and many more

The beauty of this verbosity is that we can trigger messages on a number of different use-cases, really giving us the ability to tailor the bot to our specific needs. For this example, we’ll look at using the *slack.MessageEvent type to support both indirect (within channel using @) or direct messages. From the library, The primary poll for message events leverages the websocket handler and just loops over events until we’ve received one that we want:

func (s *Slack) run(ctx context.Context) {
    slack.SetLogger(s.Logger)

    rtm := s.Client.NewRTM()
    go rtm.ManageConnection()

    s.Logger.Printf("[INFO]  now listening for incoming messages...")
    for msg := range rtm.IncomingEvents {
        switch ev := msg.Data.(type) {
        case *slack.MessageEvent:
            if len(ev.User) == 0 {
                continue
            }

            // check if we have a DM, or standard channel post
            direct := strings.HasPrefix(ev.Msg.Channel, "D")

            if !direct && !strings.Contains(ev.Msg.Text, "@"+s.UserID) {
                // msg not for us!
                continue
            }

            user, err := s.Client.GetUserInfo(ev.User)
            if err != nil {
                s.Logger.Printf("[WARN]  could not grab user information: %s", ev.User)
                continue
            }

            s.Logger.Printf("[DEBUG] received message from %s (%s)\n", user.Profile.RealName, ev.User)

            err = s.askIntent(ev)
            if err != nil {
                s.Logger.Printf("[ERROR] posting ephemeral reply to user (%s): %+v\n", ev.User, err)
            }
        case *slack.RTMError:
            s.Logger.Printf("[ERROR] %s\n", ev.Error())
        }
    }
}

Once we confirm that the message is indeed directed to us, we pass the event handler along to our askIntent function. Remember that this is a contrived example that’s just going to send back NHL game scores to the user, iff they acknowledge that specific intent. We could build up an entire workflow around this user interaction that would send different paths depending on user choices to our prompts, or have no prompts at all! Those different cases are outside the scope of this introductory post, so for now we just want to send back a quick Yes v No prompt and handle accordingly.

To do precisely that, our handler askIntent will process the message and genreate an chat.postEphemeral message to send back to the event user (aka the person asking for details). The “ephemeral” post is one that’s directed only to the requester. Though other users will see the initial request to the bot if within the same channel, the subsequent interaction with the bot will only be done between the user and the bot. From the docs:

This method posts an ephemeral message, which is visible only to the assigned user in a specific public channel, private channel, or private conversation.

With that in mind, we set up the initial response payload using the attachments spec from the API, defining a set of actions that the user is able to choose. For this part of the conversation the user must reply Yes or No for whether they’d like us to retrieve the most recent scores. If No, we reply with a basic note and continue listening; if Yes then let’s retrieve the scores!

// askIntent is the initial request back to user if they'd like to see
// the scores from the most recent slate of games
//
// NOTE: This is a contrived example of the functionality, but ideally here
// we would ask users to specify a date, or maybe a team, or even
// a specific game which we could present back
func (s *Slack) askIntent(ev *slack.MessageEvent) error {
    params := slack.NewPostEphemeralParameters()
    attachment := slack.Attachment{
        Text:       "Would you like to see the most recent scores?",
        CallbackID: fmt.Sprintf("ask_%s", ev.User),
        Color:      "#666666",
        Actions: []slack.AttachmentAction{
            slack.AttachmentAction{
                Name:  "action",
                Text:  "No thanks!",
                Type:  "button",
                Value: "no",
            },
            slack.AttachmentAction{
                Name:  "action",
                Text:  "Yes, please!",
                Type:  "button",
                Value: "yes",
            },
        },
    }

    params.Attachments = []slack.Attachment{attachment}
    params.User = ev.User
    params.AsUser = true

    _, err := s.Client.PostEphemeral(
        ev.Channel,
        ev.User,
        slack.MsgOptionAttachments(params.Attachments...),
        slack.MsgOptionPostEphemeralParameters(params),
    )
    if err != nil {
        return err
    }

    return nil
}

The attachments in the snippet above present the user with the following dialog:

Options

If the user selects No, thanks! then we reply with a basic message:

Choose No

This part of the interaction is precisely where the ngrok endpoint comes into play. The user’s interaction is not directly with our code, but instead with slack itself. The message and interaction is passed through slack and on to us at the redirect URL we specified earlier, in my case https://sebtest.ngrok.io which routes to our internal localhost:9191 interface, and from there to our postHandler as defined in our webapp router.

The tricky part here is to process the payload portion of the JSON response from the API. The POST that slack returns back to our URL is a payload form that contains a bevy of information for our interaction. In this case, the user’s response (either Yes or No) as well as a callbackID which we actually passed in our original mesage prompt to the user! This is incredibly useful, especially as you have more and more users interacting with your bot as you can specify unique actions based on the trigger. For example, if the user selects Yes we could send subsequent ephemeral messages to ask for a specific date, or maybe a certain team? We could even define the callback value as a key to a function map that would then trigger some kind of other workflow altogether (like posting to a blog resource, or checking DB credentials, etc). The options are indeed endless, but for the scope of this contrived example we just stick to the scores from last night.

func postHandler(w http.ResponseWriter, r *http.Request) {
    if r.URL.Path != "/" {
        w.WriteHeader(http.StatusNotFound)
        w.Write([]byte(fmt.Sprintf("incorrect path: %s", r.URL.Path)))
        return
    }

    if r.Body == nil {
        w.WriteHeader(http.StatusNotAcceptable)
        w.Write([]byte("empty body"))
        return
    }
    defer r.Body.Close()

    err := r.ParseForm()
    if err != nil {
        w.WriteHeader(http.StatusGone)
        w.Write([]byte("could not parse body"))
        return
    }

    // slack API calls the data POST a 'payload'
    reply := r.PostFormValue("payload")
    if len(reply) == 0 {
        w.WriteHeader(http.StatusNoContent)
        w.Write([]byte("could not find payload"))
        return
    }

    var payload slack.AttachmentActionCallback
    err = json.NewDecoder(strings.NewReader(reply)).Decode(&payload)
    if err != nil {
        w.WriteHeader(http.StatusGone)
        w.Write([]byte("could not process payload"))
        return
    }

    action := payload.Actions[0].Value
    switch action {
    case "yes":
        grabStats(w, r)
    case "no":
        w.Write([]byte("No worries, let me know later on if you do!"))
    default:
        w.WriteHeader(http.StatusNotAcceptable)
        w.Write([]byte(fmt.Sprintf("could not process callback: %s", action)))
        return
    }

    w.WriteHeader(http.StatusOK)
}

A key component to note here is the http response code; if you do not specify the http.StatusOK value in your prompt back to the API, the error message you may want to convey to the user gets eaten by the system. The default slackbot will absorb that message and reply to you (with an ephemeral message no less) with the status code, but not the messages. Long story short, whatever message you’d like to actually send back to the requester should have an http.StatusOK header.

Lastly, if our user has selected the Yes option we call out to our NHL stats api and process the results for the user!

// grabStats will process the information from the API and return the data to
// our user!
func grabStats(w http.ResponseWriter, r *http.Request) {
    n := fetch.New()

    buf, err := n.GetSchedule()
    if err != nil {
        w.WriteHeader(http.StatusNoContent)
        w.Write([]byte(fmt.Sprintf("error processing schedule; %v", err)))
        return
    }

    w.WriteHeader(http.StatusOK)
    w.Write(buf)
}

// GetSchedule calls out to the NHL API listed at APIURL
// and returns a formatted JSON blob of stats
//
// This function calls the 'schedule' endpoint which
// returns the most recent games by default
// TODO: add options to provide date range
func (n *NHL) GetSchedule() ([]byte, error) {
    var buf bytes.Buffer

    r, err := http.Get(fmt.Sprintf("%s/schedule", APIURL))
    if err != nil {
        return buf.Bytes(), err
    }
    defer r.Body.Close()

    err = json.NewDecoder(r.Body).Decode(&n.Schedule)
    if err != nil {
        return buf.Bytes(), fmt.Errorf("error parsing body: %+v", err)
    }

    for _, x := range n.Schedule.Dates {
        for idx, y := range x.Games {
            buf.WriteString(fmt.Sprintf("Game %d: %s\n", idx+1, y.Venue.Name))
            buf.WriteString(fmt.Sprintf("Home: %s -- %d\n", y.Teams.Home.Team.Name, y.Teams.Home.Score))
            buf.WriteString(fmt.Sprintf("Away: %s -- %d\n\n", y.Teams.Away.Team.Name, y.Teams.Away.Score))
        }
    }

    return buf.Bytes(), nil
}

Sample output below…

Sample Output

Congratulations, you’ve now delivered an ephemeral payload to your slack user’s request!


About The Author

Sebastian Borza is a golang, C, and python developer based in Chicago, IL.

Source Handle
freenode sborza
efnet sebito91
github sebito91
twitter @sebito91
keybase sborza
GPG E4110D3E