The State of Go Plugins
Terraform, Protocol Buffers, Waypoint or Traefik have build a flourishing ecosystem by using the Go programming language and providing pluggable interfaces. But, not all pluggable interfaces in Go are created equal. There is more than one solution to the problem.
The problem they all try to solve is, that Go is a statically typed, compiled programming language. Thus, executing and interacting with code beyond the build phase of a Go program is hard. The progress of the standard library plugin package has also been very slow in recent years.
These are the common solutions to the problem out in the wild.
- Client-server
- Stdin to Stdout
- Dynamic interpretation
- Dynamic libraries with symbol resolution
Client-Server
This is perhaps the most common approach to pluggable interfaces in Go. Terraform or Waypoint use this approach. How does this works?
Plugins are standalone Go programs (executables) that the main program executes. The main program runs a server that the plugin can connect to when run. The connection information are passed to the plugin as parameters or environment variables. Client and server commonly communicate via gRPC. Which allows for efficient communication, sharing the used interface and code generate client and server boilerplate.
The client-server communication via HTTP/2 has an impact on performance. If you want to build a pluggable program serving requests to anything on the internet it may increased response times by anything between 30-100ms.
Stdin to Stdout
The difference to the client-server solution is subtile when it comes to the details in implementation. protoc-gen-go and other plugins in the Protocol Buffers ecosystem use this approach.
Plugins are again executables that the main program runs. But in contrast to the client-server communication via gRPC the main program and the plugin communicate via the standard OS streams stdin / stdout with each other. In order to standardize the communication, many of the programs use Protocol Buffers.
Dynamic interpretation
The previous solutions have a performance penalty when it comes to transport information. Traefik and other programs which are sensitive to response times pursue a different approach.
They dynamically load, interpret and execute Go code. The folks at Traefik Labs have created Yaegi to do this. This allows for calling into the plugin code by using an interface and a bit of reflection magic of the runtime. Because the code is loaded and interpreted only once the performance penalty is only a the startup. But, it increases the complexity.
Dynamic libraries with symbol resolution
The supported for plugins in the standard library started in Go1.8. Plugins ought to be shared objects (.so) that can build with ELF symbols that can be dynamically loaded at runtime. However, the progress has been fairly limited. Currently plugins are only supported on Linux, FreeBSD, and macOS. There are many issues with sharing modules in the plugin and the main program. Beyond some examples the use of this solution has been fairly limited.