I work for Dolt, the world’s first version-controlled database. We made Dolt as a drop-in replacement for MySQL, since MySQL was the most commonly used SQL database in production when we started. But new teams by-and-large are not choosing MySQL; they’re using Postgres. So we also made Doltgres, a alternate version of Dolt that speaks the Postgres dialect.

Unsurprisingly, Dolt and Doltgres share a lot of common code. But there’s also subtle differences between MySQL and Postgres’s feature set. We want to use shared code for things that the projects have in common, and interfaces to implement behavior where they differ.

One of our big features is Git-style branches. A client session operates on a currently checked-out branch, and is completely independent of the other branches in the database. Except when I say “completely independent”, I actually mean “mostly independent.” When we made Dolt, we decided that there was one specific situation where we wanted branches to not be independent: auto incrementing columns.

In MySQL, if a table has a column declared with the AUTO_INCREMENT modifier, inserts aren’t required to specify a value for that column. Instead, a new value will be generated that is guaranteed to not conflict with any values currently in the table. This is true even when multiple sessions are making transactions concurrently, which each transaction getting a different value for the column. While transactions are typically independent from each other, MySQL makes an exception here so that the transactions won’t conflict if both are committed.

We decided that it made sense to apply that same behavior to branches: if sessions on two different branches are inserting into the same table with an auto increment column, Dolt is guaranteed to generate non-conflicting values for that column. If we didn’t do this, then any attempt to merge those branches would result in a merge conflict.

In contrast, Postgres doesn’t have AUTO INCREMENT. Instead, it has SERIAL columns, which behave similarly but have some key differences:

• In MySQL, AUTO INCREMENT values are always unsigned, while in Postgres, SERIAL values are always signed.
• SERIAL columns are backed by a data type called a Sequence, which has configuration parameters such as the min and max values, whether the generated values are incremented or decremented, whether they wrap around when they reach the end, etc.

Synchronizing the behavior across all branches and all transactions has a lot of tricky corner cases, that we’d already gotten right with AUTO INCREMENT, and duplicating that logic would be a bad idea. So we designed a data model that could handle both:

• A Sequence is an item in the database capable of producing a sequence of values, such as a MySQL table with an AUTO INCREMENT column, or a Postgres sequence.
• A SequenceValue is a value produced by a sequence.
• A SequenceState is a type that represents the current state of a Sequence and can be incremented to produce new SequenceValues.

Using these definitions, we were able to implement all of the behavior common to both MySQL tables and Postgres sequences as generic methods on these types.

This reminded me of a similar situation I encountered two years ago, involving pairs of data structures each consisting of a mutable type and an immutable type that could be converted between each other. The goal was to write something like the below, that could be used with any pair of types that satisfied this contract:

func ApplyMutations(immutable ImmutableValue, mutations []Mutation) ImmutableValue {
  mutableValue := immutable.Mutate()
  mutableValue.ApplyMutations(mutations)
  return mutableValue.Flush()
}

I wrote a blog post talking about my solution to that problem, how I’d used a Golang generic interface to model a situation where a group of types collectively implement some contract. But there’s a lot in that I got flat-out wrong about how go Golang generics worked, and I got rightfully chewed out for it in the responses to the extent that I’m a bit embarrassed to bring it up again.¹

This time, I was determined to make a better solution. This is a real problem, and it’s worth understanding the correct way to tackle it in Go because it’s a useful design pattern for writing type-safe code.

The Problem#

In Go, if you want a function to operate on multiple different types, you can define an interface. An interface is a contract: it specifies a set of constraints that the implementing type must satisfy, usually methods that the type must implement. Any type that implements these methods will satisfy the interface. But this interface only constrains a single type. Sometimes, you have a group of types that need to implement a contract together. How can we use Go’s language features to solve this problem?

What Doesn’t Work: Interfaces#

If we were to express the above-mentioned Sequence data model as interfaces, it might look something like this:

type SequenceValue interface {
    int64 | uint64
}

type SequenceState interface {
    CurrentValue() SequenceValue
    Advance() (nextValue SequenceValue, nextState SequenceState)
}

type Sequence interface {
    CurrentState() SequenceState
    UpdateFromGlobalState(SequenceState) Sequence
}

But making these types regular interfaces is a bad idea for several reasons:

• There’s a performance penalty for calling interface methods due to dynamic dispatch.
• All values of an interface type are boxed and their underlying values are stored on the heap.
• Go doesn’t actually allow interfaces with shape constraints (like SequenceValue above) to be used in method signatures.

But the biggest problem is that this fails to give us the compile-time type safety we want. It doesn’t document or enforce that a particular SequenceState implementation is always expected to have a specific type for SequenceValue. We could need to insert check-casts every time one of these methods is called. If we ever call a method with the wrong implementation, we would panic at runtime.

This isn’t a good use of interfaces, because we’re paying a cost in performance and code complexity but not getting anything out of it.

The Bad Idea: A Single Generic Interface#

This was the concept in my previous attempt that rightly got a lot of pushback.

The basic idea was that if interfaces are how you achieve polymorphism in Go, and an interface is defined by a set of behaviors on a single type, then you can achieve a contract on multiplace types by implementing an interface that accepts each of those types as a generic type parameter.

So in this case, you would have a “contract” type that defines all the necessary behavior, and generic code must call methods on this contract type:

type SequenceContract[SequenceValue any, SequenceState any, Sequence any] interface {
    SequenceState_CurrentValue(SequenceState) SequenceValue
    SequenceState_Advance(SequenceState) (nextValue SequenceValue, nextState SequenceState)
    Sequence_CurrentState(Sequence) SequenceState
    Sequence_UpdateFromGlobalState(Sequence, SequenceState) Sequence
}

The main downside of this is that it’s ugly. Shared code can’t call methods on the component types because they all have an any type constraint. Instead it must call the methods on SequenceContract, which will likely just delegate to the corresponding methods on the component types.

This approach also requires that shared code takes a SequenceContract value as an extra parameter. This also requires the shared code to be itself generic, which means that prior to Go 1.27 it couldn’t be a method and had to be a function.

There are some possible upsides to this approach: implementations of the contract type are allowed to have state, and it’s possible to define multiple contracts on the same collection of types. But these “upsides” are a double-edged sword, because now you’re adding further complexity to your data model. This is rarely the best approach.

Almost A Solution: Generic Interfaces#

We could attempt to capture the relationship between types via generic interfaces. Going back to original example with the corresponding mutable and immutable types, we could attempt to write interfaces like so:

type ImmutableValue[T ...] interface {
  Mutate() T
}
  
type MutableValue[T ...] interface {
  ApplyMutations([]Mutation)
  Flush() T
}

But what do we put in place of ”…” in the example above?

We can’t have the two interfaces reference each other (that is, we can’t write type ImmutableValue[T MutableValue]) because MutableValue isn’t a complete type. And even if we could somehow do that, prior to Go 1.26 we weren’t allowed to have both ImmutableValue and MutableValue reference each other in their type constraints.

One option is to just use any:

type ImmutableValue[T any] interface {
  Mutate() T
}
  
type MutableValue[T any] interface {
  ApplyMutations([]Mutation)
  Flush() T
}

But this alone isn’t enough to let us write a function like ApplyMutations above: if the return type of Mutate() is constrained by any, then we can’t call any methods on it.

As we’ll soon see, Go 1.26 added Recursive Type Parameters, which actually gives us something that we can put here. But as we’ll also see, putting a strict type constraint here isn’t actually necessary, and doesn’t actually change what the correct solution looks like.

The Correct Idea: Mutually Referential Type Parameters#

This is a technique originally described in the original Go type parameters proposal.

The idea is to let go of the idea that every part of your data model needs to be described with interfaces. What actually matters is your functions and structs, which describe the data that they accept. An interface is powerful because it allows you to name a set of constraints and reuse them. It’s a useful tool for deduplicating constraint defitions and for composing them, but they’re a means to an end.

So building on the previous example, we can use any type constraints in the interface definitions, and then further constraint them when these interfaces are actually used:

type ImmutableValue[T any] interface {
  Mutate() T
}
  
type MutableValue[T any] interface {
  ApplyMutations([]Mutation)
  Flush() T
}

func ApplyMutations[
    ImmutableType ImmutableValue[MutableType],
    MutableType MutableValue[ImmutableType],
](
    immutable ImmutableValue,
    mutations []Mutation,
) ImmutableValue {
  mutableValue := immutable.Mutate()
  mutableValue.ApplyMutations(mutations)
  return mutableValue.Flush()
}

And for the case of the Sequence type, which must be able to return itself, it can take a self-referential type constraint. These interfaces are then used in creating the full set of type constraints for a struct, which contains our actual business logic as methods:

type SequenceState[Self any, ValueType any] interface {
    CurrentValue() ValueType
    Advance() (nextValue ValueType, nextState Self)
}

type Sequence[Self any, StateType any, ValueType any] interface {
    CurrentState() StateType
    UpdateFromGlobalState(StateType) Self
}

type SequenceTracker[
    ValueType any,
    StateType SequenceState[StateType, ValueType],
    SequenceType Sequence[SequenceType, StateType, ValueType]
] struct {
    ...
}

Note that even though the interface defintions don’t enforce any requirements for their type parameters, the code is still fully type-safe because the types are fully constrained where they’re actually used.

The one downside is that the function and struct defintions themselves can become quite verbose. And every function that accepts these types must duplicate the type constrainted. The interface definitions can reduce the verbosity but don’t eliminate it. Fortunately, all these types can be inferred at the callsite, so the callsite remains clean.

Go 1.26 allows us to further constrain the interface definitions by allowing the type constraints to reference the interface type being defined:

type SequenceState[
    Self SequenceState[Self, ValueType],
    ValueType any
] interface {
    CurrentValue() ValueType
    Advance() (nextValue ValueType, nextState Self)
}

type Sequence[
    Self Sequence[Self, StateType, ValueType],
    StateType SequenceState[Self, ValueType],
    ValueType any,
] interface {
    CurrentState() StateType
    UpdateFromGlobalState(StateType) Self
}

However, this does not reduce any type constraints in the functions and structs that use these interfaces. In my experience the main benefit of this is to make the types more self-documenting, not to provide additional type safety.

The Blind Spot#

Everything example I’ve shown except for the last one has existed since Go 1.18, when type parameters were first added. So why didn’t I identify the proper solution previously?

There were two facets that I think blindsided me: the use of any within type constraints, and the ambiguous documentation around self-referential type constraints.

any Type Constraints#

It’s generally discouraged to use any as a parameter type, and recommended to use the most specific type possible in interfaces and APIs. Given that advice, I had attempted to avoid using any as a type constraint outside of situations where it was explicitly expected to support any type. Then, because it wasn’t possible to fully express these constraints within interface definitions prior to 1.26, I concluded that it wasn’t possible to write valid interface definitions for this situation.

But in fact, it’s perfectly fine to use any as a type constraint, and it doesn’t mean that the code you write will have to accept any as a variable type.

Think of a generic type for a data structure:

type MinHeap[T any] interface { ... }

This doesn’t imply that you’re actually going to specialize it with any. It’s just a constraint. Don’t be afraid of any in type constraints.

Ambiguity in documentation#

The original proposal for type parameters describes the above pattern as “mutually referencing type parameters”. And indeed, the only examples provided are cases where two type parameters reference each other. There are no examples of type parameters that reference themselves.

When I initially attempting to write a type constraint for the ImmutableValue interface above, I attempted to write the version with the recursive type constraints, which would not be supported until 1.26. When this was rejected, I incorrectly assumed that it was the self-referential nature of the type constraint that made it not allowed. In fact, it has always been allowed for a type constraint to reference its own type parameter.

Conclusion#

Go isn’t like other languages, and its worth learning its idioms and coding patterns. I still have my gripes: I don’t like how verbose this approach is, how it contains duplicate type constraints at every generic function that needs to operate on the defined types. But adopting the recommended coding styles has also helped to expand how I think about generic code in Go.

As always, if you have thoughts or if you want to tell me how wrong I am, feel free to join our Discord and shoot me a message.

Footnotes#

1. But you can still read it here, if you like. ↩