We're using Go to write Dolt, the world's first and only version-controlled SQL database.

Like any Go program that deals with data, we often find ourselves in need of high-level functions that transform data from one form into another. Go has historically been very light on collections packages akin to Apache Commons, in large part due to lack of support for generic types.

But now generics have launched, and official support for collection convenience functions in the standard library followed closely behind in 1.23, in the maps and slices packages. In this blog, we'll take a look at a few of the more interesting functions and learn how to use them.

Iterator methods

The slices and maps package export several convenience methods for returning iterator functions over collections. Briefly:

• slices.All() returns an iterator over a slice in its underlying order.
• slices.Backwards() returns an iterator over a slice in the opposite direction.
• maps.All() returns an iterator over a map's keys and values
• maps.Keys() returns an iterator over a map's keys
• maps.Values() returns an iterator over a map's values

All these methods return range iterators that you can use with the range keyword, as we demonstrated in our blog earlier this year. If you haven't really wrapped your head around range iterators yet, we recommend reading it to clear up the concept. They're not really hard to understand, but the method signatures make them seem like they are.

You can use range iterators with the range keyword like this:

	is := []int{0, 10, 20, 30, 40}
    for i, v := range slices.All(is) {
        fmt.Printf("%d: %v\n", i, v)
    }

But just using them for this simple purpose doesn't really add much, since maps and slices already work with the range keyword without these new library functions. The real power of these methods is in combination with the other functions in the library, many of which accept these iterators.

Creating and populating slices and maps

The slices and maps packages both have methods called Collect, which let you create a new collection from an iterator. Here's what it looks like in action:

	is := []int{0, 10, 20, 30, 40}
	m := maps.Collect(slices.All(is))

The equivalent code to do this with a loop looks like this.

	m := make(map[int]int)
	for i, v := range is {
		m[i] = v
	}

So we go from 4 lines of code to a one-liner to create and populate our map, which is very convenient when you happen to have a range iterator around (and want to map a slice's indexes to its values).

Sorting map elements

Go randomizes the iteration order of map elements as a deliberate policy, to prevent you from relying on it. But sometimes, you really want a consistent ordering of a map's elements. Now there's an easy way to get one.

	sortedKeys := slices.Sorted(maps.Keys(m))
	sortedVals := slices.Sorted(maps.Values(m))

Without using the slices and maps packages, you would have to write this code:

	keys := make([]int, 0, len(m))
	for k := range m {
		keys = append(keys, k)
	}
	sort.Ints(keys)

	values := make([]int, 0, len(m))
	for _, v := range m {
		values = append(values, v)
	}
	sort.Ints(values)

This turns a 5-line block of code into a one-liner. Pretty handy!

Copying maps and slices

Prior to Go 1.23, if you wanted to make a copy of a slice or a map, you had to use make and write a loop (or use the copy built-in for a slice). Now we get Clone() for both slices and lists:

	cloned := slices.Clone(is)
	clonedMap := maps.Clone(m)

These one-liners replace these blocks:

	cloned := make([]int, len(is))
	copy(cloned, is)

	clonedMap := make(map[int]int)
	for k, v := range m {
		clonedMap[k] = v
	}

Similarly, maps.Copy() now provides a one-line way to insert all the elements from one map into another, similar to the built-in copy for slices.

	maps.Copy(clonedMap, m)
	
	// without the maps package:
	// for k, v := range m {
	// 	clonedMap[k] = v
	// }

Finally, the slices.Concatenate() method is a great way to create a new slice from the contents of other ones.

	concatenated := slices.Concat(is, cloned, slices.Collect(maps.Keys(m)))	
	concatenated = slices.Concat(is, is, is)	

Think of the slices.Concatenate() as a better version of append for those times when you know you want to create a new slice, instead of sometimes re-using an existing one. It replaces this much more verbose code:

    // concatenated := slices.Concat(is, cloned, slices.Collect(maps.Keys(m)))	
	concatenated := make([]int, 0, len(is) + len(cloned) + len(m))
	concatenated = append(concatenated, is...)
	concatenated = append(concatenated, cloned...)
	for k := range m {
		concatenated = append(concatenated, k)
	}

    // 	concatenated = slices.Concat(is, is, is)
    concatenated = append([]int{}, appen(is, append(is, is...)...)...)

The missing methods: slices.Map() and slices.Filter()

In our first example, we showed how to create a new map from the indexes of a slice to its values:

	is := []int{0, 10, 20, 30, 40}

	m := maps.Collect(slices.All(is))

This is cute, but it's rarely what we want. What we actually want is a way to create a map from the values in a slice by applying some transform function. This would be slices.Map(), but it doesn't exist.

We can write our own:

func sliceToMap2[K comparable, V any, S any](s []S, mapFn func(S) (K, V)) map[K]V {
	m := make(map[K]V)
	for _, k := range s {
		key, val := mapFn(k)
		m[key] = val
	}
	return m
}

stringMap2 := sliceToMap2(is, func(v int) (string, string) {
	return fmt.Sprintf("key %d", v), fmt.Sprintf("value %d", v  * 10)
})

Because our slice2Map2 function is generic, we can use it on any kind of slice to produce any kind of map. The code above yields the following map contents:

string map 2: map[key 0:value 0 key 10:value 100 key 20:value 200 key 30:value 300 key 40:value 400]

But because of the anonymous func, the space savings here aren't compelling unless you pass it a func defined statically somewhere else. Depending on your code base, that might be a good idea if you find yourself doing this a lot.

Similarly, the slices package contains no Filter() method, and it was explicitly rejected for being too complex. Rob Pike himself chimed in to strike down the suggestion.

You can just use a for loop, which is more flexible in general. I am not being facetious; the filter operation tends to obscure allocation and overhead, and also tends to be overused. Although I may be in the minority, I do not believe this would be a wise addition to the standard library.

So we don't get a slices.Filter() in the standard library, but writing your own is easy:

func filtered[S any](s []S, fn func(S) bool) iter.Seq[S] {
	return func(yield func(s S) bool) {
		for _, v := range s {
			if fn(v) {
				if !yield(v) {
					return
				}
			}
		}
	}
}

filteredSlice := slices.Collect(filtered(is, func(v int) bool {
	return v > 20
}))

This has the same problem as our slice2map2 example: the anonymous func takes up a lot of space. If you want this to be a true one-liner like is possible in other languages, you'll need to define your predicate function elsewhere so you can pass it on the same line.

Conclusion

Want to talk about Go library functions? Or maybe you're curious about Dolt, the world's first version-controlled SQL database? Come by our Discord to talk to our engineering team and meet other Dolt users.