Continuous Fuzzing with Go at Dgraph
Dgraph’s flagship product is a native graph database written purely in Go. For any database product, the ...
Learn more
A little bit over a month I joined Dgraph Labs, a really cool tiny startup based (mostly) in San Francisco and Bangalore building what we believe is the next generation of Graph Databases. Marketing spiel aside, I joined because the company is coming up with some really cool tech that benefits more than Graph Database users since all the pieces of the stack are open-sourced on github.com/dgraph-io.
One of these pieces is BadgerDB, a key-value store written completely in Go following the ideas of RocksDB and LSM trees. BadgerDB has many users (e.g. go-ipfs, emitter, riot, and many more) but one of the most important one is indeed our database server Dgraph. Now, this is important because we care about our community of users: they always are the main stakeholder we keep in mind. In our case, though, our biggest user is ourselves so our own engineers become also clients that we need to delight.
This post is a report on how we got ourselves into a situation in which Go Modules were simply not good enough for us, and how in the process of understanding why we figured out that our issue might be probably with Semantic Versioning itself.
For reasons that are not relevant to this post, BadgerDB has not had a release in over a year. This does not mean that BadgerDB has not changed over this time, it indeed has changed a lot, but we have not really cut a release or tagged a version and published it as something people should be using.
This had to change and I, as VP of Product, was responsible for getting us back to a regular release pace which would allow us to release new features quickly while avoiding any impact on our current users. With that in mind, I started reviewing the changes since our last release and the ones that were coming up in the near future.
We had two different kinds of backward-incompatible changes: API changes and data format changes. API changes had already been committed to master and many of our users (including Dgraph itself) were already using this new API not fully compatible with the previous release v1.5.3. The data changes had not landed yet, but they were necessary for the next release of Dgraph and therefore imminent.
Following Semantic Versioning principles, I decided we should release the following versions after v1.5.3:
v2.0.0 which would include all of the API breaking changes. This version would support all of those already using master, which includes Dgraph 1.0.v2.1.0 which would include also the data format changes. This version’s API would be backward-compatible with v2.0.0 but files written with v2.0.0 would not be readable from v2.1.0 and vice-versa.This felt like the right solution from a Semantic Versioning point of view, so I announced the plan and got started with the release process. Then Go Modules, which we planned on supporting, appeared and interrupted the whole process because it caused us to force upon our users a new import path for BadgerDB v2.0.0. The import path would be ”github.com/dgraph-io/badger/v2", note that v2 at the end. This would mean that for our users already on master, adopting v2.0.0 would require changing every single import statement instead of simply marking on their dependency manager that they were now using v2.0.0.
This seemed to be a roadblock and I wrote about why we were deciding to postpone adding support for Go Modules on Badger until a point where our concerns had been addressed.
My complaint was and still is that forcing users to include the major version of a package in their import path is unnecessary. I do understand the benefits of having different import paths for different versions, but there are also many drawbacks to this decision so it should be up to package publishers to make it rather than the Go team.
I kept thinking about how to get out of this thorny situation since releasing a version 2 of our library without support for Go modules would put us in a situation where adopting Go modules would become almost impossible. We would have to support two different import paths for v2, one for those using Go modules and one for the rest. The other option was, of course, to hope the Go team would change the design of Go Modules making the import path rewrite an optional step … but I’m a realistic person and it seemed that boat (probably) left port long ago.
At some point, I was discussing with Manish (the CEO of Dgraph Labs) about my plans and concerns and he came out with an interesting point: is Semantic Versioning necessarily about APIs? If you visit semver.org you will see that, indeed, all of the changes they take into account are those impacting the API of a library, but what about the other ways a library can break?
In our case, there are two different change kinds: API changes and serialization changes. They both are backward-incompatible but the cost of fixing one vastly outweighs the other.
Migrating to a new API, in our case, is a pretty straight forward change. Some functions might have changed arguments or maybe even replaced byfor a completely new API. These changes, no matter how tedious, are pretty straightforward to fix especially in a strongly typed statically compiled language like Go.
Migrating to a new serialization format, though, can be much time and cost consuming as it requires transforming your whole dataset, often measured in TB, from the previous version into the new one. This takes time, and while the process is ongoing you might need still be serving traffic so the migration strategies become much more advanced than a simple code deployment.
So in our case, it turns out that the major breakage we have is not API changes, but serialization format changes! So … what if we adapted Semantic Versioning to take that into account?
Our modified version of Semantic Versioning is called Serialization Versioning. You can find a full description of the reasoning and details of the naming schema in here and here, but let me summarize it here too.
Given a version number MAJOR.MINOR.PATCH, increment the:
Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.
Taking into account this new way of interpreting Semantic Versioning, our release plan became quite different and in so adapted to what Go Modules requires in a much more natural way!
My original plan was to release v2.0.0 containing all of the API breaking changes, followed by v2.1.0 containing the serialization format changes. This had three main drawbacks:
The most expensive upgrade seemed like the easiest one. Migrating from v1.5.3 to v2.0.0 is actually quite straight forward and it can be done in a single Pull Request, but migrating from v2.0.0 to v2.1.0 can mean weeks or more of engineering efforts to migrate your dataset without causing any downtime.
Following Go Modules advice, the new versions of Badger needed to have v2 at the end of their import path. This meant that our users already using the newer API from master would have to rewrite their import paths for no real benefit.
It also felt like releasing v2 and v3 at the same time would be confusing to our users and would somehow imply that the release pace had been accelerated unnecessarily. This is why v2.1.0 felt like a better option to follow v2.0.0 even though they were clearly incompatible.
These three issues actually disappear once we consider our versioning focus to be on serialization formats rather than APIs. Our plan is now to release these two versions:
v1.6.0 includes all the API changes but no serialization one. A system using v1.5.3 for months can simply migrate its code to v1.6.0 and start benefiting from the newest features without having to worry about a data migration strategy. Once it compiles, it works.
v2.0.0 has actually the same API as v1.6.0 but a completely different serialization format. Migrating from v1.6.0 to v2.0.0 is an endeavor that should be considered carefully.
So the cost of the changes is again aligned with the MAJOR, MINOR, PATCH schema, which solves our first concern. But what about the second concern? Since v1.6.0 corresponds to our current master, our users depending on master can migrate to using v1.6.0 without any hiccups - no need to rewrite any import path either since this is still a v1.x version.
And as part of the migration to v2, we can now force our users to adopt the new import paths, since it actually has a very clear benefit and it is part of a much larger migration process. The clear benefit, at least from our point of view, is that having two different import paths for these two versions will allow us to write a tool that will migrate files from v1 formatting to v2, which would have been otherwise impossible if both versions used the same import path, as v2.0.0 and v2.1.0 would have in our original plan.
Releasing Open Source Software is hard and finding a solution that meets everyone’s needs is almost impossible. That said, Go Modules does provide a really good solution to 99% of the use cases - maybe 100% if instead of blaming Go Modules for our issues you simply point at Semantic Versioning not being the right way to version Database libraries.
I wanted to thank everyone in the Go community that participated in this conversation, especially Andrew Gerrand and Russ Cox which took their time to participate in our discuss.dgraph.io post where I originally shared my plans.
Oh, by the way, Dgraph is writing some really cool Go code to solve distributed systems problems … and yes, we’re hiring! Hit me up if you’re interested in joining us as a Software Engineer, Developer Advocate, Technical Writer, or Product Manager roles. Either in SF or Bangalore.
