Recently we had a JAM 🍺 session at GOJEK organised by ODPF (Open Data Platform community) for stencil, A Protobuf schema registry. Usually, we organise these JAM sessions every month for two days a week.

In a JAM session, Everyone is free to pitch any idea which they are in love with or pick any existing issue as a starting point. Also you are free to pair with anyone with their idea if you don't have one.

This time, I was pairing with one of my colleague Riteek Srivastav on a idea of

Displaying dependency graph of a proto file descriptor

We will go through how this feature will help us to understand proto files but before that let's understand what is stencil and why it's needed in the first place.

What is Stencil?

Stencil, is a dynamic Protobuf schema registry which helps you to work with your proto schema definitions at runtime.

Why do we need services like Stencil?

In order to work with Protobuf (a Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data), you need to generate source code for each change in proto definitions. By doing this, you will feel the pain of updating the dependencies version of all dependent services/packages. It's totally fine at a small scale but at the scale of GOJEK, it's really hard to maintain and update dependencies of proto.

To reduce the pain and effort, We want to support services, load their proto schema definition at runtime.

Protobuf allows you to define your whole proto file using FileDescriptorSet (as defined in google/protobuf/descriptor.proto) which contains list of FileDescriptorProtos. FileDescriptorProtos is used to represent each relevant proto file. Now, you can use this FileDescriptorSet on the client at runtime.

Dependency graph of a proto file descriptor? 🤔

In order to make changes in any proto file we need to understand what the file depends on. Figuring out a list of dependencies is hard for proto files due to recursive nature.

So, We build a cli tool to generate dependency graph of a proto file descriptor

stencil graph --host="stencilhost" --namespace="proto" --name="descriptor" --version="latest"

which result into a .dot file. It can be used to visualize via graphviz tools by generating .svg and/or .png images.

dot -Tpng proto_vis.dot -o outfile.png
OR
dot -Tsvg proto_vis.dot -o outfile.svg

You can also use WebGraphviz to view your .dot file in the Browser

This is a dependency graph generated for Optimus protos from proton.

Conclusion

It was fun to be part of the JAM session. I learned a lot about stencil, Protobuf and it's runtime use cases.

If you are interested to use/contribute to any of our services then you can join the ODPF community slack channel to ask any questions. Also if you have any questions or suggestions, feel free to add a comment or contact me on twitter.