Why We Started Git Workspace

Developing large projects can be a real challenge. There are lots of aspects that make the daily work on huge projects rather difficult. One - among many! - aspects is how to organize the structure of a project, i.e. how to break it down into smaller, better maintainable components and reuse them.

Our Requirements

Let’s start by checking what we wanted to achieve for our project:

  • Modularity: We wanted to be able to easily reuse parts of the project, so for new projects in the future, we would simply be able to take some of the (basic) components and include them as-is in the new project.

  • Ease of use: A system for composing a larger project from smaller components should be easy to use and understand - not everyone is an expert in configuration management, and there shall be no reason for them to become one.

  • Support for working in parallel on several components: Especially during the hot development phase, not only the main project needs to be changed, but also the other components it is made of need to be edited all the time.

  • Support for freezing a configuration: The other way round - once a certain version of the project is released, we wanted to be able to record the exact version of all the involved components so it is easy to reconstruct the source base a particular binary release has been built from.

  • Dependency management: Once we start breaking down a project into smaller components, these components in turn might also have own dependencies. We wanted to be able to have this kind of transitive dependencies.

  • Relative Dependency Paths: On top, we wanted to be able to use relative paths for dependencies. This is important when getting the main project from the server using varying URLs (e.g. https vs ssh of when some users might use a local mirror of the main git server).

Evaluation of Existing Tools

As everyone, we were lazy - in terms of development, this usually is a good thing!

So we checked if there were existing solutions that fit our needs. If there were tools fulfilling what we wanted, that it would be the easiest path just using one of them, right? So here is what we digged out:

git submodules

One of the first things that come to mind - and which we already used in other tools - were git submodules. git submodules have a lot of good things about them, that make them a strong first contender for building larger projects from several smaller components:

  • They are integrated into git since v1.5 - so they should be available defacto everywhere git is used as well.

  • Since submodules are by default included with a specific commit, each version of the parent repository has a well-defined configuration associated with it. Restoring a past configuration of the project hence is really easy.

  • Tools and platforms such as GitHub and GitLab have some support for submodules and e.g. automatically link to the referenced sub-repo in their web interface.

Seems good, right? However, we found the submodule approach also to bear some downsides as well:

  • Steep learning curve: Developers have to be aware how submodules work - especially in relation to git.

  • Especially merging can be tricky if conflicts in submodule revisions need to be resolved.

So, the obvious question was: Are there better alternatives available?

Google’s repo Tool

One other well-known tool for building workspaces from multiple git repositories is Google’s repo tool. It is a relatively simple tool written in Python which - based on a manifest assembles a workspace.

repo has some nice benefits:

  • It makes tracking synchronized branches across repositories easy.

  • In addition, it provides built-in commands for creating new branches across the entire project.

However, repo also comes with its own downsides as well:

  • It needs an extra repository where the manifest files are stored.

At least for us, repo didn’t seem to be the best choice to get the job done, so we had to venture forth…

Zephyr’s west Tool

This is another interesting option, and - on paper - it made a really great first impression! west is a tool developed in the scope of the Zephyr project. west promised to solve a lot of our requirements and also regarding their way of structuring a workspace fit quite nicely with what we had in mind:

  • west uses a manifest which is stored in the main project to include further dependencies.

  • It explicitly supports including specific revisions of a project - so including a project at a specific tag is possible without issues.

  • It also has some integration for git submodules, so the two of them can also be mixed (which is a nice, future option).

  • A manifest can import another one from the sub-projects. That way, west implements some kind of poor-man’s dependency management.

  • There is an option to create a frozen manifest where all the included projects are fixed at a particular revision - this is important when creating a release to have a well-defined configuration of the project.

The list of advantages is quite long and west really looked like a great choice at first. However, after giving it a try, we noticed downsides as well:

  • Whenever a west update is run, the tool will check out the sub-projects at a detached head state instead of a branch. That makes it difficult to really work in parallel on a main project and one or more of its dependencies - something we really need.

  • In addition, west has no option to include other projects using relative paths. This however is important if the main project can be accessed using various different URLs (e.g. when cloning via both ssh and https is possible or when a git mirror server is used).

  • Another thing we didn’t like (but this is really something one could ignore): west can actually do way more than managing workspaces - there are e.g. options for adding new commands to it by providing additional information in the manifest files. In our opinion, its better to follow the UNIX philosophy here: A tool should do only one thing and it should do this thing in a great way.

So, unfortunately, west was also out…

Using A Package Manager

Another option we considered was using a package manager like conan. In this scenario, instead of building a larger workspace consisting of multiple git repositories, we would break the project down into its components and consider every component as a standalone project which creates a (possibly binary) package as its output.

If a project requires other components, it would consume their released packages instead of including them in source form.

This approach has one obvious advantage: Reduced CI/CD run-times. Especially if there are parts of the project that rarely change, we avoid building them all over again for each push we make to the git server.

However, this approach as well has its downsides:

  • First of all, not all projects have a structure where individual parts can be compiled on their own. A notable example is hardware design. In that case, we don’t have any performance benefits.

  • And second, a package based workflow does not necessarily make working on several components in parallel easy. While conan has some features for making a single package editable within a larger workspace, this feature is still not quite easy to use and can be confusing at times.

Monorepos

Lastly, we also revisited the option putting everything in one large repository - this approach is often coined Monorepo. Nowadays, there are some tweaks and tricks available to make working with such a setup easier and at least the project configuration would be really easy (as one commit in the main project fixes all the files that are there). But obviously, this approach had too many downsides:

  • The single repository grows very large - operations like cloning will then take ages.

  • It’s all or nothing: Either a developer has access to the repo or not. There is no in between.

  • And most obviously, there is no simple way to reuse parts of the project. Besides simple copy&paste, there are some neat tricks git can play here, but this really is nothing for a daily workflow.

So, it seemed we had to get our hands dirty…

The Birth of Git Workspace

After doing our homework and checking if we simply can use another existing tool, we came to the conclusion that we had to invest some development effort and bring up our own tool. Git Workspace was born!

If you carefully study the results of our research given above, you might find that we actually were pretty fond of west. So it comes to no surprise that Git Workspace is built in a similar way and even tries to be compatible with west as far as possible. In fact, the manifest files of west are to some degree compatible with Git Workspace and we try to behave similarly where possible. However, we also want to close the gap and implement the features we think are missing in west.

Comparison Matrix

So, long story short: Here is a matrix of features vs the different tools we evaluated plus Git Workspace itself.

Comparison

git submodules

repo

west

Monorepo

git ws

Reusable Components

Ease of Use

Editable Components

[1]

Freezing Configurations

Transitive Dependencies

[2]

Relative Dependency Paths

Symlink/Copyfile-Support

Please note that we don’t claim that Git Workspace is the best choice for everyone: As usual, when looking for a tool, do your homework and make up your mind about your requirements and check them against the various available tools. Git Workspace might just be right for you - but chances are that depending on your concrete workflow one of the other tools or approaches simply works better.