Why We Started Git Workspace
Developing large projects can be a real challenge. There are lots of aspects that make the daily work on huge projects rather difficult. One - among many! - aspects is how to organize the structure of a project, i.e. how to break it down into smaller, better maintainable components and reuse them.
Our Requirements
Let’s start by checking what we wanted to achieve for our project:
Modularity: We wanted to be able to easily reuse parts of the project, so for new projects in the future, we would simply be able to take some of the (basic) components and include them as-is in the new project.
Ease of use: A system for composing a larger project from smaller components should be easy to use and understand - not everyone is an expert in configuration management, and there shall be no reason for them to become one.
Support for working in parallel on several components: Especially during the hot development phase, not only the main project needs to be changed, but also the other components it is made of need to be edited all the time.
Support for freezing a configuration: The other way round - once a certain version of the project is released, we wanted to be able to record the exact version of all the involved components so it is easy to reconstruct the source base a particular binary release has been built from.
Dependency management: Once we start breaking down a project into smaller components, these components in turn might also have own dependencies. We wanted to be able to have this kind of transitive dependencies.
Relative Dependency Paths: On top, we wanted to be able to use relative paths for dependencies. This is important when getting the main project from the server using varying URLs (e.g.
https
vsssh
of when some users might use a local mirror of the maingit
server).
Evaluation of Existing Tools
As everyone, we were lazy - in terms of development, this usually is a good thing!
So we checked if there were existing solutions that fit our needs. If there were tools fulfilling what we wanted, that it would be the easiest path just using one of them, right? So here is what we digged out:
git submodules
One of the first things that come to mind - and which we already used in other tools - were git submodules. git submodules
have a lot of good things about them, that make them a strong first contender for building larger projects from several smaller components:
They are integrated into
git
sincev1.5
- so they should be available defacto everywheregit
is used as well.Since
submodules
are by default included with a specific commit, each version of the parent repository has a well-defined configuration associated with it. Restoring a past configuration of the project hence is really easy.Tools and platforms such as GitHub and GitLab have some support for
submodules
and e.g. automatically link to the referenced sub-repo in their web interface.
Seems good, right? However, we found the submodule
approach also to bear some downsides as well:
Steep learning curve: Developers have to be aware how
submodules
work - especially in relation togit
.Especially merging can be tricky if conflicts in
submodule
revisions need to be resolved.
So, the obvious question was: Are there better alternatives available?
Google’s repo
Tool
One other well-known tool for building workspaces from multiple git
repositories is Google’s repo tool. It is a relatively simple tool written in Python which - based on a manifest assembles a workspace.
repo
has some nice benefits:
It makes tracking synchronized branches across repositories easy.
In addition, it provides built-in commands for creating new branches across the entire project.
However, repo
also comes with its own downsides as well:
It needs an extra repository where the manifest files are stored.
At least for us, repo
didn’t seem to be the best choice to get the job done, so we had to venture forth…
Zephyr’s west
Tool
This is another interesting option, and - on paper - it made a really great first impression! west is a tool developed in the scope of the Zephyr project. west
promised to solve a lot of our requirements and also regarding their way of structuring a workspace fit quite nicely with what we had in mind:
west
uses a manifest which is stored in the main project to include further dependencies.It explicitly supports including specific revisions of a project - so including a project at a specific tag is possible without issues.
It also has some integration for
git submodules
, so the two of them can also be mixed (which is a nice, future option).A manifest can import another one from the sub-projects. That way,
west
implements some kind of poor-man’s dependency management.There is an option to create a frozen manifest where all the included projects are fixed at a particular revision - this is important when creating a release to have a well-defined configuration of the project.
The list of advantages is quite long and west
really looked like a great choice at first. However, after giving it a try, we noticed downsides as well:
Whenever a
west update
is run, the tool will check out the sub-projects at a detached head state instead of a branch. That makes it difficult to really work in parallel on a main project and one or more of its dependencies - something we really need.In addition,
west
has no option to include other projects using relative paths. This however is important if the main project can be accessed using various different URLs (e.g. when cloning via bothssh
andhttps
is possible or when agit
mirror server is used).Another thing we didn’t like (but this is really something one could ignore):
west
can actually do way more than managing workspaces - there are e.g. options for adding new commands to it by providing additional information in the manifest files. In our opinion, its better to follow the UNIX philosophy here: A tool should do only one thing and it should do this thing in a great way.
So, unfortunately, west
was also out…
Using A Package Manager
Another option we considered was using a package manager like conan. In this scenario, instead of building a larger workspace consisting of multiple git
repositories, we would break the project down into its components and consider every component as a standalone project which creates a (possibly binary) package as its output.
If a project requires other components, it would consume their released packages instead of including them in source form.
This approach has one obvious advantage: Reduced CI/CD run-times. Especially if there are parts of the project that rarely change, we avoid building them all over again for each push we make to the git
server.
However, this approach as well has its downsides:
First of all, not all projects have a structure where individual parts can be compiled on their own. A notable example is hardware design. In that case, we don’t have any performance benefits.
And second, a package based workflow does not necessarily make working on several components in parallel easy. While
conan
has some features for making a single package editable within a larger workspace, this feature is still not quite easy to use and can be confusing at times.
Monorepos
Lastly, we also revisited the option putting everything in one large repository - this approach is often coined Monorepo. Nowadays, there are some tweaks and tricks available to make working with such a setup easier and at least the project configuration would be really easy (as one commit in the main project fixes all the files that are there). But obviously, this approach had too many downsides:
The single repository grows very large - operations like cloning will then take ages.
It’s all or nothing: Either a developer has access to the repo or not. There is no in between.
And most obviously, there is no simple way to reuse parts of the project. Besides simple copy&paste, there are some neat tricks
git
can play here, but this really is nothing for a daily workflow.
So, it seemed we had to get our hands dirty…
The Birth of Git Workspace
After doing our homework and checking if we simply can use another existing tool, we came to the conclusion that we had to invest some development effort and bring up our own tool. Git Workspace was born!
If you carefully study the results of our research given above, you might find that we actually were pretty fond of west
. So it comes to no surprise that Git Workspace is built in a similar way and even tries to be compatible with west
as far as possible. In fact, the manifest files of west
are to some degree compatible with Git Workspace and we try to behave similarly where possible. However, we also want to close the gap and implement the features we think are missing in west
.
Comparison Matrix
So, long story short: Here is a matrix of features vs the different tools we evaluated plus Git Workspace itself.
|
|
|
Monorepo |
|
|
---|---|---|---|---|---|
Reusable Components |
✅ |
✅ |
✅ |
➖ |
✅ |
Ease of Use |
➖ |
✅ |
✅ |
✅ |
✅ |
Editable Components |
➖ [1] |
✅ |
➖ |
✅ |
✅ |
Freezing Configurations |
✅ |
✅ |
✅ |
✅ |
✅ |
Transitive Dependencies |
➖ [2] |
➖ |
✅ |
➖ |
✅ |
Relative Dependency Paths |
✅ |
✅ |
➖ |
➖ |
✅ |
Symlink/Copyfile-Support |
➖ |
✅ |
➖ |
➖ |
✅ |
Please note that we don’t claim that Git Workspace is the best choice for everyone: As usual, when looking for a tool, do your homework and make up your mind about your requirements and check them against the various available tools. Git Workspace might just be right for you - but chances are that depending on your concrete workflow one of the other tools or approaches simply works better.