Greetings, fellow developers! It's me, Aleksei, working on improving productivity in Commmune, and I'm here today with the new part of insights. A few months have passed since my last article, and I'm excited to share my first-hand experience with GitHub Actions. We recently changed our CI/CD pipeline, and I must admit that Actions has proven to be a reliable and convenient tool. However, it's important to acknowledge that there are some limitations that are worth discussing. Join me as we delve into the challenges I encountered and explore potential workarounds and solutions together. Let's get started!
Managing monorepo with path filtering is hard
GitHub Actions has a great option to run workflows based on paths. If you want to run your tests only when something changes in a specific folder, you can do that:
name: workflow for frontend on: push: paths: - 'app/frontend'
name: workflow for backend on: push: paths: - 'app/backend'
But there are problems with this approach. It doesn't scale well:
- You need to duplicate your workflows for each path you have and you can end up with tons of workflow files
- No parallel execution of tests in a single worker
- No mechanism to determine the dependencies between projects. You have to add shared paths manually
- You can't reuse and test this functionality in your local environment
There is another option; you can use a great action called path-filter. It allows you to do pretty much the same thing, but at the step level of a single workflow job:
- uses: dorny/paths-filter@v2 id: changes with: filters: | backend: - 'backend/**' frontend: - 'frontend/**' - if: steps.changes.outputs.frontend == 'true' run: npm start frontend-tests # ...
Unfortunately, it still suffers from pretty much the same problems as the previous approach. So since we have a TypeScript monorepo, I decided to use nx and its approach of building a pipeline for your project that you can reuse and test locally. Now we have a single workflow file for everything that looks like this:
on: pull_request: types: [opened, synchronize, reopened, ready_for_review] # ... - name: Spell run: npx nx affected --target=cspell --parallel=3 - name: Linting run: npx nx affected --target=lint --parallel=3 - name: Formatting run: npx nx format:check --verbose
Nice and clean-looking workflow, and if you set up your workspace correctly, you don't need to manually declare the dependencies between your applications and libraries, Nx will do it for you. And of course, there are some drawbacks to this approach:
- It adds another rather large and complex technology to your project. It's not a big deal because if you manage monorepo, you probably have Nx or something similar like Lerna or pnpm workspaces
- You still need to configure Nx, especially if you're adding it to an existing project that doesn't have an Nx-specific structure. This is a pretty big topic that I'll probably cover in another article
GitHub Action's Artifact registry problems
One day I noticed that we had too many failed e2e tests. It was normal to retry after a test failure once in a while, but there was a completely different problem that had nothing to do with our code. To support Japanese text in e2e tests, we need to install fonts-noto, and we were using the Debian package for this. For a while, apt-get fonts-noto worked fine, it only took 15 seconds to download and install, no big deal. Then I noticed that sometimes this time increases drastically and it takes 10-20 minutes just to download. Look at this nightmare:
Need to get 307 MB of archives. After this operation, 778 MB of additional disk space will be used. Get:1 http://azure.archive.ubuntu.com/ubuntu jammy/main amd64 fonts-noto-core all 20201225-1build1 [12.2 MB] ... Fetched 307 MB in 12min 33s (407 kB/s) Selecting previously unselected package fonts-noto-core.
It was very painful since there was no good solution without some kind of workaround. GitHub support said they fixed it, and they had been fixing it over and over again for several weeks... And still, there were timeouts from time to time because of the long response time of the artifact registry, so I decided to just abandon the idea of using apt-get and put the whole font file into our monorepo. And it works perfectly for GitHub Action's Ubuntu runner, so don't hesitate to do the same:
- name: Install fonts-noto run: | sudo mkdir -p ~/.fonts/truetype sudo cp -R fonts ~/.fonts/truetype # Don't forget to update font cache sudo fc-cache -f -v
Github doesn't share cache between branches
Actually, this is not a bug, but a feature of GitHub Actions. It was completely unexpected for me, and I spent a lot of time trying to figure out why the cache miss was happening all the time. But it's actually quite useful because if you make a mistake in your cache key definition, it won't fetch the completely wrong one and just recalculate the cache. Even though it's useful, it's annoying that you can't configure this behavior.
If you really want to share the cache between branches, just calculate it for your default branch. Cache resolution will fall back to the default branch's cache if it can't find it for the current one.
GitHub standard token can't trigger other workflows
This is another not-obvious feature that should be highlighted in red at the top of the page like the cache one. For example, if you want to trigger some workflows on the automatically added label, you have to use the GitHub App token instead of the default GITHUB_TOKEN provided by workflow. The approach is well described in this article, so please take a look if you face the same problem.
There is no convenient way to trigger a workflow after the approval of the PR
To trigger the workflow after PR approval, you can use the following event:
pull_request_review: types: [submitted]
and further filter by
if ${{github.event.review.state === 'approved'}}
Unfortunately, it works only if you require exactly one approval for the PR before merging. Also, there is no way to trigger this event only for a specific branch, such as pull_request branches: [develop] and you pollute other branches' checks. At first, I just used an action to return the number of approvals and then run E2E if that number is >= 2.
Then I decided to make the workflow cleaner and moved this count into a separate workflow that adds an approved label to the pull request. Then you can just subscribe to the label using:
pull_request: types: [labeled]
The other problem is that when you press the update branch button on your PR's screen (which merges your target branch into the current branch), this removes all PR checks, but not the reviews. Even with a fully approved PR, you need to re-run the checks or somehow skip them. You also need to add synchronize to your pull_request types array to keep track of this event. Unfortunately, it causes another problem - the tests will be triggered in case you push the commit into approved PR because it takes some time to remove the approved label. So this solution isn't perfect, but it still feels better than the previous one.
Conclusion
GitHub Actions is a great way to set up your CI/CD for your GitHub repos, and the idea of doing everything related to the project in one place seems very convenient. However, it's still a bit raw, and you need to be willing to spend some time figuring out how to make it work the way you want. I hope this article helps you avoid some of the problems I encountered during implementation. Follow us for more articles about our experiences with GitHub Actions in the future!
If you want to help us with this difficult task and are not afraid of the actively changing startup environment, please check out our open positions. We are trying to make our environment more international and foreigner-friendly, and at this stage, your impact will be enormous. Let's build a great culture together!
