grouparoo node.js typescript
2020-05-07 - Originally posted at https://www.grouparoo.com/blog/grouparoo-monorepo-deployment
Coming from more traditional web & app development, I’m a big fan of git-flow style workflow. Specifically the following features:
featurebranches, an integration branch where features are merged together (usually called
main), and finally the "live" branch that customers are using (often called
mainbranch is always deployable (and should be deployed automatically with a CI/CD tool)
Setting up processes and tools to automate and enforce this workflow is possible with tools like CircleCI, Github Actions, and even Fastlane + CodePush for mobile apps. However, since Grouparoo is building software that our customers run themselves, what does “pushing to production” really mean? What do automated releases look like? This blog post outlines our processes and the tools we use to automate our deployments and builds.
Our 4 major steps are:
Grouparoo leverages the Node.js and NPM ecosystems to manage distribution to our customers. Our open-source software is distributed via the public NPM repository, and our paid plugins via NPM Enterprise. This means that all our customers need to do in order to obtain Grouparoo is create a
package.json and keep it up to date (more detail here).
package.json will have it’s versions locked in place with npm (or yarn), but can be easily updated via
npm update, as the newest version of each package requested is
latest rather than a specific version.
The backbone of any good automated workflow is a robust test suite. You need to be sure that your new code works the way you expect, and hasn’t broken anything. We run our tests on CirleCI, and make use of Jest and man other tools. I’ll talk about our test suite in more detail in a later post, but we have a test suite for every package we publish. The Grouparoo Monorepo is a collection of many inter-related packages which we manage together via Lerna. Lerna helps you keep all of your versions & packages in sync, and more importantly, rely on each-other while developing them! A change in one package might effect the rest, so we test them all in concert.
Since Grouparoo is an Open Source project, you can check on the test suite of our
main branch here: CircleCI At the moment we are:
feature branch has been merged into the
main branch, we want to immediately deploy it onto a staging server so we can do acceptance testing and share it with our partners. At this step, we use Heroku’s Github Integration to deploy our
main branch on any change, after the tests all pass of course.
We use Lerna here to build every project within the monorepo, but running the project within the monorepo has some caveats. Specifically, since Lerna will use symlinks to relate projects within the monorepo to each other, the paths the project sees are not the same as when it will be installed via a normal
npm install. The app we run on staging looks a lot like our client example above, except that was sprinkle the environment variable
GROUPAROO_MONOREPO_APP around (example here).
GROUPAROO_MONOREPO_APP to change its require paths for its peer dependencies, mainly the other Grouparoo plugins. Rather than
project/node_modules/@grouparoo/plugin , the runtime within a Lerna project is more like
root/packages/@grouparoo/plugin. We’ve isolated the majority of plugin loading to this module. In this way, we can closely emulate the experience of installing Grouparoo and related plugins locally without needing to publish every version to NPM. We use a similar paradigm when developing locally.
Once we’ve got our new features deployed on our staging servers, we want to release our NPM packages in a way that our customers can try out. For us, this means a weekly release of our packages every Friday. We once again use Circle CI to run our test suite on a schedule:
This mode of running our CI suite include an extra job called “publish”. Assuming again that our tests all pass, the publish command does a few things which you can see here.
lerna version prerelease --preid alphawould yield a version like
v0.1.2-alpha.4. We create a new git tag for the release and push that to Github
lerna-changelogpackage to automatically create our release notes from our merged pull requests & push those to Github along with our new git tag
There are a number of CI secrets we need to manage access to NPM and Github, but they can all be stored in CircleCI’s secrets management tool. Of note, there is at this time no way to automate (or skip) a 2FA token for publishing to NPM. To overcome this, we’ve created a user who can only publish from CI which doesn’t use 2FA.
Now, our customers can opt into our alpha releases by changing their dependencies from
next in their
package.json file. When a normal package is published to NPM, it automatically has the
latest tag, and that’s what will be installed wit a normal
npm install @grouparoo/core. However, you can publish your packages to any other tag you want to create parallel distribution channels.
The last stage of our release process is to publish the
latest (read: normal channel) NPM packages. We do this by a having a human make the call that we are ready to do this by merging the release candidate (from
main or another branch) into the
stable branch. This will then run the same
publish CI command as with our prerelease, but with a few changes:
lerna version patchwould take our last pre-release version like
v0.1.3We create a new git tag for the release and push that to Github
mainbranch so we are ready for the next round of
alphaprereleases to start.
Those are the steps we use to continuously deliver Grouparoo to our customers. We use NPM release tags to regularly publish an
alpha tagged pre-release every week, and have a human review process for our
latest stable releases.
The latest version of Grouparoo is just an