engineering grouparoo node.js typescript
Thu Jan 21 2021 - Originally posted at https://www.grouparoo.com/blog/defering-side-effects-in-node
At Grouparoo, we use Actionhero as our Node.js API server and Sequelize for our Object Relational Mapping (ORM) tool - making it easy to work with complex records from our database. Within our Actions and Tasks, we often want to treat the whole execution as a single database transaction - either all the modifications to the database will succeed or fail as a unit. This is really helpful when a single activity may create or modify many database rows.
Take the following example from a prototypical blogging site. When a user is created (
In this example, we:
This works as long as nothing fails mid-action. What if we couldn’t update the user’s password? The new user record would still be in our database, and we would need a try/catch to clean up the data. If not, when the user tries to sign up again, they would have trouble as there would already be a record in the database for their email address.
To solve this cleanup problem, you could use transactions. Using Sequelize’s Managed Transactions, the run method of the Action could be:
Managed Transactions in Sequelize are very helpful - you don’t need to worry about rolling back the transaction if something goes wrong! If there’s an error
throw-n, it will rollback the whole transaction automatically.
While this is safer than the first attempt, there are still some problems:
transactionobject to every Sequelize call
user.updatePassword()... that probably needs to write to the database, right?)
Sending the email as-written will happen even if we roll back the transaction because of an error when creating the new post… which isn’t great if the user record wasn’t committed! So what do we do?
The solution to our problem comes from a wonderful package called
cls-hooked. Using the magic of
AsyncHooks, this package can tell when certain code is within a callback chain or promise. In this way, you can say: "for all methods invoked within this async function, I want to keep this variable in scope". This is pretty wild! If you opt into using Sequelize with CLS-Hooked, every SQL statement will check to see if there is already a transaction in scope... You don't need to manually supply it as an argument!
CLS: "Continuation-Local Storage"
Continuation-local storage works like thread-local storage in threaded programming, but is based on chains of Node-style callbacks instead of threads.
There is a performance penalty for using
cls-hooked, but in our testing, it isn’t meaningful when compared to
await-ing SQL results from a remote database.
cls-hooked, our Action's run method now can look like this:
Ok! We have removed the need to pass
transaction to all queries and sub-methods! All that remains now is the
user.sendWelcomeEmail() side-effect. How can we delay this method until the end of the transaction?
Looking deeper into how
cls-hooked works, we can see that it is possible to tell if you are currently in a namespace, and to set and get values from the namespace. Think of this like a session... but for the callback or promise your code is within! With this in mind, we can write our run method to be transaction-aware. This means that we can use a pattern that knows to run a function in-line if we aren’t within a transaction, but if we are, defer it until the end. We’ve wrapped utilities to do this within Grouparoo’s CLS module.
With the CLS module you can write code like this:
You can see here that once you
async function, you can defer the execution of anything wrapped with
CLS.afterCommit() until the transaction is complete. The order of the
afterCommit side-effects is deterministic, and
in-line happens first.
You can also take the same code and choose not apply
CLS.wrap() to it to see that it still works, but the order of the side-effects has changed:
Now that it is possible to take arbitrary functions and delay their execution until the transaction is complete, we can use these techniques to make a new type of Action and Task that has this functionality built in. We call these
CLSTask. These new classes extend the regular Actionhero Action and Task classes, but provide a new
runWithinTransaction method to replace
run, which helpfully already uses
CLS.wrap(). This makes it very easy for us to opt-into an Action in which automatically runs within a Sequelize transaction, and can defer it's own side-effects!
Putting everything together, our new transaction-safe Action looks like this:
If the transaction fails, the email won’t be sent, and all the models will rolled back. There won't be any mess to clean up 🧹!
cls-hooked module is a very powerful tool. If applied globally, it unlocks the ability to produce side-effects throughout your application worry-free. Perhaps your models need to enqueue a Task every time they are created... now you can if you
cls.wrap() it! You can be sure that task won’t be enqueued unless the model was really saved and committed. This unlocks powerful tools that you can use with confidence.