Background Tasks in Node.js: A survey with Redis.

Today I gave a talk at RedisConf in San Francisco entitled: Background Tasks in Node.js: A survey with Redis.

This was a talk I have wanted to give for a long time. As a DevOps engineer, there are a lot of ways to do everything, and when it comes to background processing, there are many ways to consume that job/worker/event/message.

This talk went though my personal evolution with background jobs, starting by doing them in-web-thread (PHP) and moving all the way up to event-bus systems. What was good and bad with each implementation, where and why did was it worth adding complexity, etc. These strategies discussed are:

• Foreground (in-line)
• Parallel (thread-ish)
• Local Messages (fork-ish)
• Remote Messages (*MQ-ish)
• Remote Queue (redis + Resque)
• Immutable Event Bus (Kafka-ish)

Learn about many of the possible background task strategies you can use in your app… and how they are better when you use node.js and redis!

My favorite part of the presentation was building Kafka-in-redis. This included using lua to make an atomic "ReadAndIncr" method to move the shared pointer when reading. You can see the LUA below, and how to use it in the codebase for the project.

Supporting Code here:

evantahler/background_jobs_node

Thanks!