Last year, I saw José Valim give his keynote at the ElixirLive conference in Warsaw, where he talked about the motivation for his new Elixir libraries: GenStage and Flow. Even though I heard about those before, it was the keynote when I “got” what the libraries were good for and why they were neat - and I decided to play around with them.
When I came back to London I started working on a small project that would make use of GenStage and Flow - Crawlie the crawler. Since then the project took shape and allowed me to learn a bit about concurrent event processing pipelines in Elixir. As I’ll probably be ramping down the work around Crawlie now to focus on other things (I have my eye on Riak Core and architecting distributed apps in Elixir in general) I thought it might be a good idea to share what I have learned with others who want to try GenStage and Flow.
Here are the slides if you want to click along or copy the code:
It is a bit dry, but if you want to get started on GenStage and Flow anyways, it might help you bootstrap yourself quicker and avoid some pitfalls. Also: only after watching the recording of the talk afterwards I have noticed how plentiful and distracting all the “umm”s and “ah”s are - I’m going to try to work on it :)
There’s many more recorded talks on the meetup’s SkillsMatter site, touching a diverse set of subjects, from the basics to some really advanced Ecto Sandbox implementation details - have a look around while you’re there.
As always, I’m happy to hear any and all feedback :)