How to use data spanning multiple data sources in Elixir
Unleashing the power of the Ecto preload functions
Below is a reprint of the article I wrote for the Rekki Medium page.
REKKI builds tools that help people along the restaurant supply chain do their jobs better.
We have a free mobile app that lets restaurants order and chat with suppliers, and a web-based tool for suppliers that helps them process orders, manage product codes and catalogues, and communicate more easily with their customers. The majority of REKKI’s backend is written in Elixir, working hand in hand with services written in Go and Node. The Elixir services handle most of what the user sees in the app like the real-time communication with the supplier and the status of the orders.
We use Ecto to talk to our databases, which is the de facto standard. It provides a nice, composable data querying and manipulation DSL and an ORM-like way to map information stored in the database to Elixir structs using Schemas. In this post, we’re going to focus on advanced usage of one of its features: “preloading” of the related data.
If you’re already an Ecto power user and know how we usually use preloads, you might want to skip ahead to the “Non-trivial scenario,” or straight to the “Using preload functions” section.
setting the scene
Let’s say we’re building a blogging platform, a bit like Medium, where users can write posts and comments. We’d probably model posts and comments as separate tables in a relational db, where each comment belongs to a post, a post can have multiple comments, and both posts and comments belong to individual users.
In that scenario, if we wanted to load all comments written by a certain user, we’d write something like this:
Trust issues: trouble in package paradise
Code BEAM STO 2019 talk
Earlier this year I gave a talk at Code BEAM STO about a proposed solution to the ever more real risk of hidden malicious code in our library dependencies. You can watch the whole thing here:
String.to_existing_atom/1
...is a double-edged sword
I’d argue Elixir has relatively few gotchas. It’s a simple and consistent language and when you first learn it there’s only a few things that are genuinely counter-intuitive and catch you by surprise.
One of the examples could be the difference between binaries and charlists and why iex sometimes seems to do weird things to your lists:
iex> l = [19, 7, 16, 119, 97, 116]
[19, 7, 16, 119, 97, 116]
iex> Enum.drop(l, 1)
[7, 16, 119, 97, 116]
iex> Enum.drop(l, 2)
[16, 119, 97, 116]
iex> Enum.drop(l, 3)
'wat'
One of the other ones comes when you start working with atoms and get a little too trigger-happy with them. What you could hear from your more experienced teammates is something like this:
You shouldn’t really use
String.to_atom/1on user-supplied data. The BEAM has a limit on how many different atoms you can have and they’re not garbage collected!With data coming from outside the system, stick to strings or use
String.to_existing_atom/1instead!
This is good advice and the official docs agree. It seems like an easy choice too - if you take the approach that all atoms you expect to see in the system are known at compile time and you won’t be creating any new ones during runtime, there’s no reason not to do it! You get all the safety and no problems!
From my personal experience it’s true the vast majority of time. But there are situations where it could blow up when you least expect it (or just in production). Pull up a chair, let me tell you a story…