Fixing Cloudflare 523 errors
(when using Cloudflare Tunnels)
For some context: Recently I’ve been sharpening my Kubernetes skills by setting up a small 6 node k3s cluster at home. The place I currently live doesn’t have a public IP address, so I chose to set up a Cloudflare Tunnel to expose services to the internet.
I chose to have the cloudflared daemon running on the host machines and the overall setup quick and pain-free. The whole thing seemed to work well, but I noticed that over time (within hours) the tunneled
services would start responding more slowly and eventually Cloudflare would display 523 errors.
The cloudflare dashboard was showing the tunnels as healthy and the services were working correctly when accessed from LAN (over their .internal domains).
The cloudflared logs were showing some errors that didn’t suggest any obvious solutions:
$ journalctl -u cloudflared | grep 21:05
Feb 23 21:05:13 suplex6 cloudflared[3473875]: 2026-02-23T20:05:13Z ERR failed to accept incoming stream requests error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=3 event=0 ip=198.41.200.13
Feb 23 21:05:13 suplex6 cloudflared[3473875]: 2026-02-23T20:05:13Z ERR failed to run the datagram handler error="context canceled" connIndex=3 event=0 ip=198.41.200.13
Feb 23 21:05:13 suplex6 cloudflared[3473875]: 2026-02-23T20:05:13Z WRN failed to serve tunnel connection error="accept stream listener encountered a failure while serving" connIndex=3 event=0 ip=198.41.200.13
Feb 23 21:05:13 suplex6 cloudflared[3473875]: 2026-02-23T20:05:13Z WRN Serve tunnel error error="accept stream listener encountered a failure while serving" connIndex=3 event=0 ip=198.41.200.13
Feb 23 21:05:13 suplex6 cloudflared[3473875]: 2026-02-23T20:05:13Z INF Retrying connection in up to 1s connIndex=3 event=0 ip=198.41.200.13
Feb 23 21:05:15 suplex6 cloudflared[3473875]: 2026-02-23T20:05:15Z ERR failed to accept incoming stream requests error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.73
Doing the lazy thing and setting up some staggered cloudflared service restarts in cron didn’t really do the trick (and wasn’t really a solution). Some googling led me to this github issue and this reddit thread, which got me on the right track.
The tunnels by default try to use the QUIC protocol to talk to the Cloudflare edge, falling back to HTTP/2 based on some obscure criteria. QUIC is UDP-based, which can make it harder to troubleshoot, especially when you don’t have control of all of the network infrastructure involved. Switching the tunnels to use HTTP/2 seemed like a good thing to try.
A small gotcha is that it’s not a setting you can change in the Cloudflare dashboard (even though it might seem like it could be, in the Speed -> Settings -> Protocol Optimization on the domain dashboard page) - you gotta use the cloudflared setting.
I was already using Ansible to manage all of the cluster machines, so the change was pretty small - I had to make sure the cloudflared tunnel invocation of the cloudflared.service used the --protocol http2 flag. Here’s the (gist of) the ansible task file:
- name: Create /usr/share/keyrings directory
file:
path: /usr/share/keyrings
state: directory
mode: '0755'
become: true
- name: Add Cloudflare GPG key
ansible.builtin.get_url:
url: https://pkg.cloudflare.com/cloudflare-public-v2.gpg
dest: /usr/share/keyrings/cloudflare-public-v2.gpg
mode: '0644'
become: true
- name: Add Cloudflare repository into sources list
ansible.builtin.apt_repository:
repo: 'deb [signed-by=/usr/share/keyrings/cloudflare-public-v2.gpg] https://pkg.cloudflare.com/cloudflared any main'
filename: cloudflared
state: present
become: true
- name: Install Cloudflared
become: true
apt:
update_cache: true
state: latest
name:
- cloudflared
- name: Populate service facts
ansible.builtin.service_facts:
no_log: true
- name: install the cloudflared service
ansible.builtin.command:
argv:
- /usr/local/bin/cloudflared
- service
- install
- ""
when: ansible_facts.services['cloudflared.service'] is not defined
no_log: true
become: true
## START OF THE FIX
- name: ensure cloudflared uses http2
ansible.builtin.replace:
path: /etc/systemd/system/cloudflared.service
regexp: 'tunnel run'
replace: 'tunnel --protocol http2 run'
after: '\[Service\]'
become: true
register: cloudlflared_protocol_replace
- name: conditionally reload the systemd daemon if the cloudflared service file was modified
ansible.builtin.systemd:
daemon_reload: true
when: cloudlflared_protocol_replace.changed
become: true
## END OF THE FIX
- name: insantly restart cloudflared service
ansible.builtin.systemd:
name: cloudflared.service
state: restarted
become: true
A cleaner solution would probably be to maintain a config file, but it was pretty late at night and this used the fewest moving parts to get the job done.
And like that, the problem went away - the communication with the services behind the tunnels have been quick and solid in the last day (knock on wood), and another heisenbug got squashed.
The whole story isn’t super exciting, I just didn’t want to be the “nvm, I solved it myself!” guy on the internet :D
Elixir string operations seem slow
...and why it's a good thing
I personally hate it when people post clickbait titles and take their sweet time getting to the point, so let’s do this first:
TL;DR: Some Elixir string operations, most notably String.at/2 work in linear time,
as opposed to constant time, like the intuition might suggest. This is because the String module is UTF-8 aware.
UTF-8 encodes characters outside of ASCII with more than one byte, so in order to find the n-th character in a string you need to process it from the beginning, you can’t just use an offset in memory.
In this blogpost I go a bit more in depth about how UTF-8 works and compare some approaches to getting performant results, so even if you knew the tl;dr you might find something interesting here regardless.
Context
I as doing an algorithmic exercise containing string manipulation in Elixir and saw that even though my approach was correct and seemed pretty optimal, it was timing out on larger inputs - and timing out by a lot. After looking into it a bit I realised
Notion Buttons
...and what they need to be truly useful
I’m a long time power-user of Notion. For the last couple of years I’ve been using it for all of my note-taking, work organisation and tracking, online documentation, storing cooking recipes and much more. Recently they introduced a new Buttons feature and it excited many people who thought it would be the missing piece in their workflow organisation.
I played around with it for a while and I can see the current functionality is a great starting point, but it needs a bit more to actually be useful (to me)