TL;DR: iamroot.tech uses a custom-built, asynchronous MySQL database synchronization system handling ~10 million daily changes across several distributed nodes without consensus algorithms or distributed coordinators.
Key Architecture & Trade-offs:
- Primary Keys: UUIDs to allow conflict-free row generation on any node.
- Change Tracking: Database triggers log INSERT, UPDATE, and DELETE operations into a simplified change log.
- Replication Mechanism: Asynchronous background workers process log entries.
- Conflict Avoidance: Application-level single-writer authority per dataset
- Fault Tolerance: Nodes catch up automatically after offline periods by replaying the simple change log backlog.
The setup - servers will disappear
iamroot.tech runs across numerous servers in different geographical locations. They are deliberately inexpensive machines, and an important assumption behind the entire architecture is that individual servers will - at some point in time - disappear, underperform or otherwise misbehave.
The goal has never been to make every server perfectly reliable. The goal is to make each server failure completely uninteresting.
That creates an interesting problem for the databases. A server needs access to shared data, but I don't want a write on server A to depend on server B being available. In fact, I don't really want server A to care about server B at all while serving a request.
So I ended up building my own synchronization system. It currently keeps roughly 50–60 tables synchronized across numerous servers and handles around 10 million changes on a normal day.
It isn't particularly clever. In fact, it's about as simple as it gets.
And that is rather the point.
A failing server should create a backlog, not an outage
The most important principle is that database writes are local operations. If the application running on server A needs to update something, it writes to the local MySQL database and continues.
Synchronization happens afterwards.
If server B happens to be slow, offline or has been gone for three days, that doesn't change what happens on A. It simply means B has more catching up to do when it eventually returns.
In practice, this gives iamroot.tech a distributed data architecture without trying to turn MySQL itself into a distributed database. Each server owns its local database and can operate independently, while relevant data is eventually synchronized between them.
Temporary differences between nodes are therefore expected. Eventual consistency isn't something the architecture reluctantly tolerates; it is part of the design.
That trade-off works because the application itself understands how the data is used.
The implementation is almost embarrassingly simple
Tables that need to be shared use UUIDs as their primary keys. That allows records to be created independently on different servers without coordinating ID ranges or worrying about primary key collisions later.
Each server then maintains its own local change log. Synchronized tables have triggers for INSERT, UPDATE and DELETE, and whenever something changes, a small entry containing the table, record ID, operation and timestamp is written to the log.
A background service processes those entries and sends the changes to whichever nodes should receive them. There is no central replication queue and no synchronization master.
For inserts and updates, I use MySQL's INSERT ... ON DUPLICATE KEY UPDATE. If a server somehow missed the original INSERT but later receives an UPDATE, the record simply gets created anyway.
Replicated writes bypass the change-log triggers for that database connection, while normal application writes still trigger them. Otherwise A would update B, B would send the same change back to A, and my wonderfully simple synchronization system would become a wonderfully effective infinite loop.
DELETEs work through the same mechanism. The DELETE entry stays in the change log until the relevant nodes have processed it, effectively making it a temporary replication tombstone. A server can therefore disappear for days and still learn which records were deleted while it was gone.
That's basically it.
Avoiding problems instead of solving them
This is not conflict-safe multi-master replication.
If two servers independently modify the same record at exactly the wrong moment, there is no clever distributed algorithm deciding which version represents universal truth.
Instead, I largely avoid the problem at the application level.
Individual records normally have one logical writer - one server or subsystem that is effectively king for that data. The code working with those records knows about that architecture.
That is an important distinction. I could add version vectors, distributed locking, automatic conflict resolution and considerably more machinery. Those are perfectly valid solutions to problems other systems have.
But if I can structure the application so the problem doesn't normally exist, I would rather do that.
The same philosophy applies elsewhere. Synchronization is at-least-once rather than exactly-once. Duplicate work happens, but the operations are idempotent, so that is mostly harmless. The change log isn't an event store either. If a record changes ten times, I don't need to reconstruct all ten historical versions. I care about getting the current state where it needs to go.
Recovery is deliberately boring
For every table and destination node, the synchronizer remembers how far through the local change log that node has progressed.
Important data is typically synchronized every 10 – 20 seconds, while less important tables may only be processed every ten minutes. If a destination disappears, its position simply stops advancing while everybody else continues.
When it returns, there is normally nothing for me to do. The synchronizer knows where it stopped and continues from there. Once all relevant nodes have moved beyond old change-log entries, those entries can be removed.
This isn't just how I hope it works. Servers have disappeared and returned plenty of times in production.
The recovery procedure is dead simple: sit back and wait.
I like recovery procedures like that.
Why not just use MySQL replication?
MySQL already provides very capable replication, and this isn't an attempt to build a better replacement.
I wanted slightly different properties.
Not every table needs to exist everywhere. Different data can have different synchronization intervals and destinations. More importantly, replication isn't part of the availability model for an individual application server. A slow destination creates work for the synchronizer, not latency for the application serving a user.
There are obvious trade-offs. The synchronizer doesn't automatically understand referential integrity between tables, although I can control their processing order. It doesn't negotiate schemas either; keeping compatible schemas across the relevant servers is my responsibility.
Those are not necessarily features waiting to be implemented. They are boundaries I know about and can design around.
Boring as a feature
Around 10 million changes pass through this setup every day, with the ASN database alone sometimes generating several million updates during a processing run.
Could I make the synchronization system considerably smarter? Absolutely.
But I didn't set out to build a distributed database. I wanted a multitude of servers that could operate independently, share the data they needed, tolerate each other disappearing, and recover without me getting involved.
The synchronization system is simply what fell out of those requirements.
There are plenty of situations where this design would be completely inappropriate. There are also plenty where sophisticated distributed database technology solves very real problems.
But complexity has a real cost.
For iamroot.tech, I'd rather have infrastructure I can explain on a single piece of paper, predict when something breaks, and largely leave alone when a server disappears.
Extremely simple, asynchronous and a tad inefficient in the actual operations, yes - but very efficient at solving the problem, very predictable in the way it works, and - most importantly - it does not require any human intervention at any point.
That is all I need.