Rendered at 11:01:27 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
sureglymop 50 minutes ago [-]
Mostly unrelated but shopify is incredibly annoying. They introduced this delivery tracking app called "shop" and it has become unavoidable when buying electronics from china. Recently looked at it with mitmproxy and it ships home more than gets shipped to me.
doublerabbit 26 minutes ago [-]
Mood. It happens on all things now unless they use a domestic courier. They then still use dark-ui to hide the tracker link.
ramon156 10 minutes ago [-]
Why even have a blog when you can't be arsed to write the posts. This is so obviously LLM-written. I have a positive view of Shopify engineers, but this kind of made a dent in that confidence.
manbash 9 hours ago [-]
> Instead of one row per item with a quantity column, we use one row per sellable unit. An item with 10 units has 10 rows.
> But one row per unit for all inventory would break down at scale—an item with 50,000 units across 10 locations would mean 500,000 rows, and the reserve query would slow as it scans through them. Instead, we maintain a bounded pool of available rows, capped at 1,000 per item/location combination. Reservations consume rows from this pool; a replenishment process refills it from the inventory ledger.
Shouldn't I feel uncomfortable with such approach? It seems to create a backoff (pool) for lowering the chance of having a synchronization issue.
fauigerzigerk 20 minutes ago [-]
I agree, it does seem awfully complicated and there are quite a few pieces missing for this to be a complete solution.
I'm a bit surprised about the scalability case against a simpler solution. This is not about Shopify's scale. We're talking about contention for a specific SKU of a specific seller at a specific warehouse location.
How many shopping carts are competing for a single SKU at the payment stage at peak hours? Can this really be too much lock contention for a single database row?
I realise Shopify engineers are neither stupid nor inexperienced. Hence my surprise. I would have liked to hear more about that specific problem.
esjeon 5 hours ago [-]
I would call this one-row-per-contract-type, and this is the most general model for the problem (e.g. the model cannot be further broken down into finer level), thus, the most scalable model given storage is dirt cheap.
LunaSea 3 hours ago [-]
Storage might be cheap but search and RAM isn't
ponector 1 hours ago [-]
RAM may be expensive for a person, but still is quite cheap for the company.
e12e 39 minutes ago [-]
Maybe the example numbers are just bad - but now you expect your system to fall down if you scale from 10 to 100 locations?
onion2k 32 minutes ago [-]
"Number of locations" is an input so if the system has been designed to handle up to 10, and not 100, then yes I would absolutely expect it to fail with the higher value.
Developers (and everyone else really) need to think about systems, with the system taking inputs like "number of locations", and producing outputs like "available inventory", and when the input parameters change outside of the designed scope, without the system itself changing, then you should expect things to break.
sandeepkd 9 hours ago [-]
Comes down to type of items, when you have physical inventory the number is limited so more manageable and interestingly enough the problem only applies to physical inventory.
You are just spending some more disk space to avoid synchronization issues. Denormalization for performance is a really common pattern, just that people do not start with it in the first place itself
bijowo1676 7 hours ago [-]
you should, their design is not the best. There is middle ground between "one row per SKU" and "1000 rows per SKU".
Its called one row per shopping cart*SKU combo.
if two people order 100 and 500 items of the same SKU, respectively, the table should have only two rows: for order1 and order2. Not 600 rows.
codedokode 2 hours ago [-]
The problem is that in this case you have to do splits/merges. And while there are products that are sold by 100 units at a time, I think in most cases people by 1-2 items so the hassle might be not worth it.
Also you might not understand the original problem. Imagine if 100 customers want to buy product A. One thread starts a transaction, searches for amount of product A and UPDATE's it and goes searching for other products. The database locks the row until the end of transaction and other 99 treads cannot continue until first transaction commits (they can read but cannot update the rows).
This is why they made a row per item. In this case, transaction 1 hopefully locks only several rows with items of product A. Transaction 2 instead of waiting for lock release skips them (due to SKIP LOCK) and locks several next rows. And so on.
Obviously you do not need to make a row per item - if the available amount is really large (10 000 items), you could have for example 100 rows having 100 items each. In this case each transaction locks the whole row (100 items) even if it wants to reserve just one item. The problem though is that now every row might have different amount of available items and you have to do more work to reserve the amount you want.
jakewins 7 hours ago [-]
Can you explain how that works? With the row-per-item I can see how you’d use locking primitives etc easily to deal with multiple concurrent shopping carts claiming available inventory.. but how does your solution solve contention? There’d need to be some “number of items in inventory” row, wouldn’t there be contention on that?
The point of one row per item is that thousands of concurrent shoppers don’t need to block each other as they can each claim as many free rows as they need for themselves?
sroussey 4 hours ago [-]
One other advantage is item serial numbers. Or something else that makes an item that seems the same but actually be unique (perhaps the warehouse it’s in?)
I guess it depends on how the replenishment process works. Unless you're ordering over 1000 of an item, I doubt it would be a problem.
bijowo1676 7 hours ago [-]
replenishment is an unnecessary cludge that only exists due to poor design. an "algorithmical smell" if you wish
szundi 28 minutes ago [-]
[dead]
jghn 7 hours ago [-]
Depends on the scale. Most companies don't approach the scale where this matters.
dbbk 4 hours ago [-]
I'm familiar with the reserved row approach (I use SELECT FOR UPDATE SKIP LOCKED) and yeah this replenishing idea terrifies me.
progx 15 minutes ago [-]
Why is it so hard for many people to accept, that this is a solution for a specific problem of shopify? They did not say that Redis is bad and MySql is good. They only a solve their problem.
gregoriol 10 minutes ago [-]
Even the good teams may make bad choices, that's why it is interesting to read their ideas and discuss them
isignal 9 hours ago [-]
It seems there could be a simpler solution.
1. Deduct the reservation from the inventory when the user starts to order, but in the same txn also maintain a separate row for the in progress order flow.
2. If the order flow is aborted or times out have a background process that returns these to the inventory.
That seems simpler than this approach and involves no locking. Though their presented approach is also reasonable, there must be some reason not to choose a simpler flow. It is not that difficult to have a gc service that scales, but may be they didn't want to separate that.
firasd 8 hours ago [-]
My understanding is: your proposal is not very different from what Shopify is doing except they are tracking 'reserved units' (one per row) and you are proposing tracking 'orders' as the temporary state to then reconcile back with inventory quantities.
isignal 8 hours ago [-]
Yes, at a high level. It doesn't rely on skip locked, which is not cheap at DB level. DB has to still typically run query and keep going until it finds an unlocked item. Deducting and checking inventory counts are simpler ops inside the DB.
treis 7 hours ago [-]
This seems like what triggers are for and how we do similar type things. Update trigger on order does select for update on the inventory and increases/decreases it as appropriate.
I don't think you really need that even. An indexed lookup is fast and you don't need to store a computed quantity generally.
sandeepkd 9 hours ago [-]
The moment you added a background process you just replaced the complexity.
1. Backgrounds process can back up
2. They need context of the user and need to switch context per user
3. What if they fail, you create some DLQ or another process to handle the failure
4. Who looks on those failure and how do they act
TLDR; there is always a cost
0x696C6961 8 hours ago [-]
The design in the shoppify post already had a background process for the item replenishment.
soontimes 8 hours ago [-]
Can you clarify why this involves no locking? There can still be 2 actors fighting for the same row.
isignal 8 hours ago [-]
Two concurrent deductions of inventory do contend but only during the actual DB update. That is just normal DB locking for SQL isolation levels. The blog refers to explicit locking by the app, which is where skip locked comes in.
soontimes 7 hours ago [-]
Yes, the point is to spread contention across multiple rows. They also mention this in the beginning of the article
vxxzy 9 hours ago [-]
now you have two problems. what happens when your reservation system backs up?
dbbk 4 hours ago [-]
You don't sell stuff I guess
sieabahlpark 8 hours ago [-]
[dead]
stillpointlab 7 hours ago [-]
I was investigating Durable Objects (DO) and had Fable walk me through where in my app they might be appropriate. One place had a dependency with billing (where I use a transaction now) and the proposed re-work to allow for concurrent editing with DO looked very much like this, reservations with idempotency keys. And if you add hierarchical allotments then it scales pretty well.
I disagree with the other posters about the bg process, if you have any bg processing already you should be able to handle the few edge cases without too much trouble.
pjmlp 44 minutes ago [-]
I never spent much time with the whole NoSQL movement, it always seemed something out of people that don't get how to optimise SQL queries, or suffer from SQL allergy, only to reinvent it badly in custom languages.
szundi 26 minutes ago [-]
[dead]
codedokode 2 hours ago [-]
Could not they shard the inventory table by shop_id? As I understand, the order includes only items from one store, so there is no need to keep all the stores in a single table.
Also, I wonder why they could not have a row status (available/reserved) and UPDATE it instead of deleting the rows.
arichard123 54 minutes ago [-]
I had a client and they weren't to bothered if they sold the last item twice, they would call the customer, apologise, and offer a discount on an alternative and keep the sale.
zhivota 9 hours ago [-]
"But the hardest lesson wasn't about database design. It was discovering that the real bottleneck wasn’t what we were observing and measuring."
Horffupolde 9 hours ago [-]
But was it load bearing?
CoastalCoder 9 hours ago [-]
Even better.
It's web-scale.
ares623 8 hours ago [-]
load = bearing
gun = smoking
insight = key
gap = closed
summary = executived
7 hours ago [-]
KingMob 7 hours ago [-]
belt = suspended
jtbaker 7 hours ago [-]
boot = strapped
tweakimp 4 hours ago [-]
Until now I thought it was boots-trapped. I am not a native speaker :)
crabmusket 3 hours ago [-]
The idiom is to "pull oneself up by one's boot straps":
It's honestly weird Claude converges on this language because it's incredibly wordy and hard to parse.
One would think semantic density would win out in training.
true_religion 6 hours ago [-]
Why? This is a common transition that people use in speech and text.
Close out previous paragraph. Segue to completely different topic.
How else are you supposed to go on a tangent?
Foobar8568 2 hours ago [-]
I feel that Claude converge to more claudism while Chatgpt sounds more natural.
Both sucks in French but at least chatgpt prose is readable while Claude is awful.
peyton 8 hours ago [-]
Who knows. I wish ant harshly penalized speaking litotically because it’s essentially reward hacking as it can often be read multiple ways.
It’s also annoying as a human because Claude et al rate their own writing very highly, putting human<>LLM interactions at a disadvantage to human->LLM<>LLM interactions.
Jolter 4 hours ago [-]
Thanks for teaching me the word ”litotically”!
sroussey 4 hours ago [-]
Each version minor version of Claude has its own preferences for vocabulary.
MagicMoonlight 25 minutes ago [-]
[dead]
jasonlotito 8 hours ago [-]
[flagged]
nozzlegear 8 hours ago [-]
It's not hard to parse, but it's a dense pair of sentences that say nothing. It just pads the length of the article and gives readers mental fatigue trying to read between the lines to figure out what the point is.
CoolestBeans 7 hours ago [-]
I actually don't think this article was LLM generated but these two sentences suck. I think they were moved from another part of the article without being modified.
First, "the hardest lesson". What lesson? It is out of context. Nobody was talking about lessons before this.
Second, "the bottleneck wasn't what we were measuring and observing". Of course the bottleneck itself wasn't that. They couldn't discover what the bottleneck was using the information in their measurements and observations.
It is a clunky and frankly incorrect passage in an otherwise well written article.
ares623 6 hours ago [-]
Sure, it's not technically hard to read.
But it suuucks, making it hard to read, the same way (some) fast/junk food is hard to swallow.
They have access to a trillion dollar writing machine god, and they choose to publish that.
bijowo1676 7 hours ago [-]
not the best design to have 1000 rows for each shop*SKU combination. If a candidate proposed this solution during Shopify's System Design interview, i doubt he would be vetted for Senior+ position.
Instead of having 1000 rows per shop*SKU, why not just have one row per shopping cart*SKU?
That way a single row would represent a single cart, and will hold info of multiple items of the same SKU.
No need a cludge with 1000 rows limit and replenishment process. Instead of dealing with N rows, you always deal with a single row.
idoubtit 4 hours ago [-]
> not the best design [...]
So those engineers at Shopify worked hard for months on a more performant system, but they missed the obvious structure? They chose a complex denormalization for no good reason?
It may be true, but I think it's presumptuous to belittle their work when we have only partial information. My guess is that they had good reasons to think that the more obvious ways would not scale.
And from reading your comments in this thread, I believe your structure would fail at their scale. A SQL query that uses 2 sub-queries with "group by" is probably too heavy. From the post, at peaks there would be millions of active shopping carts.
BTW, I suspect most orders are just for 1 or 2 of each item, so the denormalization is not as heavy as it seems.
bijowo1676 4 hours ago [-]
i also work in big tech and know that a lot of bullshit design creeps into system design and prod, because everyone is overworked, overstressed, wants to just get things done for the quarterly performance review as to not get shitcanned with severance
re concurrency, it is not a big issue at all. stock exchanges deal with HFT traders and can easily deal with concurrency of orders. Same can be implemented with shopify, but I doubt they face the same level of concurrency as stock exchange anywhere near
post_below 4 hours ago [-]
You might not have noticed that essentially the entire blog post was AI written.
There's even this bit where they discover a remarkable trick:
> Each round trip to the database has a cost. For carts with multiple line items, we batch reservation queries using UNION ALL so we fetch all needed units in one round trip
Insights like that really don't read like senior level output, and of course, it's LLM output. I'm not sure it's presumptuous to question it.
Chyzwar 1 hours ago [-]
There is now new type of comment in HN, if you disagree with article you attack the fact that ai was used in post editorial process. People here now dismiss anything that have em dash.
maxrev17 58 minutes ago [-]
Yeah ai doesn’t mean bad content. It does make for unreadable and unbearable articles though
atomicnumber3 6 hours ago [-]
I have never worked anywhere where describing how their system actually works would pass the company's own system design interview
maxrev17 57 minutes ago [-]
Hahaha best comment on the whole thing! So true!
matwood 3 hours ago [-]
Others are almost never as dumb as you hoped, and you’re rarely ever as smart as you think.
soontimes 7 hours ago [-]
> Instead of having 1000 rows per shopSKU, why not just have one row per shopping cartSKU?
At what point that row is inserted?
bijowo1676 7 hours ago [-]
per my reading of the article, the protection is only needed for a few seconds, while payment is being processed by the payment system.
so the row is inserted when Payment is initiated, and row is deleted when Payment succeeds
What is oversell protection?
Reserve: When payment starts, we mark items as reserved (a short hold, e.g. several minutes).
Claim: When payment succeeds, we permanently deduct quantity from the inventory ledger (source of truth).
but that system could be easily improved to reserve item when user Adds item to a cart, to prevent scenario when user adds item to a cart, goes through checkout, and after initiating payment gets "soldout error":
1. Let user add item to a cart by default (happy path)
2. Initiate async check in the background for SKU and quantity
2a. The check sums up rows for all SKUs and compares to Inventory table (very cheap check since its done to only active shopping carts)
3. After few seconds the check comes back, and we let user know that item is soldout, before/the moment user goes to Checkout.
soontimes 7 hours ago [-]
Ok, but before inserting you must ensure that inventory is not depleted, which means you need to know the count and you need to lock the row. So you still have contention on that item. Them having a 1k buffer allows not to take a lock on a single row every time, and only do it when buffer is empty
bijowo1676 7 hours ago [-]
there is no need to lock the row, since you a dealing with a shopping cart, not individual item piece. when you run aggregate functions, lock is no needed, it is actually better to run it with SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; for aggregation
the check for oversold items is extremely cheap:
with current_order as (
select $SKU1, $q2 as quantity
union
select $SKU2, $q2 as quantity
),
with carts as (
select sku, sum(quantity) as reserved
from active_carts
group by sku
),
with warehouse as (
select sku, available_units
from inventory
group by sku
)
select * from current_order
inner join carts using (sku)
inner join warehouse using (sku)
where warehouse.available_units - carts.reserved < current_order.quantity
assuming there are indexes on sku field in both, results in efficient index seek and agg over 2 tables
codedokode 5 minutes ago [-]
The item is reserved when the user decides to place an order, but before paying for it. Not when a product is added to the cart because the user can keep it there for a month and end up not buying.
You reserve the product by creating an "active_cart" entry. Your solution has a problem, that when you run the check, it might say the product is available, but before you create an "active_cart" to reserve it from thread A, another thread B reserves it and you end up reserving a product that is not available anymore. You end up with SUM(active_cart.quantity) > inventory.available_units.
That is exactly why the database has locks - to prevent this situation. With locks, thread A decrements inventory.available_units and that row is locked until the end of transaction. Other threads (if they do SELECT FOR UPDATE instead of SELECT) cannot see the old, invalid value until thread A either commits and the value is updated or rollbacks. However, locks cause performance issues and that is why shopify uses the architecture from the article - instead of 100 users fighting for the lock on the same row with available amount, each user locks only rows with units they plan to buy.
I don’t understand how this should prevent oversold. You have a check that reports empty or oversold inventory. But how does that check prevent 2 concurrent actors fighting for the last item from inserting 2 rows?
bijowo1676 6 hours ago [-]
how does current design resolve concurrent actors fighting for the last item ?
there is ultimately needs to be some global mechanism resolving this conflict. Currently it is an order in which db engine processes transactions by locking rows for a transaction, whoever got the first lock, wins the last remaining items.
my design is the same, except it does not need this dance with moving rows between tables, locking them, and the cludge with replenishment process.
in the simplest form, run the sum() over active non-finished orders and compare to inventory. you get the same result: whoever got the first to run sum() and get positive answer will get the last remaining items.
but the problem as formulated, imho, is not even correctly defined.
Shopify incorrectly formulated the very problem they are trying to solve.
Trying to solve it at the payment time is too late, its better to resolve it earlier, before the checkout.
the "PAY" button should only do one thing: deduct money from cc and that's it. Resolving inventory availability must be solved way earlier, the moment user clicks Checkout, not when user clicks Pay.
So ideally, the error for oversold items should be shown to a user when he clicks Checkout, not when he click PAY
admax88qqq 5 hours ago [-]
> Shopify incorrectly formulated the very problem they are trying to solve.
That’s a bold overconfident statement. Cart abandonment is real. People never clear their carts they just walk away
Shopify purposefully chooses to do it at payment time because doing it earlier results in lost sales as people “reserve” items and then walk away causing other to see out of stock and then also walk away
Whoever puts up the money first gets the item
That’s the design constraint they chose you can’t just say “their solution is wrong because they solved the wrong problem”. Each design is a different user experience and I think it’s safe to say they chose which experience they want consciously.
bijowo1676 5 hours ago [-]
that's why I mentioned active carts in my post, there are ways to define active cart to get rid of abandoned carts ( ignore carts where last user action was > N seconds ago).
Ok, let's accept the design goal that whoever paid first wins. You can use the same metric (how many milliseconds ago did user click PAY) and impose a global monotonic non-decreasing counter to distribute the scarce inventory. This is how order matching engines work at stock exchanges with HFT orders (FIFO logic).
the goal is to know with 100% certainty, before sending payment request to payment processor, who will have item and who won't, and you dont need to move mountains of rows for that.
the payment processor should be just a binary answer: payment succeeded or not, but currently it combines Inventory availability check & payment processing, which is the root cause of confusion. For clarity it is better to make that stage of order processing an explicit separage stage, instead of coupling it with payment stage.
some stores split payment into two stages: Payment and Final order confirmation. at the Payment stage you can pre-authorize money at cc and do inventory availability, and at final confirmation you capture $$
hanikesn 4 hours ago [-]
Most payment methods in the world don't support separate authorization and capture.
bijowo1676 3 hours ago [-]
i dont know about the world, by authorize.net and Stripe, which work globally and work with global credit cards, they do support separate authorize and separate capture, which seems to be part of PCI standard
Clearly this is for high concurrency cases where there are many people racing to get all the available items. It's not clear that it's in shopifys or the sellers interest to let items get sequestered in people's shopping carts, which is a spot where there isn't a strong commitment to complete the purchase. At payment time, you can be more assured that the item will actually be purchased.
Still I think their solution is a bit weird. I'd want to commit the reservation transaction with inventory decrement along with a payment key and then use a different transaction to drop the reservation when the transaction completes. If the transaction does not complete in a timely manner you probably need to query external systems anyway to resolve whether the payment actually occurred or not.
They talk about lock contention in this case, but I also wonder about latch contention since these rows are adjacent. If it's a small transaction that's not interactive, does mysql resolve it with just the latches on the needed tables?
neerajsi 2 hours ago [-]
I was curious about what Tiger Beetle does. It has two phase transfers, which appears tailor built to handle this case. But maybe Tiger Beetle isn't the right database to track all your product stocks.
soontimes 6 hours ago [-]
> how does current design resolve concurrent actors fighting for the last item ?
It resolves with skip locked. Assuming we have only 1 item left. First query scans the buffer table, locks as many rows as needed (1 in our case), and moves rows to another table. Second query scans the table, finds no rows (even if first one hasn’t finished yet, the row is locked and ignored), checks if it can increase buffer, finds out that it’s fully sold and aborts. Db guarantees that you can’t oversold.
> my design is the same, except it does not need this dance with moving rows between tables, locking them, and the cludge with replenishment process.
I can’t evaluate whether it’s the same or not, because you still haven’t clarified when exactly you’re going to insert the row. In the article they’re inserting in the same transaction. Would you also do it in the transaction? Because if you’ll introduce a separate global mechanism to resolve conflicts, on a high level it would be the same as their approach with redis (you need to have 2 systems)
EDIT: wording
bijowo1676 5 hours ago [-]
think about for a moment what that skip locked actually means, all these 1000 rows per SKU are logically equivalent to a Inventory table with a single row where available_units=1000 per SKU.
now let's think again, do we need to lock 900 rows to place order on 900 items? or can we insert a single row where order_quantity=900 ?
shopify's design relies on DB to lock rows for transaction as a way to "decrement the counter" of available units. What I am suggesting, is you can just decrement counter by updating a single row, no need to lock 900 rows. Shopify moved from one extreme (single global variable in redis) to another extreme (1000 rows in db) and forgot about the middle ground.
The dance with moving rows per each item between tables is completely unnecessary, it's like counting numbers one by one in a for loop, when you can just substract number directly.
if I were to solve the problem, I would have solved it differently, at the Checkout state, before user clicks PAY. This removes the race condition at the user UI level, before any request lands in backend/db:
1. Have a table with active shopping carts (cart_id, cart_status, sku, quantity)
2. when cart_status changes to 'Checkout' run inventory availability check
3. If inventory availability check fails, show error to user (before he clicks Pay) and suggest replacement items.
4. If inventory availability succeeds, proceed to charge cc
availability check is the SQL above: inventory-sum(active_carts.quantity)-current_order must be > 0
soontimes 2 hours ago [-]
> if I were to solve the problem, I would have solved it differently, at the Checkout state, before user clicks PAY. This removes the race condition at the user UI level, before any request lands in backend/db:
In order to avoid races you need to insert reservation and decrement availability atomically. Your proposed approach is not atomic. For it to be atomic you will need to lock whole range, to make sure no new rows appeared between the points “check for availability” and “record reservation”. Actors will be effectively competing for the single aggregate row. This is the same as having a single inventory row with quantity field, which they rejected in the beginning of the article
> now let's think again, do we need to lock 900 rows to place order on 900 items? or can we insert a single row where order_quantity=900 ?
In the proposed schema nobody is waiting for these locks, they’re skipped by concurrent queries. In your schema actors would have to wait before they can insert without breaking invariants.
pas 5 hours ago [-]
assuming their "reserve item" function is just "update the table set N rows to reserved=true where reserved==false"
more transactions can commit at the same time, but with one counter they would conflict (as it did in the Redis case)
they should use CRDT (and trying to model that with this 1000 row workspace, no?)
still, eventually at some point they need to do the math
Godsend69 5 hours ago [-]
[dead]
edoceo 5 hours ago [-]
Thanks! I don't uSe 'with' enough
raverbashing 4 hours ago [-]
And that's why these interviews can be stupid, you can mention the real solution and interviewers might reject because it's not the textbook solution
Pair programming and forced AI, that sounds like absolute hell. Glorification of Lütke who didn't do that much in open source and now props up his ego by thinking "AI can do it so it wasn't all that difficult all along."
I don't think he ever worked on complex parts of Ruby. The people he now oppresses did.
Ruby should note that this company is actively repelling people from using the language. I really want to switch, but then I see Claude contributions in Ruby core, the influence of this slop company, and think it isn't worth it.
Oh, and they bought DHH in 2024 for his 180° turnaround on AI. He is now an AI booster, so Rails is out of the question as well.
mrloopex 7 hours ago [-]
This is absolutely fascinating. I enjoy real life stories like this. I went to a Node meetup in 2013 when Target had just switched to Node from PHP and it was a similar experience to see their metrics and hear their strategy.
firasd 9 hours ago [-]
Makes sense... if you are counting something in MySQL and now your counter is in Redis that's already strange
But I guess the point is that even in the MySQL scenario the 'reserved_quantities' is almost like a temporary table so either way is not the 'Real' inventory
REPLicated2 3 hours ago [-]
> 3. Consistent lock ordering: avoiding deadlocks
This section is badly written. For example, it refers to different table names than those previously introduced.
The slop shows. While I appreciate the post, I wonder why they didn't bother using an LLM in a way that would at least ensure internal consistency.
eviks 2 hours ago [-]
What way would that be?
srcreigh 8 hours ago [-]
It’s fascinating that in order to do this, they had to remove 50% of reads and 33% of transactions from the main DB.
jdw64 2 hours ago [-]
Is it really the right choice to drop Redis and go back to a disk based relational database just to wrap transactions into a single unit?
Redis handles tens of thousands of concurrent connections in a single event loop, while MySQL uses one thread per connection. No matter how I look at it, that seems like a step backward.
Of course, performance isn't everything. And if performance isn't a problem, having everything in one place does make it easier to reason about. But I'm worried that under spike traffic, this approach might actually cause more problems.
I think putting a scheduling layer in front of the DB would be a better approach. The application server could handle concurrent connections and only write to MySQL when correctness is actually needed. That seems like a cheaper way to do it. but is it different for large-scale enterprise distributed systems?
codedokode 2 hours ago [-]
Redis doesn't have transactions and persistence.
No persistence means the data gets lost if machine shuts down or process crashes. Furthermore, after restart you will need to regenerate the data which can take time. That's why Redis is a cache and not a database. You can fix the persistence issue (Redis can write WAL log, don't remember if it does fsync or not), but then Redis won't be able to handle those thousands of concurrent connections.
Redis (and other NoSQL storages) don't have some magic architecture that gives them advantages over SQL databases. They just cut corners on ACID guarantees and skip fsync. Once you start doing fsync, your transaction throughput will drop to SQL database level.
Redis also doesn't have transactions which means every app error damages the data. You will spend engineer hours investigating and fixing the problems. Transactions save so much time and worries.
el1s7 59 minutes ago [-]
Everything you said is incorrect. Redis does have transactions, as well as data persistence. All cloud providers provide managed redis instances with automatic backups as well.
jdw64 1 hours ago [-]
Does Redis become that slow when you enable both AOF and RDB? Sure, there's a write cost, but it doesn't lose its ability to maintain tens of thousands of connections. Redis supports AOF and lets you choose the fsync policy.
But I think using only MySQL is unnecessarily expensive, just to get single transaction tracking for bug tracing. So the article's argument seems to be:
'Use only MySQL as a solution to the distributed transaction consistency problem between two different storage systems, Redis and MySQL!'
But I think using Redis is much more elegant. It's easier to scale. I'd even argue that something like Saga would be a better approach. Of course, we might just have different opinions. But in my experience, reducing layers always ends up making things more complicated in the long run.
p.s. We have different views, but I do think some of your points are valid, so I upvoted your comment
nurettin 4 hours ago [-]
At that revenue, why not make your own filesystem, database and index structure? There is no way mysql is the best possible software for this use case. Why stop innovation and hand everything over to ops?
znpy 4 hours ago [-]
Most likely? Time.
Using off the shelf software means you mostly design how to plumb things together and how to make them correct , safe and scalable.
The things you mention, on the other hand, carry the same requirements but are also much complex to develop AND to maintain.
kgeist 2 minutes ago [-]
The social network VK internally uses highly specialized database engines per business domain. They don't use stock DBs. They have a DB engine for posts, a DB engine for likes, etc. They have a team of DB engineers. They had something like 250 mln RPS load on the databases 3 years ago. Stock DBs were harder to scale for them. I guess if you have immense highload, having a team of DB engineers may be cheaper because you can save a lot on servers. I reviewed their code. A DB engine's source code is pretty compact and simple because they deal with very specific domain entities, so they don't have to account for all the possible user query combinations that a general-purpose DB would have to support. It was mostly shards+binlog+snapshots+views in RAM.
culi 8 hours ago [-]
They were really so proud of that AI image that they just had to tack it on at the end? Did nothing but make the blog post feel like cheap mass produced slop
nozzlegear 8 hours ago [-]
This is Shopify, the leadership is full steam ahead on AI in a big way and they review employee performance based on AI usage.
throwatdem12311 6 hours ago [-]
And Lutke is a fascist.
All the biggest proponents of AI seem to be fascists.
Weird.
tayo42 8 hours ago [-]
The blog probably was.shopify was pretty early and publicly all in on using AI for everything
kennywinker 9 hours ago [-]
Shopify’s founder and their coo both fund far-right extremism, and its founder thinks only rich people should be able to vote. But anyway, they switched databases.
The link you shared is just a podcast and does not contain even contain “far right”. Can you provide specific concerns, otherwise I don’t see anything wrong with a CEO of the most successful tech company should not be concerned about a horribly performing country from a GDP perspective.
kennywinker 5 hours ago [-]
I included a link to a podcast because that's a good high level overview of the topic. If you want to dig into it further, the podcast episode page I linked to has links to more reporting and plenty of keywords you can type into google.com if that's not enough.
Anyway, if you think a country having a low gdp per capita is how you measure if it should suspend voting rights for disabled people and stay at home parents, then I suspect you're not actually reading any of this.
croes 6 hours ago [-]
You should be concerned what they spot as the problem and what they propose as the solution
Yeah it's an awful place to work unless you're a far-right bro. My old director used to use slurs and vape in the office. The founder hires pro gamers with no technical expertise because he thinks they're cool.
chucksmash 8 hours ago [-]
Don't get shy now. Which slurs?
hdndjsbbs 37 minutes ago [-]
Let's say some racial ones (including hard-r n-word) and some ableist ones? Guy also talked about jerking off a lot and joked about rape.
KingMob 7 hours ago [-]
Why are you trying to get someone to repeat slurs?
chucksmash 7 hours ago [-]
Because "slurs" is vague and covers a wide variety of utterances that run the gamut from unprofessional to unemployable and I had a tingle of Spidey sense that OP might have chosen the vague phrasing specifically to inflate the sins of the nameless Shopify director in the mind of the reader.
kennywinker 7 hours ago [-]
Which slurs are ok, and which aren't?
chucksmash 6 hours ago [-]
Great question. It's a vague term and a moving target that not everyone agrees on.
For instance, retarded is now considered a slur. If OP is saying "my director said the R word," then I would question whether OP spared the gory details out of concern for polite company or if they're being oblique in service of their point.
And if managers at Shopify are dropping N-bombs, I want to know that too.
so this is interesting to me, im in retail i work closely with platforms ive used shopify ive used magento ive used smaller players ive helped implement various pieces of all of them.
and i was excited to get some insight, then i realized that this whole thing was written by AI and im going to guess the idea and implementation were probably very AI driven.
> The solution: SKIP LOCKED
> Core idea: one row per unit, bounded by design
cool, thanks claude.
Now I'm wondering what the engineering culture is even like at shopify.
Here's the thing. I like databases, I think there's a lot of shit in this space that went and smoked a shit ton their own good stuff to come up with these pure event driven designs that lock you into event workflows with no isolation and remove the ability to do broader bulk-functions.. and then do something even stupider and say "all you need for the interface is graphql" and such service/platform doesn't give you any other way to reconcile or do reporting for your org you have to warehouse from graphql.. this is crap. So seeing a headline where shopify says they want to kinda get behind a unified database strat behind the scenes even if it's not necessarily customer facing, like that's good imo. SQL is many decades of relational algebra that makes insane computations acrossed vast sets of data pure magic and one of the best query dml interfaces of all time.
..however i dont even agree with the claim their making here that redis isnt the tech for a reservation system. redis when used correctly feels like an insanely awesome way to do a reservation system, i lurv redis for stuff like that.
I'm just gonna go forward with the assumption that current and future shopify updates are pure vibeslop. I already hate their data interfaces, but compared to other saas offerings i appreciate that they do have bulk-features.
akamaka 9 hours ago [-]
I found Shopify’s post very easy to read, and learned about some features of MySQL. On the other hand, I didn’t get any value from reading your comment. You seem to have a bunch of opinions about how things should be done, but haven’t given any details about how you came to these conclusions.
trueno 6 hours ago [-]
I've done multiple large scale implementations with shopify paired with many flavors of order managmeent systems as well as competing offerings in the space. The only one i haven't touched that i'd like to get my feet wet with is commerce tools.
I really just disagreed with the assessment that redis is not good enough for the job for a reservation system. I use sql database all the time, I prefer them. But I'm seeing a claude written article here that seems to heel turn on a proven technology, it would at most be insightful if there was human content in here from actual engineers at shopify who want to vouch for and explain the challenges they were up against with redis rather than just expect me to take claudes word for it. Anyone who's been dabbling with AI knows damn well that you can convince claude to write up a dissertation on any hill you want to die on.
dawnerd 6 hours ago [-]
I found it really hard to read, the llm-isms are just too distracting. Does no one proof blog posts anymore?
benmmurphy 9 hours ago [-]
You should be able to do these increments/decrements in a database at the rate you can write WAL to the disk. But the problem is in a lot of these databases the transaction will hold locks until the WAL hits the disk which causes a massive serialisation problem when you have lots of writes to the same row.
For example if it takes 20ms to write a batch to the WAL then if you do 5 updates to the same row then that is a minimum of 100ms. But without waiting on locks if you can batch all the WAL writes together then this could be just 20ms.
I don’t think holding locks while waiting for WAL is strictly necessary. There is definitely some anomalies that can happen if you don’t wait for WAL to be durable because transactions that don’t write WAL can observe non-durable writes in some situations. So for example conditional updates that don’t perform work. But I assume this can be fixed by making these wait on the commit for dependent transactions to become durable if they are empty. There is also the problem of failing writes that reveal information about non-durable writes which is more tricky. For example you try to insert into a unique index and it fails, but the duplicate was due to a non-durable write that is lost.
Pure reads should be fine when using MVCC because you just show the latest durable version of the DB. I know some other replication systems will run all transactions including reads through the WAL/replicated log in order to not have anomalies.
tybit 9 hours ago [-]
They do heavily use AI, but you haven’t refuted their point that if inventory is in SQL, storing reservation in a second storage system increases complexity.
trueno 6 hours ago [-]
I mean I'm all for everything collapsing into sql. SQL all the things. Not really against it, I'd just rather not-AI write the challenges they were up against. It seems like these are all very behind-the-scenes scaling issues they faced, so it'd be cool to hear from them. Redis has great qualities, I don't use it often but I've also for years now understood why Redis was put in front of these use cases to handle them. Complexity be damned generally you're trying to enforce a first come first served or some level of idempotent behavior, so if sqls doing that now then hell yeah. It's just super off-putting to try and upend an important design pattern with an AI written article.
The lengths companies will go to avoid running different pieces of software...
matwood 3 hours ago [-]
Most companies would be best served picking MySQL or PG and only adding something else if absolutely necessary. Every piece of software added increases complexity.
anonymars 9 hours ago [-]
It can be easier and cheaper to solve problems via technology changes than operations and people
Now you only need MySQL expertise and maintenance rather than Redis and MySQL
4 hours ago [-]
kirici 3 hours ago [-]
The default should be that every additional piece needs to be justified
> But one row per unit for all inventory would break down at scale—an item with 50,000 units across 10 locations would mean 500,000 rows, and the reserve query would slow as it scans through them. Instead, we maintain a bounded pool of available rows, capped at 1,000 per item/location combination. Reservations consume rows from this pool; a replenishment process refills it from the inventory ledger.
Shouldn't I feel uncomfortable with such approach? It seems to create a backoff (pool) for lowering the chance of having a synchronization issue.
I'm a bit surprised about the scalability case against a simpler solution. This is not about Shopify's scale. We're talking about contention for a specific SKU of a specific seller at a specific warehouse location.
How many shopping carts are competing for a single SKU at the payment stage at peak hours? Can this really be too much lock contention for a single database row?
I realise Shopify engineers are neither stupid nor inexperienced. Hence my surprise. I would have liked to hear more about that specific problem.
Developers (and everyone else really) need to think about systems, with the system taking inputs like "number of locations", and producing outputs like "available inventory", and when the input parameters change outside of the designed scope, without the system itself changing, then you should expect things to break.
You are just spending some more disk space to avoid synchronization issues. Denormalization for performance is a really common pattern, just that people do not start with it in the first place itself
Its called one row per shopping cart*SKU combo.
if two people order 100 and 500 items of the same SKU, respectively, the table should have only two rows: for order1 and order2. Not 600 rows.
Also you might not understand the original problem. Imagine if 100 customers want to buy product A. One thread starts a transaction, searches for amount of product A and UPDATE's it and goes searching for other products. The database locks the row until the end of transaction and other 99 treads cannot continue until first transaction commits (they can read but cannot update the rows).
This is why they made a row per item. In this case, transaction 1 hopefully locks only several rows with items of product A. Transaction 2 instead of waiting for lock release skips them (due to SKIP LOCK) and locks several next rows. And so on.
Obviously you do not need to make a row per item - if the available amount is really large (10 000 items), you could have for example 100 rows having 100 items each. In this case each transaction locks the whole row (100 items) even if it wants to reserve just one item. The problem though is that now every row might have different amount of available items and you have to do more work to reserve the amount you want.
The point of one row per item is that thousands of concurrent shoppers don’t need to block each other as they can each claim as many free rows as they need for themselves?
1. Deduct the reservation from the inventory when the user starts to order, but in the same txn also maintain a separate row for the in progress order flow. 2. If the order flow is aborted or times out have a background process that returns these to the inventory.
That seems simpler than this approach and involves no locking. Though their presented approach is also reasonable, there must be some reason not to choose a simpler flow. It is not that difficult to have a gc service that scales, but may be they didn't want to separate that.
I don't think you really need that even. An indexed lookup is fast and you don't need to store a computed quantity generally.
1. Backgrounds process can back up
2. They need context of the user and need to switch context per user
3. What if they fail, you create some DLQ or another process to handle the failure
4. Who looks on those failure and how do they act
TLDR; there is always a cost
I disagree with the other posters about the bg process, if you have any bg processing already you should be able to handle the few edge cases without too much trouble.
Also, I wonder why they could not have a row status (available/reserved) and UPDATE it instead of deleting the rows.
It's web-scale.
gun = smoking
insight = key
gap = closed
summary = executived
https://en.wikipedia.org/wiki/Bootstrapping#Etymology
One would think semantic density would win out in training.
Close out previous paragraph. Segue to completely different topic.
How else are you supposed to go on a tangent?
Both sucks in French but at least chatgpt prose is readable while Claude is awful.
It’s also annoying as a human because Claude et al rate their own writing very highly, putting human<>LLM interactions at a disadvantage to human->LLM<>LLM interactions.
First, "the hardest lesson". What lesson? It is out of context. Nobody was talking about lessons before this.
Second, "the bottleneck wasn't what we were measuring and observing". Of course the bottleneck itself wasn't that. They couldn't discover what the bottleneck was using the information in their measurements and observations.
It is a clunky and frankly incorrect passage in an otherwise well written article.
But it suuucks, making it hard to read, the same way (some) fast/junk food is hard to swallow.
They have access to a trillion dollar writing machine god, and they choose to publish that.
Instead of having 1000 rows per shop*SKU, why not just have one row per shopping cart*SKU?
That way a single row would represent a single cart, and will hold info of multiple items of the same SKU.
No need a cludge with 1000 rows limit and replenishment process. Instead of dealing with N rows, you always deal with a single row.
So those engineers at Shopify worked hard for months on a more performant system, but they missed the obvious structure? They chose a complex denormalization for no good reason?
It may be true, but I think it's presumptuous to belittle their work when we have only partial information. My guess is that they had good reasons to think that the more obvious ways would not scale.
And from reading your comments in this thread, I believe your structure would fail at their scale. A SQL query that uses 2 sub-queries with "group by" is probably too heavy. From the post, at peaks there would be millions of active shopping carts.
BTW, I suspect most orders are just for 1 or 2 of each item, so the denormalization is not as heavy as it seems.
re concurrency, it is not a big issue at all. stock exchanges deal with HFT traders and can easily deal with concurrency of orders. Same can be implemented with shopify, but I doubt they face the same level of concurrency as stock exchange anywhere near
There's even this bit where they discover a remarkable trick:
> Each round trip to the database has a cost. For carts with multiple line items, we batch reservation queries using UNION ALL so we fetch all needed units in one round trip
Insights like that really don't read like senior level output, and of course, it's LLM output. I'm not sure it's presumptuous to question it.
At what point that row is inserted?
so the row is inserted when Payment is initiated, and row is deleted when Payment succeeds
but that system could be easily improved to reserve item when user Adds item to a cart, to prevent scenario when user adds item to a cart, goes through checkout, and after initiating payment gets "soldout error":the check for oversold items is extremely cheap:
assuming there are indexes on sku field in both, results in efficient index seek and agg over 2 tablesYou reserve the product by creating an "active_cart" entry. Your solution has a problem, that when you run the check, it might say the product is available, but before you create an "active_cart" to reserve it from thread A, another thread B reserves it and you end up reserving a product that is not available anymore. You end up with SUM(active_cart.quantity) > inventory.available_units.
That is exactly why the database has locks - to prevent this situation. With locks, thread A decrements inventory.available_units and that row is locked until the end of transaction. Other threads (if they do SELECT FOR UPDATE instead of SELECT) cannot see the old, invalid value until thread A either commits and the value is updated or rollbacks. However, locks cause performance issues and that is why shopify uses the architecture from the article - instead of 100 users fighting for the lock on the same row with available amount, each user locks only rows with units they plan to buy.
Interestingly, MySQL docs has the documentation page with a similar case: https://dev.mysql.com/blog-archive/mysql-8-0-1-using-skip-lo...
there is ultimately needs to be some global mechanism resolving this conflict. Currently it is an order in which db engine processes transactions by locking rows for a transaction, whoever got the first lock, wins the last remaining items.
my design is the same, except it does not need this dance with moving rows between tables, locking them, and the cludge with replenishment process.
in the simplest form, run the sum() over active non-finished orders and compare to inventory. you get the same result: whoever got the first to run sum() and get positive answer will get the last remaining items.
but the problem as formulated, imho, is not even correctly defined.
Shopify incorrectly formulated the very problem they are trying to solve.
Trying to solve it at the payment time is too late, its better to resolve it earlier, before the checkout.
the "PAY" button should only do one thing: deduct money from cc and that's it. Resolving inventory availability must be solved way earlier, the moment user clicks Checkout, not when user clicks Pay.
So ideally, the error for oversold items should be shown to a user when he clicks Checkout, not when he click PAY
That’s a bold overconfident statement. Cart abandonment is real. People never clear their carts they just walk away
Shopify purposefully chooses to do it at payment time because doing it earlier results in lost sales as people “reserve” items and then walk away causing other to see out of stock and then also walk away
Whoever puts up the money first gets the item
That’s the design constraint they chose you can’t just say “their solution is wrong because they solved the wrong problem”. Each design is a different user experience and I think it’s safe to say they chose which experience they want consciously.
Ok, let's accept the design goal that whoever paid first wins. You can use the same metric (how many milliseconds ago did user click PAY) and impose a global monotonic non-decreasing counter to distribute the scarce inventory. This is how order matching engines work at stock exchanges with HFT orders (FIFO logic).
the goal is to know with 100% certainty, before sending payment request to payment processor, who will have item and who won't, and you dont need to move mountains of rows for that.
the payment processor should be just a binary answer: payment succeeded or not, but currently it combines Inventory availability check & payment processing, which is the root cause of confusion. For clarity it is better to make that stage of order processing an explicit separage stage, instead of coupling it with payment stage.
some stores split payment into two stages: Payment and Final order confirmation. at the Payment stage you can pre-authorize money at cc and do inventory availability, and at final confirmation you capture $$
https://docs.stripe.com/payments/place-a-hold-on-a-payment-m...
https://support.authorize.net/knowledgebase/Knowledgearticle...
Still I think their solution is a bit weird. I'd want to commit the reservation transaction with inventory decrement along with a payment key and then use a different transaction to drop the reservation when the transaction completes. If the transaction does not complete in a timely manner you probably need to query external systems anyway to resolve whether the payment actually occurred or not.
They talk about lock contention in this case, but I also wonder about latch contention since these rows are adjacent. If it's a small transaction that's not interactive, does mysql resolve it with just the latches on the needed tables?
It resolves with skip locked. Assuming we have only 1 item left. First query scans the buffer table, locks as many rows as needed (1 in our case), and moves rows to another table. Second query scans the table, finds no rows (even if first one hasn’t finished yet, the row is locked and ignored), checks if it can increase buffer, finds out that it’s fully sold and aborts. Db guarantees that you can’t oversold.
> my design is the same, except it does not need this dance with moving rows between tables, locking them, and the cludge with replenishment process.
I can’t evaluate whether it’s the same or not, because you still haven’t clarified when exactly you’re going to insert the row. In the article they’re inserting in the same transaction. Would you also do it in the transaction? Because if you’ll introduce a separate global mechanism to resolve conflicts, on a high level it would be the same as their approach with redis (you need to have 2 systems)
EDIT: wording
now let's think again, do we need to lock 900 rows to place order on 900 items? or can we insert a single row where order_quantity=900 ?
shopify's design relies on DB to lock rows for transaction as a way to "decrement the counter" of available units. What I am suggesting, is you can just decrement counter by updating a single row, no need to lock 900 rows. Shopify moved from one extreme (single global variable in redis) to another extreme (1000 rows in db) and forgot about the middle ground.
The dance with moving rows per each item between tables is completely unnecessary, it's like counting numbers one by one in a for loop, when you can just substract number directly.
if I were to solve the problem, I would have solved it differently, at the Checkout state, before user clicks PAY. This removes the race condition at the user UI level, before any request lands in backend/db:
availability check is the SQL above: inventory-sum(active_carts.quantity)-current_order must be > 0In order to avoid races you need to insert reservation and decrement availability atomically. Your proposed approach is not atomic. For it to be atomic you will need to lock whole range, to make sure no new rows appeared between the points “check for availability” and “record reservation”. Actors will be effectively competing for the single aggregate row. This is the same as having a single inventory row with quantity field, which they rejected in the beginning of the article
> now let's think again, do we need to lock 900 rows to place order on 900 items? or can we insert a single row where order_quantity=900 ?
In the proposed schema nobody is waiting for these locks, they’re skipped by concurrent queries. In your schema actors would have to wait before they can insert without breaking invariants.
more transactions can commit at the same time, but with one counter they would conflict (as it did in the Redis case)
they should use CRDT (and trying to model that with this 1000 row workspace, no?)
still, eventually at some point they need to do the math
But the real world is different
https://www.shopify.com/careers/disciplines/engineering-data
Pair programming and forced AI, that sounds like absolute hell. Glorification of Lütke who didn't do that much in open source and now props up his ego by thinking "AI can do it so it wasn't all that difficult all along."
I don't think he ever worked on complex parts of Ruby. The people he now oppresses did.
Ruby should note that this company is actively repelling people from using the language. I really want to switch, but then I see Claude contributions in Ruby core, the influence of this slop company, and think it isn't worth it.
Oh, and they bought DHH in 2024 for his 180° turnaround on AI. He is now an AI booster, so Rails is out of the question as well.
But I guess the point is that even in the MySQL scenario the 'reserved_quantities' is almost like a temporary table so either way is not the 'Real' inventory
This section is badly written. For example, it refers to different table names than those previously introduced.
The slop shows. While I appreciate the post, I wonder why they didn't bother using an LLM in a way that would at least ensure internal consistency.
Redis handles tens of thousands of concurrent connections in a single event loop, while MySQL uses one thread per connection. No matter how I look at it, that seems like a step backward.
Of course, performance isn't everything. And if performance isn't a problem, having everything in one place does make it easier to reason about. But I'm worried that under spike traffic, this approach might actually cause more problems.
I think putting a scheduling layer in front of the DB would be a better approach. The application server could handle concurrent connections and only write to MySQL when correctness is actually needed. That seems like a cheaper way to do it. but is it different for large-scale enterprise distributed systems?
No persistence means the data gets lost if machine shuts down or process crashes. Furthermore, after restart you will need to regenerate the data which can take time. That's why Redis is a cache and not a database. You can fix the persistence issue (Redis can write WAL log, don't remember if it does fsync or not), but then Redis won't be able to handle those thousands of concurrent connections.
Redis (and other NoSQL storages) don't have some magic architecture that gives them advantages over SQL databases. They just cut corners on ACID guarantees and skip fsync. Once you start doing fsync, your transaction throughput will drop to SQL database level.
Redis also doesn't have transactions which means every app error damages the data. You will spend engineer hours investigating and fixing the problems. Transactions save so much time and worries.
But I think using only MySQL is unnecessarily expensive, just to get single transaction tracking for bug tracing. So the article's argument seems to be:
'Use only MySQL as a solution to the distributed transaction consistency problem between two different storage systems, Redis and MySQL!'
But I think using Redis is much more elegant. It's easier to scale. I'd even argue that something like Saga would be a better approach. Of course, we might just have different opinions. But in my experience, reducing layers always ends up making things more complicated in the long run.
p.s. We have different views, but I do think some of your points are valid, so I upvoted your comment
Using off the shelf software means you mostly design how to plumb things together and how to make them correct , safe and scalable.
The things you mention, on the other hand, carry the same requirements but are also much complex to develop AND to maintain.
All the biggest proponents of AI seem to be fascists.
Weird.
https://www.techwontsave.us/episode/340_shopifys_leaders_are...
But here's another link if you need something that includes the phrase "far right": https://pressprogress.ca/shopify-executives-right-wing-media...
Anyway, if you think a country having a low gdp per capita is how you measure if it should suspend voting rights for disabled people and stay at home parents, then I suspect you're not actually reading any of this.
https://www.ctvnews.ca/business/article/shopify-ceo-draws-cr...
For instance, retarded is now considered a slur. If OP is saying "my director said the R word," then I would question whether OP spared the gory details out of concern for polite company or if they're being oblique in service of their point.
And if managers at Shopify are dropping N-bombs, I want to know that too.
and i was excited to get some insight, then i realized that this whole thing was written by AI and im going to guess the idea and implementation were probably very AI driven.
> The solution: SKIP LOCKED > Core idea: one row per unit, bounded by design
cool, thanks claude.
Now I'm wondering what the engineering culture is even like at shopify.
Here's the thing. I like databases, I think there's a lot of shit in this space that went and smoked a shit ton their own good stuff to come up with these pure event driven designs that lock you into event workflows with no isolation and remove the ability to do broader bulk-functions.. and then do something even stupider and say "all you need for the interface is graphql" and such service/platform doesn't give you any other way to reconcile or do reporting for your org you have to warehouse from graphql.. this is crap. So seeing a headline where shopify says they want to kinda get behind a unified database strat behind the scenes even if it's not necessarily customer facing, like that's good imo. SQL is many decades of relational algebra that makes insane computations acrossed vast sets of data pure magic and one of the best query dml interfaces of all time.
..however i dont even agree with the claim their making here that redis isnt the tech for a reservation system. redis when used correctly feels like an insanely awesome way to do a reservation system, i lurv redis for stuff like that.
I'm just gonna go forward with the assumption that current and future shopify updates are pure vibeslop. I already hate their data interfaces, but compared to other saas offerings i appreciate that they do have bulk-features.
I really just disagreed with the assessment that redis is not good enough for the job for a reservation system. I use sql database all the time, I prefer them. But I'm seeing a claude written article here that seems to heel turn on a proven technology, it would at most be insightful if there was human content in here from actual engineers at shopify who want to vouch for and explain the challenges they were up against with redis rather than just expect me to take claudes word for it. Anyone who's been dabbling with AI knows damn well that you can convince claude to write up a dissertation on any hill you want to die on.
For example if it takes 20ms to write a batch to the WAL then if you do 5 updates to the same row then that is a minimum of 100ms. But without waiting on locks if you can batch all the WAL writes together then this could be just 20ms.
I don’t think holding locks while waiting for WAL is strictly necessary. There is definitely some anomalies that can happen if you don’t wait for WAL to be durable because transactions that don’t write WAL can observe non-durable writes in some situations. So for example conditional updates that don’t perform work. But I assume this can be fixed by making these wait on the commit for dependent transactions to become durable if they are empty. There is also the problem of failing writes that reveal information about non-durable writes which is more tricky. For example you try to insert into a unique index and it fails, but the duplicate was due to a non-durable write that is lost.
Pure reads should be fine when using MVCC because you just show the latest durable version of the DB. I know some other replication systems will run all transactions including reads through the WAL/replicated log in order to not have anomalies.
Now you only need MySQL expertise and maintenance rather than Redis and MySQL