In-memory order updater #5872

benjaminwil · 2024-10-11T21:37:40Z

Summary

This pull request proposes a replacement to the default Spree::OrderUpdater that has new and improved functionality.

We don't expect this to be the default order updater implementation in the next minor version of Solidus, but we would like to propose it as the default for the next major version of Solidus.

Note: The commits on this pull request have a long list of co-authors, as the Super Good team is approaching this as a collaborative mob programming exercise.

Below we'll add some more context about exactly what benefits the in-memory order updater brings to Solidus.

Increased performance for order updates

By default, the new order updater make fewer write calls to the database. This means that order- and checkout-related controller actions can resolve faster, with fewer round trips to the database server whenever an order is updated (which is a very often).

Order update simulation

By passing persist: false, to the in-memory order updater, an order can be recalculated and rolled back to its last state without any database writes. This enables¹ stores to let administrators and customers preview changes to orders before accepting those changes.

We know that many stores add custom features and enhancements to the order model and the order updater, so we have also added a database write query monitor that warns developers if their application code is making unintended writes when persist: false has been set.

Test performance, deeper test coverage

Because order updates require fewer round trips, this also benefits the Solidus test suite: the main side effect is that any order scaffolding and factories should just be faster.

This pull request includes additional test coverage against the current order updater as well as test coverage for the new in-memory order updater. This includes integration test coverage against both solidus_promotions and solidus_legacy_promotions.

Anatomy of this pull request

Obviously, this is a massive change. We appreciate your careful review, and if you're confused about any of these changes please leave your comments and questions. We touch a lot code across the Solidus meta gem, but all of our changes are in service of:

Ensuring all of the order dependencies (line items, shipments, adjustments, and so on) can be trusted to not generate database write unless directed to via the order updater. So, many commits concern persistence toggling, attribute assignment, and the representation of records currently loaded in memory.
Ensuring there is integrated order test coverage across all the parts of Solidus that result in order updates.
Ensuring database writes can be logged as warnings for both Solidus developers and application developers.

Checklist

Check out our PR guidelines for more details.

The following are mandatory for all PRs:

I have written a thorough PR description.
I have kept my commits small and atomic.
I have used clear, explanatory commit messages.

By enables we mean that application developers can build features that preview order changes. But as of this implementation, no part of solidus_admin, solidus_backend, or the Solidus starter frontend make use of this functionality. ↩

jarednorman · 2024-12-11T22:52:21Z

Just making a note that we are waiting on Alistair to rebase #6026 against this.

@AlistairNorman could you just make sure that gets done at some point before the 20th? That way we can have it for our next session.

While working on the in-memory updater in solidusio#5872, we found the need to change how item totals were being calculated, so that we could mark adjustments for destruction without actually destroying them, while still keeping tax adjustments intact. This change is completely backwards-compatible with the current OrderUpdater, so to reduce the scope of our PR, we wanted to make this change separately. Since the OrderUpdater is already very large, this helps reduce its responsibilities and makes it easier to test this behaviour. We don't see it as necessary to make this a configurable class, but this change leaves that option open in the future. Co-authored-by: Adam Mueller <[email protected]> Co-authored-by: Alistair Norman <[email protected]> Co-authored-by: Chris Todorov <[email protected]> Co-authored-by: Harmony Bouvier <[email protected]> Co-authored-by: Sofia Besenski <[email protected]> Co-authored-by: benjamin wil <[email protected]>

codecov · 2025-01-31T22:57:29Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.42%. Comparing base (d070f94) to head (71881b2).
⚠️ Report is 14 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5872      +/-   ##
==========================================
+ Coverage   89.35%   89.42%   +0.06%     
==========================================
  Files         961      964       +3     
  Lines       20195    20319     +124     
==========================================
+ Hits        18046    18170     +124     
  Misses       2149     2149

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

A commit was made in solidusio#6315 (5979f25) in service of the in memory order updater solidusio#5872 that changed the behavior in the OrderTaxation class to set the adjustment value but not persist it. Instead, the persistence happens when the order is saved. If a shipment is changed, and an associated adjustment is changed, the shipment adjustment will be autosaved when the order is. However, if a shipment is *not* changed, but the adjustment is, the adjustment will *not* be autosaved when the order is. A scenario where this can happen is a recalculation that just adjusts shipment taxes on an order that would otherwise not change. Let's walk through what happens in the OrderUpdater 1. `update_totals` is called which in turn... 2. calls `update_adjustment_total`, which in turn... 3. calls `recalculate_adjustments` which in turn... 4. Calls `update_tax_adjustments` - `update_tax_adjustments` creates and sets the values of the tax adjustments for a shipment. At this point, the shipment adjustment has been changed, but not persisted, and the shipment has *not* been changed. 5. Calls `update_item_totals` which calls... 6. `Spree::Config.item_total_class.new(item).recalculate!` - `Spree::Config.item_total_class` sets the `included_tax_total`, `additional_tax_total` and `adjustment_total` on the shipment to their new values, calculated from the non-persisted adjustment. At this point, both the shipment *and* adjustment have been changed but not persisted. - If you were to call `order.save!` at this point, the shipment adjustment would autosave. 7. Next, `item.update_columns` is called with the columns that the `item_total_class` updated. - At this point, the shipment, it's changes having been persisted to the database, will no longer be marked as `changed`. - This means that the default behavior of autosave in rails will *not* save the shipment when the order is saved, which means the associated (and changed) adjustment will not get saved either In solidusio#6315, we ensured that these changed associations cannot get left behind by setting `autosave: true`, which forces the model to check if it has any associations that need saving instead of telling it's relation "nope I haven't changed" We can do the same thing here for the shipment to ensure changing adjustments will make the shipment tell it's order "I have changed", even if none of it's direct properties have changed. You can test this change by removing the `autosave: true` option the adjustments and running 'when the address has changed to a different state' specs. They fail with the error "expected `order.shipments.first.adjustments.first.amount` to have changed from 1 to 2, but did not change", demonstrating a scenario where the shipment adjustment changes do not get persisted.

A commit was made in solidusio#6315 (5979f25) in service of the in memory order updater solidusio#5872 that changed the behavior in the OrderTaxation class to set the adjustment value but not persist it. Instead, the persistence happens when the order is saved. If a shipment is changed, and an associated adjustment is changed, the shipment adjustment will be autosaved when the order is. However, if a shipment is *not* changed, but the adjustment is, the adjustment will *not* be autosaved when the order is. In solidusio#6315, we ensured that these changed associations cannot get left behind by setting `autosave: true`, which forces the model to check if it has any associations that need saving instead of telling it's relation "nope I haven't changed" We can do the same thing here for the shipment to ensure changing adjustments will make the shipment tell it's order "I have changed", even if none of it's direct properties have changed. You can test this change by removing the `autosave: true` option the adjustments and running 'when the address has changed to a different state' specs. They fail with the error "expected `order.shipments.first.adjustments.first.amount` to have changed from 1 to 2, but did not change", demonstrating a scenario where the shipment adjustment changes do not get persisted.

A commit was made in #6315 (5979f25) in service of the in memory order updater #5872 that changed the behavior in the OrderTaxation class to set the adjustment value but not persist it. Instead, the persistence happens when the order is saved. If a shipment is changed, and an associated adjustment is changed, the shipment adjustment will be autosaved when the order is. However, if a shipment is *not* changed, but the adjustment is, the adjustment will *not* be autosaved when the order is. In #6315, we ensured that these changed associations cannot get left behind by setting `autosave: true`, which forces the model to check if it has any associations that need saving instead of telling it's relation "nope I haven't changed" We can do the same thing here for the shipment to ensure changing adjustments will make the shipment tell it's order "I have changed", even if none of it's direct properties have changed. You can test this change by removing the `autosave: true` option the adjustments and running 'when the address has changed to a different state' specs. They fail with the error "expected `order.shipments.first.adjustments.first.amount` to have changed from 1 to 2, but did not change", demonstrating a scenario where the shipment adjustment changes do not get persisted. (cherry picked from commit b47e0f0)

This change introduces some high-level integration tests and a patch similar to the one for the regular order updater, which provides a compatibility layer for the legacy promotion system, specifically around the eligible flag on adjustments. We want to protect against manipulative database queries in the legacy promotion system. We also want to ensure that objects in memory have their attributes changed correctly but not persisted. Co-authored-by: Jared Norman <[email protected]> Co-authored-by: Kendra Riga <[email protected]> Co-authored-by: Adam Mueller <[email protected]> Co-authored-by: Alistair Norman <[email protected]> Co-authored-by: Senem Soy <[email protected]> Co-authored-by: Sofia Besenski <[email protected]> Co-authored-by: Noah Silvera <[email protected]>

Instead of destroying. See code comments and spec changes. This is similar to how we handled this in the legacy promotion system. Co-authored-by: Adam Mueller <[email protected]> Co-authored-by: Alistair Norman <[email protected]> Co-authored-by: An Stewart <[email protected]> Co-authored-by: benjamin wil <[email protected]> Co-authored-by: Harmony Evangelina <[email protected]> Co-authored-by: Jared Norman <[email protected]> Co-authored-by: Nick Van Doorn <[email protected]> Co-authored-by: Senem Soy <[email protected]> Co-authored-by: Sofia Besenski <[email protected]>

When computing item totals after a line item benefit is removed, we need to ensure we aren't using the marked for destruction items. We also needed to make a change to the legacy promotion order updater patch because it is incorrectly being used in our tests even when using the new promotion system. It also did not have logic to ensure adjustments that were marked for destruction were ignored in the adjustment totals calculation. Co-authored-by: Jared Norman <[email protected]> Co-authored-by: Noah Silvera <[email protected]>

In an associated change we made the `CreateDiscountedItem` benefit create in-memory line items. That causes an issue during the recalculation of the item totals. Co-authored-by: benjamin wil <[email protected]> Co-authored-by: Kendra Riga <[email protected]> Co-authored-by: Noah Silvera <[email protected]> Co-authored-by: Jared Norman <[email protected]>

This will be used by the in-memory order updater to log manipulative queries when the persist flag is set to false. This will inform application developers whether their application is complying with the in-memory order updater's "no writes" philosophy. Previously the `db-query-matchers` gem we use to check for manipulative queries was only used in tests. However, we now use it in application code so the dependency must be moved to the gemspec. Co-authored-by: Alistair Norman <[email protected]> Co-authored-by: Sofia Besenski <[email protected]> Co-authored-by: Adam Mueller <[email protected]> Co-authored-by: Chris Todorov <[email protected]> Co-authored-by: benjamin wil <[email protected]>

Enabling manipulative query logging will show the number of manipulative queries on each call to Spree::InMemoryOrderUpdater#recalculate. Co-authored-by: Chris Todorov <[email protected]> Co-authored-by: Adam Mueller <[email protected]> Co-authored-by: Alistair Norman <[email protected]> Co-authored-by: Senem Soy <[email protected]> Co-authored-by: benjamin wil <[email protected]>

This method will be used in the in memory order updater. We only want to persist to the database once we perform the action, not when we are computing the amount.

We want to make sure we don't persist anything when computing amounts becuase that will cause issues with the in-memory order updater.

Co-authored-by: benjamin wil <[email protected]> Co-authored-by: Chris Todorov <[email protected]>

Instead of calling update_columns, we should just use assign_attributes everywhere and rely on our order persistance bubbling down to the line item when appropriate. This avoids issues with the in-memory order updater when attempting to update values on newly created promotional items that won't get persisted until later in the recalculation flow. Co-authored-by: Jared Norman <[email protected]> Co-authored-by: Noah Silvera <[email protected]>

Sets up some shared examples so that we can run the full promotion integration suite against both the existing order updater and the in-memory order updater. Co-authored-by: Jared Norman <[email protected]> Co-authored-by: Noah Silvera <[email protected]>

We no longer want to have individual methods use a persist: flag, instead we can save at the end if needed. Co-authored-by: Chris Todorov <[email protected]> Co-authored-by: Senem Soy <[email protected]> Co-authored-by: benjamin wil <[email protected]>

We no longer want to have individual methods use a persist: flag, instead we can save at the end if needed.

We don't want our in-memory order updater to have to persist shipment state. Because we no longer rely on `Shipment#update_state` which uses `update_column`, we are now logging any additional `Shipment#state` changes when the order is saved. The changes to the specs reflect this change in behaviour. Co-authored-by: Jared Norman <[email protected]> Co-authored-by: Noah Silvera <[email protected]> Co-authored-by: Alistair Norman <[email protected]>

Due to upstream changes after a rebase, these tests started failing because the `order` being set up (`Spree::Order.create`) did not have proper shipments or inventory units created for it. The new tests use factories to more realistically set up orders, shipments, and inventory units in various states to simulate order updater functionality. Co-authored-by: Alistair Norman <[email protected]> Co-authored-by: Kendra Riga <[email protected]> Co-authored-by: Chris Todorov <[email protected]>

On the classic order updater, we were persisting updates to shipment and payment states in the respective recalculate methods. We are no longer persisting changes in those methods; now we only save changes in `#persist_totals`, so this is where we should be logging. Co-authored-by: Senem Soy <[email protected]> Co-authored-by: Adam Mueller <[email protected]> Co-authored-by: Kendra Riga <[email protected]>

This removes the deprecation warnings from our test suite. Co-authored-by: Adam Mueller <[email protected]> Co-authored-by: Kendra Riga <[email protected]> Co-authored-by: Senem Soy <[email protected]>

We're proposing a new default for this configuration value, so this change updates the test in the new promotion system to not be dependent on the current default value. This test will fail if the default configuration changes. Co-authored-by: Benjamin Willems <[email protected]> Co-authored-by: Kendra Riga <[email protected]> Co-authored-by: Adam Mueller <[email protected]>

tvdeyen

Woooow. I am really excited to see this. Really great work which will allow many refactorings of this kind in the future and will benefit even the existing order updater.

I really would like many of the changes to be extracted into separate PRs so we can reduce the amount of change in this PR and to already introduce deprecations of the order updater method name changes.

tvdeyen · 2025-11-03T14:45:45Z

promotions/app/patches/models/solidus_promotions/order_recalculator_patch.rb

-    Spree::Config.order_recalculator_class.prepend self
+
+    Spree::OrderUpdater.prepend self
+    Spree::InMemoryOrderUpdater.prepend self


can we extract this fix into a separate PR

tvdeyen · 2025-11-03T14:47:53Z

core/app/models/spree/in_memory_order_updater.rb

      order.shipment_state
    end
+    alias_method :update_shipment_state, :recalculate_shipment_state
+    deprecate update_shipment_state: :recalculate_shipment_state, deprecator: Spree.deprecator


could be extracted into a PR

tvdeyen · 2025-11-03T14:48:16Z

core/app/models/spree/in_memory_order_updater.rb

      order.payment_state
    end
+    alias_method :update_payment_state, :recalculate_payment_state
+    deprecate update_payment_state: :recalculate_payment_state, deprecator: Spree.deprecator


could also be extracted into PR

tvdeyen · 2025-11-03T14:49:57Z

core/app/models/spree/in_memory_order_updater.rb

-      update_shipment_total
+      recalculate_payment_total
+      recalculate_item_total
+      recalculate_shipment_total


these should also be deprecated because even though they are private we all have evidence that this are overwritten in subclasses. can be extracted into a PR

mamhoff · 2025-11-03T17:35:45Z

promotions/spec/models/solidus_promotions/order_adjuster_spec.rb

  let(:promotion) { create(:solidus_promotion, apply_automatically: true) }
  let(:calculator) { SolidusPromotions::Calculators::Percent.new(preferred_percent: 10) }

+  context "adding order level adjustments" do


Can you extract this commit into its own PR? Also, I'm trying to get rid of the level language with promotion benefits. The language I'm trying to use is "order automations", so for the commit message:

We were missing a spec for automations like the `CreateDiscountedItem` benefit.

and here

context "adding discounted line items" do

jarednorman · 2025-11-05T19:07:05Z

I can't actually assign him to the PR (maybe not a member of the org), but I'm assigning @forkata to take over this from @benjaminwil, since he's no longer living in Solidus-land after today.

github-actions bot added the changelog:solidus_core Changes to the solidus_core gem label Oct 11, 2024

benjaminwil changed the title ~~In memory order updater~~ In-memory order updater Oct 11, 2024

benjaminwil force-pushed the in-memory-order-updater branch 2 times, most recently from 27f19a8 to 340032c Compare October 11, 2024 22:19

forkata force-pushed the in-memory-order-updater branch from d22210b to d244646 Compare November 8, 2024 22:34

AlistairNorman force-pushed the in-memory-order-updater branch from a3bb1fc to 06e3a2a Compare December 19, 2024 18:27

stewart force-pushed the in-memory-order-updater branch from 9fd5c8d to 27e4988 Compare December 20, 2024 21:30

jarednorman force-pushed the in-memory-order-updater branch from 4a0f623 to 90f5420 Compare January 17, 2025 22:03

jarednorman mentioned this pull request Jan 17, 2025

Refactor Line Item Total Calculations #6080

Merged

3 tasks

adammathys force-pushed the in-memory-order-updater branch from 9ecbd13 to 92fc679 Compare January 31, 2025 22:42

github-actions bot added changelog:solidus_legacy_promotions Changes to the solidus_legacy_promotions gem changelog:solidus_promotions Changes to the solidus_promotions gem labels Jan 31, 2025

AlistairNorman force-pushed the in-memory-order-updater branch from 92fc679 to e7faef3 Compare February 7, 2025 22:23

senemsoy force-pushed the in-memory-order-updater branch from f3d92a4 to a5ed6c6 Compare February 28, 2025 22:28

stewart force-pushed the in-memory-order-updater branch 2 times, most recently from d0322b2 to 07add2a Compare March 7, 2025 22:28

forkata force-pushed the in-memory-order-updater branch 2 times, most recently from d22dbf5 to 770e527 Compare March 14, 2025 21:31

nvandoorn force-pushed the in-memory-order-updater branch from 770e527 to c8e434e Compare March 28, 2025 20:22

forkata force-pushed the in-memory-order-updater branch from 7bf4adb to 2d497bd Compare April 3, 2025 18:16

jarednorman mentioned this pull request Apr 8, 2025

Cart doesn't always recalculate when the underlying data changes #6206

Open

benjaminwil force-pushed the in-memory-order-updater branch from 67186f3 to defde7e Compare April 10, 2025 17:36

AlistairNorman force-pushed the in-memory-order-updater branch from 5423a0e to 62209cf Compare April 17, 2025 19:05

benjaminwil force-pushed the in-memory-order-updater branch from 2b60e3b to 083fafe Compare May 8, 2025 19:03

Noah-Silvera mentioned this pull request Oct 17, 2025

Fix shipment adjustments not persisting on order recalculate #6334

Merged

5 tasks

Noah-Silvera and others added 18 commits October 23, 2025 11:31

Don't persist line item actions in compute_amount

4a344d1

This method will be used in the in memory order updater. We only want to persist to the database once we perform the action, not when we are computing the amount.

Add tests to ensure compute amount doesn't persist

5d62337

We want to make sure we don't persist anything when computing amounts becuase that will cause issues with the in-memory order updater.

Add call stack to ManipulativeQueryMonitor warnings

14c3d61

Co-authored-by: benjamin wil <[email protected]> Co-authored-by: Chris Todorov <[email protected]>

Rework item total updating to not use persist flag

de5feb7

We no longer want to have individual methods use a persist: flag, instead we can save at the end if needed.

Update classic order updater specs

be8c3a1

This removes the deprecation warnings from our test suite. Co-authored-by: Adam Mueller <[email protected]> Co-authored-by: Kendra Riga <[email protected]> Co-authored-by: Senem Soy <[email protected]>

senemsoy force-pushed the in-memory-order-updater branch from fd321b6 to 71881b2 Compare October 23, 2025 18:32

tvdeyen reviewed Nov 3, 2025

View reviewed changes

mamhoff reviewed Nov 3, 2025

View reviewed changes

Noah-Silvera mentioned this pull request Nov 6, 2025

Rename order updater methods to better reflect behaviour #6368

Open

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

In-memory order updater #5872

In-memory order updater #5872

benjaminwil commented Oct 11, 2024 •

edited

Loading

Uh oh!

jarednorman commented Dec 11, 2024

Uh oh!

codecov bot commented Jan 31, 2025 •

edited

Loading

Uh oh!

tvdeyen left a comment

Uh oh!

tvdeyen Nov 3, 2025

Uh oh!

tvdeyen Nov 3, 2025

Uh oh!

tvdeyen Nov 3, 2025

Uh oh!

tvdeyen Nov 3, 2025

Uh oh!

mamhoff Nov 3, 2025

Uh oh!

jarednorman commented Nov 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

Uh oh!

In-memory order updater #5872

Are you sure you want to change the base?

In-memory order updater #5872

Conversation

benjaminwil commented Oct 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Increased performance for order updates

Order update simulation

Test performance, deeper test coverage

Anatomy of this pull request

Checklist

Footnotes

Uh oh!

jarednorman commented Dec 11, 2024

Uh oh!

codecov bot commented Jan 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

tvdeyen left a comment

Choose a reason for hiding this comment

Uh oh!

tvdeyen Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

tvdeyen Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

tvdeyen Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

tvdeyen Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

mamhoff Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

jarednorman commented Nov 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

benjaminwil commented Oct 11, 2024 •

edited

Loading

codecov bot commented Jan 31, 2025 •

edited

Loading