Add Gurobi Direct MINLP solver in contrib.solver #3745

emma58 · 2025-10-06T17:25:16Z

Fixes #3522 (except for the request to have a persistent MINLP interface to Gurobi)

Summary/Motivation:

This adds a new solver interface gurobi_direct_minlp that supports sending MINLP problems to Gurobi via the Python API. currently this is implemented as a separate Gurobi direct interface in contrib.solver so that we don't have negative performance implications for gurobi_direct_v2 (and to punt for the moment on integrating the matrix-vector inferface with the general MINLP one).

Changes proposed in this PR:

Adds a walker that makes the (required, in Gurobi 12) conversion from general nonlinear constraints to a mixture of aux_var == general_nonlinear_expr equalities and linear/quadratic constraints involving the auxiliary and other variables.
Implements a pretty minimalistic writer that uses the above walker: The special thing here is that some Pyomo constraints will create more than one constraint in the gurobi model because of the transformation mentioned in the bullet point above.
Implements a new solver interface that inherits from GurobiDirect in contrib.solver: The only thing special here is the solve function which is again just unique because of the custom walker/writer explained above.
Gets the test coverage to 89%, which rounds to 90%... The only thing interesting here is that I added a method for converting general nonlinear Gurobi expression trees back to Pyomo expressions so that we can test them that way.

Outstanding TODOs, none of which should prevent merging unless we want them to:

Move all the tests that manually test Gurobi's expression tree onto the gurobi_nl_to_pyo_expr function, which will make them a lot more readable, and easier to maintain if every there is a change to the public API in gurobipy.
Add some tests for the sake of coverage--we're missing some cases

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

I agree my contributions are submitted under the BSD license.
I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

…n linear walker

… from Gurobi's perspective. Starting to write the gurobi walker modeled after the sympy one

…on visitor, no real tests to speak of yet though

…I try to get the constraint objects...

…ngs like AbsExpression, we can't rely completely on the expression evaluation visitor, and I need to know the types of expressions. Starting to build out tests of the walker.

…urned it

…inear expression tree, yay!!

…ressions in the RHS

…sionExpressions from the writer

emma58 · 2025-10-15T22:57:10Z

@jsiirola @mrmundt If the tests pass, I really am done messing with this now, I think!

jsiirola

A couple minor changes, and this should be good to go!

jsiirola · 2025-10-17T05:12:29Z

pyomo/contrib/solver/solvers/gurobi_direct.py

            raise NoSolutionError()

-        iterator = zip(self._pyo_vars, self._grb_vars.x.tolist())
+        iterator = zip(self._pyo_vars, map(operator.attrgetter('x'), self._grb_vars))


Is fetching the values one at a time slower than getting them in a single vector? (Same question applies to get_primals, get_duals. and get_redued_costs below)

I don't know? If it is, this is much harder to generalize because the Gurobi MINLP version doesn't have the gurobi vars as a vector--they're already a list.

jsiirola · 2025-10-17T05:12:57Z

pyomo/contrib/solver/solvers/gurobi_direct.py

-                gurobi_model, A, x, repn.rows, repn.columns, repn.objectives
+                gurobi_model,
+                A,
+                x.tolist(),


Why convert x to a list?

Because of the thing you commented on above, actually. I'm making this look the same as Gurobi MINLP so that they can use an identical SolutionLoader. Gurobi MINLP doesn't have the luxury of declaring the variables as a vector (I don't think? Although if I can dynamically add to one, that's another option, maybe?), so they're a list at this point right now.

jsiirola · 2025-10-17T05:13:42Z

pyomo/contrib/solver/solvers/gurobi_direct.py

+                gurobi_model,
+                A,
+                x.tolist(),
+                list(map(operator.itemgetter(0), repn.rows)),


Why de-tupalize rows? aren't both values needed by the loader?

I don't think so--the only time we need anything from this at all is to get the duals. And then you have to get the Pi attribute off each constraint. The difference is that was vectorized before too...

pyomo/core/expr/numeric_expr.py

pyomo/contrib/solver/solvers/gurobi_direct_minlp.py

mrmundt · 2025-10-17T19:48:00Z

pyomo/contrib/solver/tests/solvers/gurobi_to_pyomo_expressions.py

@@ -0,0 +1,78 @@
+from pyomo.common.dependencies import attempt_import


Needs copyright.

mrmundt · 2025-10-17T19:48:12Z

pyomo/contrib/solver/tests/solvers/test_gurobi_minlp.py

@@ -0,0 +1,226 @@
+import math


Needs copyright.

mrmundt · 2025-10-17T19:48:30Z

pyomo/contrib/solver/tests/solvers/test_gurobi_minlp_walker.py

+#  ___________________________________________________________________________
+#
+#  Pyomo: Python Optimization Modeling Objects
+#  Copyright (c) 2008-2024


Wrong copyright year.

mrmundt · 2025-10-17T19:48:43Z

pyomo/contrib/solver/solvers/gurobi_direct_minlp.py

+#  ___________________________________________________________________________
+#
+#  Pyomo: Python Optimization Modeling Objects
+#  Copyright (c) 2008-2024


Wrong copyright year.

mrmundt · 2025-10-17T19:49:22Z

pyomo/contrib/solver/tests/solvers/test_gurobi_minlp_writer.py

+#  ___________________________________________________________________________
+#
+#  Pyomo: Python Optimization Modeling Objects
+#  Copyright (c) 2008-2024


Wrong copyright year.

mrmundt · 2025-10-17T19:52:47Z

I am not pushing the copyright changes yet - waiting for tests to finish since all the copyright stuff is NFC.

mrmundt · 2025-10-17T19:53:01Z

But I DO have the changes ready to push as soon as the tests finish.

mrmundt · 2025-10-17T19:55:37Z

pyomo/contrib/solver/tests/solvers/test_gurobi_minlp_walker.py

+        x1 = visitor.var_map[id(m.x1)]
+        x2 = visitor.var_map[id(m.x2)]
+
+        # Also linear, whoot!


Not changing but this is an abomination of a word. Woot? Yes. Hoot? Yes. Whoo? Yes. But WHOOT???

emma58 and others added 30 commits January 7, 2025 13:48

Initial (not working) draft of Gurobi direct MINLP walker

653f7a6

Adding minimum gurobipy version

d2edb21

blackify

33ce433

Notes on how we're actually going to have to implement it--building o…

ad3c8a3

…n linear walker

Switching onto quadratic walker to try to write constraints correctly…

7ad1794

… from Gurobi's perspective. Starting to write the gurobi walker modeled after the sympy one

Adding start of integration test

4dcb73e

Making the test file names make sense

162ab75

Working version of the Gurobi walker based on the expression evaluati…

16def08

…on visitor, no real tests to speak of yet though

black

dbac17f

The gurobi printStats method agrees with me, but not what I get when …

a9f6c3a

…I try to get the constraint objects...

Going back to an explicity exit node dispatcher because for weird thi…

62b93d2

…ngs like AbsExpression, we can't rely completely on the expression evaluation visitor, and I need to know the types of expressions. Starting to build out tests of the walker.

black

01b3a7a

Fixing a bug in the UnaryExpression handler, more tests

a6d01a6

black

f8b8709

Merge branch 'main' into gurobi-minlp

c362117

Changes I don't remember for Gurobi MINLP

8c23638

Almost figured out how to test the Gurobi expression tree, woohoo

c3d1350

Fixing a bug where I forgot to update the gurobipy model before I ret…

b53b8ef

…urned it

Moving all the hacky tests to use the public API for getting the nonl…

536844b

…inear expression tree, yay!!

black

65980c9

Merge branch 'main' into gurobi-minlp

ec3aeed

Fixing a typo

03ba09b

Adding a fix and a test for a bug where we didn't account for NPV exp…

767428f

…ressions in the RHS

Adding a fix and test for a problem calling _apply_operation for Divi…

9eed315

…sionExpressions from the writer

Fixing a bug with handling of fixed variables

39b7ea1

black

971f324

Merge branch 'main' into gurobi-minlp

4142b2f

Adding missing imports

51f35ec

BooleanVars are welcome to hang out on the model

7609f09

Adding support for named expressions

1cda2fc

emma58 added 3 commits October 15, 2025 16:49

black

98b3e76

Emma learns to spell accommodating

a5dd19e

Merge branch 'main' into gurobi-minlp

8e515bc

jsiirola reviewed Oct 17, 2025

View reviewed changes

blnicho moved this from Todo to Review In Progress in Pyomo 6.9.5 Release Oct 17, 2025

emma58 and others added 10 commits October 17, 2025 10:09

Not copying result in apply_operation for sum

66b76dd

Accommodating general variable domains

a789a18

NFC: comment correction

7ccb122

NFC: comment typo

917026b

Merge branch 'main' into gurobi-minlp

cacb14e

NFC: apply black

bf3401c

Sorting unrecognized components

2a0c67e

Merge remote-tracking branch 'emmaFork/gurobi-minlp' into gurobi-minlp

ebd031f

Ensure the unknown component exception is deterministic

a9d049b

Prevent gurobipy from being imported with environ

dfa97dd

mrmundt reviewed Oct 17, 2025

View reviewed changes

pyomo/contrib/solver/tests/solvers/test_gurobi_minlp.py

@@ -0,0 +1,226 @@

import math

Copy link

Contributor

mrmundt Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs copyright.

mrmundt reviewed Oct 17, 2025

View reviewed changes

Fix copyright statements

a69dc6c

mrmundt approved these changes Oct 17, 2025

View reviewed changes

github-project-automation bot moved this from Review In Progress to Reviewer Approved in Pyomo 6.9.5 Release Oct 17, 2025

mrmundt merged commit 590ecfd into Pyomo:main Oct 17, 2025
1 check was pending

github-project-automation bot moved this from Reviewer Approved to Done in Pyomo 6.9.5 Release Oct 17, 2025

jsiirola approved these changes Oct 17, 2025

View reviewed changes

		@@ -0,0 +1,78 @@
		from pyomo.common.dependencies import attempt_import

Add Gurobi Direct MINLP solver in contrib.solver #3745

Add Gurobi Direct MINLP solver in contrib.solver #3745

Uh oh!

Conversation

emma58 commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Fixes #3522 (except for the request to have a persistent MINLP interface to Gurobi)

Summary/Motivation:

Changes proposed in this PR:

Legal Acknowledgement

Uh oh!

emma58 commented Oct 15, 2025

Uh oh!

jsiirola left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emma58 Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrmundt commented Oct 17, 2025

Uh oh!

mrmundt commented Oct 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

emma58 commented Oct 6, 2025 •

edited

Loading

emma58 Oct 17, 2025 •

edited

Loading