Add precompiled query generation to the dbcontext optimize command #33747

AndriySvyryd · 2024-05-17T22:49:42Z

Also adds publish integration for precompiled queries and combines it with compiled model generation.

For the compiled model and precompiled queries to be generated when publishing with NativeAOT the only action needed is to reference Microsoft.EntityFrameworkCore.Tasks from all projects containing a DbContext or a query.

For solutions where specifying the startup project is necessary, EFStartupProject should be set.

EFOptimizeContext can be set to true to enable code generation outside of NativeAOT.

EFScaffoldModelStage and EFPrecompileQueriesStage can be set to either publish or build to control at what stage will the code be generated. Any other value will disable the corresponding generation (in case the code is generated manually using dotnet ef dbcontext optimize)

If there's more than one context and DbContextName is not set, then the compiled model will be generated for all of them.

EFTargetNamespace and EFOutputDir can be used to further fine-tune the generation.

Fixes #33103
Fixes #33558

src/EFCore.Design/Query/Internal/PrecompiledQueryCodeGenerator.cs

AndriySvyryd · 2024-05-24T23:44:54Z

src/EFCore.Tasks/buildTransitive/Microsoft.EntityFrameworkCore.Tasks.targets

+             ContinueOnError="$(ContinueOnError)"
             Condition="'$(PublishAot)'=='true'"
-             Properties="Configuration=$(Configuration);Platform=$(Platform);EFOptimizeContext=false;PublishAot=false" />
+             Properties="Configuration=$(Configuration);Platform=$(Platform);PublishAot=false;_EFGenerationStage=$(_EFGenerationStage)" />


Do we need to always pass Configuration and Platform? Should we pass other properties?

AndriySvyryd · 2024-05-24T23:55:20Z

src/EFCore.Tasks/buildTransitive/Microsoft.EntityFrameworkCore.Tasks.targets

          DependsOnTargets="_EFProcessGeneratedFiles"
-          Condition="'$(EFOptimizeContext)'=='true'"
+          Condition="'$(_EFGenerationStage)'==''"
          Inputs="$(MSBuildAllProjects);


Is there a better way of determining whether CoreCompile is going to be skipped other than duplicating its inputs and outputs?

AndriySvyryd · 2024-05-24T23:57:07Z

src/EFCore.Design/Design/Internal/DbContextOperations.cs

+
+        MSBuildLocator.RegisterDefaults();
+        // TODO: pass through properties
+        var workspace = MSBuildWorkspace.Create();


What properties should we pass through from the MsBuild invocation?

Good question, I'm not sure... I'm assuming it's reasonable to expect the project to be buildable as-is (without properties), but the user may have tweaked their project with various properties...

Even if it's buildable the code might contain #ifdefs that significantly alter the logic based on DefineConstants.

roji · 2024-05-31T22:33:29Z

Just to say that this is on my list to review... Will work up to it in the next few days.

AndriySvyryd · 2024-05-31T22:34:59Z

Just to say that this is on my list to review... Will work up to it in the next few days.

Ok. Just note that the longer you take to review the more code will need to be reviewed 😉

roji · 2024-05-31T22:47:32Z

That's a pretty effective way to speed up a code review :)

roji

LGTM, the MSBuild Tasks part is a total 🐑 🇮🇹 though :) Is there someone who knows this part a bit (@ajcvickers)?

Just to confirm: after this PR, the precompiled query code will use the unsafe accessors generated into the compiled model, rather than generate its own (via the code I wrote in LinqToCSharpSyntaxTranslator), right? That would be a good thing, just making sure.

src/EFCore.Design/Design/Internal/DbContextOperations.cs

roji · 2024-06-05T10:05:32Z

src/EFCore.Design/Design/Internal/DbContextOperations.cs

+
+        MSBuildLocator.RegisterDefaults();
+        // TODO: pass through properties
+        var workspace = MSBuildWorkspace.Create();


Good question, I'm not sure... I'm assuming it's reasonable to expect the project to be buildable as-is (without properties), but the user may have tweaked their project with various properties...

src/EFCore.Design/Design/Internal/DbContextOperations.cs

src/EFCore.Design/Design/OperationExecutor.cs

src/EFCore.Design/Scaffolding/Internal/CSharpModelGenerator.cs

src/EFCore.Design/Query/Internal/PrecompiledQueryCodeGenerator.cs

roji · 2024-06-05T10:20:55Z

src/EFCore.Design/Query/Internal/PrecompiledQueryCodeGenerator.cs

+"""
+        );
+
+        var name = Uniquifier.Uniquify(


We could also uniquify only at the very top, i.e. before actually dumping the files to disk, rather than dealing with it at each point where we create a ScaffoldedFile... In that sense the Name would only be the "desired" name (or a hint).

The uniquifier needs to be before the suffix (e.g. .g) and before the extension. If we do it at the top then we'd either need to parse the name or make ScaffoldedFile to contain all the building parts of the name separately.

Another problem is that even if the filenames are uniquified, the class names could still be clashing. I haven't solved this yet, but it would need to be done at this level anyway.

src/EFCore.Design/Query/Internal/PrecompiledQueryCodeGenerator.cs

src/EFCore/Metadata/Internal/IRuntimeModel.cs

src/EFCore.Tasks/buildTransitive/Microsoft.EntityFrameworkCore.Tasks.props

rainersigwald · 2024-06-05T15:54:52Z

I would find it very helpful if there were a Markdown doc describing what this is doing, ideally with some Mermaid diagrams. I was going to try to document it myself but I'm getting lost in the details. Can you share a binlog of a project building/publishing with this stuff turned on that I can look at?

AndriySvyryd · 2024-06-05T17:36:55Z

Just to confirm: after this PR, the precompiled query code will use the unsafe accessors generated into the compiled model, rather than generate its own (via the code I wrote in LinqToCSharpSyntaxTranslator), right? That would be a good thing, just making sure.

Yes, and I added an Assert in case any unsafe accessor is generated by LinqToCSharpSyntaxTranslator

AndriySvyryd · 2024-06-05T17:58:44Z

@rainersigwald There are basically two flows:
For Build

_EFReadGeneratedFilesList and _EFProcessGeneratedFiles add the files generated previously to Compile to make incremental build work
If compilation needs to be performed again then _EFPrepareForCompile removes the previously generated files from Compile as they are probably outdated. And if EFOptimizeContext is true it also calls _EFRegisterProjectToOptimize on the startup project to mark the current project for optimization. Startup project could also be the same as the project containing the derived DbContext, but if it's not the tooling needs the compiled startup assembly to be able to generate code for any other project.
If any project was marked then _EFGenerateFilesAfterBuild in the startup project calls _EFGenerateFiles which in turn calls OptimizeDbContext on the marked projects
OptimizeDbContext generated optimized/NativeAOT-compatible code and writes the list of generated files for _EFReadGeneratedFilesList to read when recompiling

For Publish

If PublishAOT is true _EFPrepareDependenciesForPublishAOT in the startup project invokes _EFPrepareForPublish on all dependencies to mark them for optimization even if they don't set EFOptimizeContext to true. Otherwise _EFPrepareForPublish runs on the projects before Publish
If any project was marked then _EFGenerateFilesBeforePublish in the startup project calls _EFGenerateFiles and the rest is similar to the Build flow

rainersigwald · 2024-06-05T18:05:31Z

Startup project could also be the same as the project containing the derived DbContext, but if it's not the tooling needs the compiled startup assembly to be able to generate code for any other project.

Am I understanding correctly that this means you must first build the whole solution (or a meaningful subset), then generate code, then rebuild all of the assemblies that had code generated?

AndriySvyryd · 2024-06-05T20:19:20Z

Am I understanding correctly that this means you must first build the whole solution (or a meaningful subset), then generate code, then rebuild all of the assemblies that had code generated?

Yes. And it's even worse when PublishAOT is true as the produced assembly cannot be used for code generation. So, in that case we build the solution with PublishAOT set to true, then if code needs to be generated we rebuild the solution with PublishAOT set to false, generate code then rebuild the solution with PublishAOT set to true

roji · 2024-06-06T07:09:40Z

BTW @AndriySvyryd consider copy-pasting some version of the explanation above into the targets themselves for future us.

Part of #33103

…unless a single one is specified Fixes #33558

AndriySvyryd requested a review from roji May 17, 2024 22:49

AndriySvyryd force-pushed the Issue33103 branch 2 times, most recently from 881f8e9 to b840cc6 Compare May 21, 2024 01:07

AndriySvyryd marked this pull request as ready for review May 21, 2024 01:07

AndriySvyryd commented May 21, 2024

View reviewed changes

src/EFCore.Design/Query/Internal/PrecompiledQueryCodeGenerator.cs Show resolved Hide resolved

AndriySvyryd force-pushed the Issue33103 branch 4 times, most recently from 620a708 to 0eaf687 Compare May 24, 2024 23:17

AndriySvyryd commented May 24, 2024

View reviewed changes

AndriySvyryd force-pushed the Issue33103 branch 4 times, most recently from 3d181c1 to b067140 Compare May 31, 2024 21:03

roji mentioned this pull request Jun 5, 2024

Work on Cosmos primitive collections, subquery and general query infra #33895

Merged

roji approved these changes Jun 5, 2024

View reviewed changes

AndriySvyryd force-pushed the Issue33103 branch from b067140 to c399751 Compare June 5, 2024 22:05

AndriySvyryd and others added 2 commits June 6, 2024 15:59

Add precompiled query generation to the dbcontext optimize command

7d016cd

Part of #33103

Add publish integration for precompiled queries

76352f5

AndriySvyryd added 2 commits June 6, 2024 15:59

Auto-generate compiled models for all contexts in the given assembly …

4122644

…unless a single one is specified Fixes #33558

Review feedback

ced7ad1

AndriySvyryd force-pushed the Issue33103 branch from c399751 to ced7ad1 Compare June 6, 2024 22:59

AndriySvyryd merged commit 4b80cb9 into main Jun 7, 2024

AndriySvyryd deleted the Issue33103 branch June 7, 2024 00:19

AndriySvyryd mentioned this pull request Jun 19, 2024

Document MsBuild integration dotnet/EntityFramework.Docs#4700

Closed

Add precompiled query generation to the dbcontext optimize command #33747

Add precompiled query generation to the dbcontext optimize command #33747

Uh oh!

Conversation

AndriySvyryd commented May 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

roji commented May 31, 2024

Uh oh!

AndriySvyryd commented May 31, 2024

Uh oh!

roji commented May 31, 2024

Uh oh!

roji left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rainersigwald commented Jun 5, 2024

Uh oh!

AndriySvyryd commented Jun 5, 2024

Uh oh!

AndriySvyryd commented Jun 5, 2024

Uh oh!

rainersigwald commented Jun 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AndriySvyryd commented Jun 5, 2024

Uh oh!

roji commented Jun 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AndriySvyryd commented May 17, 2024 •

edited

Loading

rainersigwald commented Jun 5, 2024 •

edited

Loading