Dictionary (hstore, json, and jsonb) query support #3285

yinzara · 2024-09-19T18:44:41Z

This PR adds support for querying on hstore type columns which can map to either Dictionary<string, string> or ImmutableDictionary<string, string> in C# and for querying any other Dictionary<,> or ImmutableDictionary<,> for json and jsonb types.

See npgsql/doc#369 for full documentation.

Fixes #212

test/EFCore.PG.FunctionalTests/Query/HstoreQueryTest.cs

src/EFCore.PG/Query/ExpressionTranslators/Internal/NpgsqlHstoreTranslator.cs

test/EFCore.PG.FunctionalTests/Query/HstoreQueryTest.cs

test/EFCore.PG.FunctionalTests/TestModels/Dictionary/DictionaryContainerEntity.cs

test/EFCore.PG.FunctionalTests/Query/HstoreQueryTest.cs

roji

Thanks @yinzara, looks pretty good overall! See comments below.

roji · 2024-09-22T18:21:19Z

I attempted Keys and Values and could never get it to work correctly.

If you translate Keys via akeys, that simply returns an array, so shouldn't be complicated (note that the provider will automatically call unnest on that array if you start composing arbitrary operators on it). If you translate via skeys which returns a set, things are definitely more complicated.

yinzara · 2024-10-11T09:26:03Z

@roji
I think I've got another incarnation that's worth your eyes to at least give some guidance if you think I'm headed in the right direction. I still have a few more tests to write if you agree.

This PR is now focused 100% on all forms of Dictionary<string,TValue> and ImmutableDictionary<string,TValue> based querying no matter if it's hstore, json or jsonb no matter their TValue types.

I removed all of the DBFunction extensions that could ever have a translation that would be supported without the DBFunctions. A future PR (or maybe if I get ansy) can address doing the expression visitor changes to support the other queries.

I now have a test case with a Dictionary<string, Dictionary<string, string>> NestedDictionary {get;set;} that shows how the functionality even works recursively :-) and another test case with Dictionary<string, int> to show it works with numbers as well.

Finally I made the new DictionaryTranslator not look up any TypeMappings inside the constructor so it's very efficient to create new instances of it with no overhead.

This does not fix dotnet/efcore#26903 as I was thinking it might, but it still seems quite useful overall and eventually it could support it.

yinzara · 2024-11-22T16:51:06Z

I know it's been a while since you've had time to look at this. Any chance you could take a peak and at least tell me if you are ok with my direction now? If so I'll update all the test cases and make sure it's ready for merge.

yinzara · 2024-11-22T16:53:20Z

src/EFCore.PG/Query/Expressions/Internal/PgBinaryExpression.cs

                    PgExpressionType.JsonExists => "?",
                    PgExpressionType.JsonExistsAny => "?|",
                    PgExpressionType.JsonExistsAll => "?&",
+                    PgExpressionType.JsonValueForKeyAsText => "->>",


I realize this isn't necessary now and I'll make sure it's removed before the PR merges.

roji · 2024-11-23T16:05:17Z

@yinzara I'll do my best to look soon, but it's a very busy time in the post 9.0 phase; so it may take a little while. Don't hesitate to ping me again in a month or two if you see no progress here.

yinzara · 2025-02-20T01:17:52Z

I've now rebased this PR off of the hotfix/9.4 release as the main branch has an SDK that has not been released publically yet.

All tests are now passing in this PR and I think it's ready for a real review.

Would this go into a maintenance release of 9 or will this PR eventually have to be targeted against main before it can be merged so that it only comes out with 10?

roji · 2025-02-20T08:57:55Z

@yinzara sorry this is taking so long to merge. In principle, as this is a new feature, it needs to go into 10 and not into 9; aside from a few exceptions, only bug fixes go into patch releases. So can you please rebase back on top of main? Re the preview SDK, it should be pretty easy to set up - you can use the dotnet-install scripts to download the specific version of the SDK specified in the project's global.json, and use that.

yinzara · 2025-02-20T17:06:14Z

I've rebased against main. Thank you! Happy to make any other changes you'd like.

yinzara · 2025-02-20T17:11:29Z

src/EFCore.PG/Query/ExpressionTranslators/Internal/NpgsqlDictionaryTranslator.cs

+        SqlExpression? instance,
+        MethodInfo method,
+        IReadOnlyList<SqlExpression> arguments,
+        IDiagnosticsLogger<DbLoggerCategory.Query> logger)


Here I attempted to "return null quickly" as much as possible so that you did the least amount of operations before returning null. The logic below was what I found to be the best combination while still being readable and efficient.

yinzara · 2025-02-20T17:21:42Z

src/EFCore.PG/Query/ExpressionTranslators/Internal/NpgsqlDictionaryTranslator.cs

+    ///     any release. You should only use it directly in your code with extreme caution and knowing that
+    ///     doing so can result in application failures when updating to a new Entity Framework Core release.
+    /// </summary>
+    public SqlExpression? Keys(SqlExpression instance)


I choose to make these methods public and not private so they could be used externally to this translator eventually (i.e. in some expression visitor translator for more complex translations). Honestly internal probably would have been enough but I didn't see that used in a lot of places.

All public methods accept arguments of either json/jsonb or hstore arguments and return the translation or null if the arguments are not json/jsonb/hstore.

Thoughts?

yinzara · 2025-02-20T17:24:48Z

src/EFCore.PG/Query/ExpressionTranslators/Internal/NpgsqlMemberTranslatorProvider.cs

                new NpgsqlDateTimeMemberTranslator(typeMappingSource, sqlExpressionFactory),
                new NpgsqlJsonDomTranslator(typeMappingSource, sqlExpressionFactory, model),
                new NpgsqlLTreeTranslator(typeMappingSource, sqlExpressionFactory, model),
+                new DictionaryTranslator(typeMappingSource, sqlExpressionFactory, model),


Needs to be before the JsonPocoTranslator as it needs to intercept methods on Dictionary types that are json types.

src/EFCore.PG/Query/ExpressionTranslators/Internal/NpgsqlMemberTranslatorProvider.cs

src/EFCore.PG/Query/Internal/NpgsqlQueryableMethodTranslatingExpressionVisitor.cs

Fixes npgsql#212

yinzara · 2025-02-20T18:43:48Z

I have now verified that all tests are passing locally after the rebase to main and all unused code and formatting changes have been removed.
I believe this is ready for a final review and approval for testing in the build system.

yinzara · 2025-04-04T02:46:32Z

Ping

yinzara · 2025-05-09T18:19:56Z

I know this is kind of a big PR to review but I really feel like this could be a big .NET 10 feature. Queryable dictionaries of every type seems pretty helpful!

Any chance you could get back to it? @roji

yinzara · 2025-06-09T15:32:23Z

@roji It has now been 10 months from my original submission. This feature is one that has been requested by users for more than 8 years and it is fully implemented. You had said you would get to this after the .NET 9 release which was now more than 6 months ago. This feature is fully implemented IMHO and ready for merge.

Can I please get a review?

klinki · 2025-09-09T10:32:31Z

This is a great PR, I hope it will get into version 10 release. Especially mapping of Dictionary<string, string> is super useful.

roji · 2025-09-09T11:23:07Z

@yinzara sorry that this hasn't received more attention in the previous months, I've simply been too busy (the PG provider has generally received very little attention). I admit this isn't on the top of my list generally, since hstore isn't exactly a widely-used/requested feature (the fact that something is open for X years isn't an indication that it's highly-needed, especially given it has received 6 votes total in all that time).

But in any case I'll do my best to devote some time to EFCore.PG in the coming weeks and try to get this merged for 10.

yinzara force-pushed the feature/212 branch 2 times, most recently from 8f6dad8 to 49431a8 Compare September 19, 2024 19:21

yinzara commented Sep 19, 2024

View reviewed changes

test/EFCore.PG.FunctionalTests/Query/HstoreQueryTest.cs Outdated Show resolved Hide resolved

yinzara force-pushed the feature/212 branch 2 times, most recently from 92992e5 to 1d6ee17 Compare September 19, 2024 21:38

roji mentioned this pull request Sep 22, 2024

Check why we avoid parameterizing indexer method parameters in the funcletizer dotnet/efcore#34733

Open

roji reviewed Sep 22, 2024

View reviewed changes

roji requested changes Sep 22, 2024

View reviewed changes

yinzara force-pushed the feature/212 branch 21 times, most recently from df2adf1 to b4e4966 Compare October 4, 2024 07:15

yinzara mentioned this pull request Oct 24, 2024

JSON viewer for working with JSON/JSONB columns jakob/Postico#395

Open

yinzara commented Nov 22, 2024

View reviewed changes

yinzara force-pushed the feature/212 branch from 79293d7 to b8d82db Compare February 20, 2025 01:13

yinzara changed the base branch from main to hotfix/9.0.4 February 20, 2025 01:14

yinzara force-pushed the feature/212 branch from b8d82db to 3c3a816 Compare February 20, 2025 01:15

yinzara force-pushed the feature/212 branch from 3c3a816 to 3da5e9d Compare February 20, 2025 17:05

yinzara changed the base branch from hotfix/9.0.4 to main February 20, 2025 17:05

yinzara commented Feb 20, 2025

View reviewed changes

src/EFCore.PG/Query/ExpressionTranslators/Internal/NpgsqlMemberTranslatorProvider.cs Show resolved Hide resolved

yinzara commented Feb 20, 2025

View reviewed changes

src/EFCore.PG/Query/ExpressionTranslators/Internal/NpgsqlMemberTranslatorProvider.cs Outdated Show resolved Hide resolved

yinzara commented Feb 20, 2025

View reviewed changes

src/EFCore.PG/Query/Internal/NpgsqlQueryableMethodTranslatingExpressionVisitor.cs Show resolved Hide resolved

yinzara force-pushed the feature/212 branch 2 times, most recently from 9bcdc30 to c76f04d Compare February 20, 2025 18:24

Dictionary query support

ce54d29

Fixes npgsql#212

yinzara force-pushed the feature/212 branch from c76f04d to ce54d29 Compare February 20, 2025 18:39

yinzara requested a review from roji February 20, 2025 18:44

Uh oh!

Dictionary (hstore, json, and jsonb) query support #3285

Are you sure you want to change the base?

Dictionary (hstore, json, and jsonb) query support #3285

Uh oh!

Conversation

yinzara commented Sep 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

roji left a comment

Choose a reason for hiding this comment

Uh oh!

roji commented Sep 22, 2024

Uh oh!

yinzara commented Oct 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yinzara commented Nov 22, 2024

Uh oh!

yinzara Nov 22, 2024

Choose a reason for hiding this comment

Uh oh!

roji commented Nov 23, 2024

Uh oh!

yinzara commented Feb 20, 2025

Uh oh!

roji commented Feb 20, 2025

Uh oh!

yinzara commented Feb 20, 2025

Uh oh!

yinzara Feb 20, 2025

Choose a reason for hiding this comment

Uh oh!

yinzara Feb 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yinzara Feb 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yinzara commented Feb 20, 2025

Uh oh!

yinzara commented Apr 4, 2025

Uh oh!

yinzara commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yinzara commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

klinki commented Sep 9, 2025

Uh oh!

roji commented Sep 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yinzara commented Sep 19, 2024 •

edited

Loading

yinzara commented Oct 11, 2024 •

edited

Loading

yinzara Feb 20, 2025 •

edited

Loading

yinzara commented May 9, 2025 •

edited

Loading

yinzara commented Jun 9, 2025 •

edited

Loading