Update docs and JSON schema with "query-only" encryption changes #18

CDThomas · 2024-10-17T01:10:13Z

This change updates the EQL query docs to use the new field "q" (for query). This field tells proxy to perform encryption for a specific query operation (instead of performing source encryption and encryption for all indexes).

This change also updates the JSON schema:

add new "for query" field
add missing STE vec index field
remove ".1" suffix from m, u, and o index fields
add variants for the encrypted payloads for queries. These variants are used internally by proxy for query-only encryption (for filters in WHERE clauses, for example).

CDThomas · 2024-10-17T01:59:30Z

src/cs_encrypted_v1.schema.json

      "title": "kind",
      "type": "string",
-      "enum": ["pt", "ct", "en"]
+      "enum": ["pt", "ct", "en", "qm", "qo", "qu", "qejp", "qsv"]


When "for query" is set on the plaintext payload, Proxy will encrypt the payload as one of the new variants here:

qm = query match

qo = query ORE

qu = query unique

qejp = query EJSON path

qsv= query STE vec

Proxy will only generate the index needed for the specific query and set the relevant kind/index field in the payload.

For example, if the client sends through a plaintext payload with "for query" set to "match":

{ "v": 1, "k": "pt", "p": "abc", "i": { "t": "users", "c": "email_encrypted" }, "q": "match" }

Proxy would encrypt that as the match query variant:

{ "v": 1, "k": "qm", "m": [1, 2, 3] }

Note that the "m" key in the match query payload lines up with the same key used for the match index that's stored in the column. This allows the EQL SQL funcs to look up the index field using the same key (whether the value comes from the column or from a param):

-- $1 here should be encrypted for a match query SELECT * FROM users WHERE cs_match_v1(email_encrypted) @> cs_match_v1($1)

Are the new kinds required?
The postgres function shouldn't need the kind as long as the actual field is encoded correctly.

They aren't strictly required (we could re-use the existing "ct"/ciphertext variant), but I'd prefer to model this as new variants for a few reasons.

On the Proxy/Rust side:

The source ciphertext ("c" field) in the existing "ct"/ciphertext variant would have to become optional (since queries wouldn't use it)

All of the indexes are optional in the existing "ct"/ciphertext variant, which is bit awkward to work with when encrypting for a query because behaviour-wise only one (required) index field should be set in the payload depending on the operation

Basically, specific variants for each makes it harder for us to construct something incorrectly and means that we can lean more on the compiler to catch problems

On the PG side:

If PG attempts to store a q* variant, something has gone wrong. It's convenient to be able to reject anything that isn't the ct variant

It's nice for debugging to be able to be able to identify with specific variant was chosen

I think having variants for every option adds complexity.
We could have a single "query" variant to disambiguate from ct and pt.

Coupled with having the "q" value be the identifier, we can then validate.

"knd": "qt", "qry": "u", "u": "etc",

if knd == qt
qry is required and qry.value (u) should exist as a field.

I think that would be an OK middle ground if we're concerned about the number of variants.

Conceptually there's a variant for each and I rather make that explicit than have to model it as a single type that requires extra validation logic to handle what the compiler could catch for us.

coderdan · 2024-10-17T02:40:47Z

README.md

+Note that plaintext payloads for query operations should set the `"q"` (for query) property.
+This property tells CipherStash Proxy to only perform encryption necessary for a specific operation.
+Otherwise, Proxy will perform source encryption and encryption for all indexes configured for the given column.
+
+For reference, the EQL payload is defined as a `jsonb` with a specific schema:
+
+```json
+{
+  "v": 1,
+  "k": "pt",
+  "p": "[email protected]",
+  "i": {
+    "t": "users",
+    "c": "email_encrypted"
+  },
+  "q": "match"
+}


Nice. I like that.

coderdan · 2024-10-17T02:45:41Z

README.md

 ```rb
 # Create the EQL payload using helper functions
-payload = eqlPayload("users", "encrpyted_field", "plaintext value")
+payload = EQL.for_match("users", "encrypted_field", "plaintext value")


There are probably more idiomatic ways to do this in Ruby but not something we need to worry about now. Just sharing thoughts while they're fresh.

You could do:

EQL.table("users").column("encrypted_field").match("plaintext value")

If you made EQL.table("users") a class method:

class User def self.eql EQL.table("users") end end

And used method_missing to handle the column name, you could do something like:

User.eql.encrypted_field = "foo" User.eql.encrypted_field > 10 # etc

Just ideas for later.

Very fair. At the moment I'm mostly aiming for examples that get the point across and would be easy to port to other languages. Agreed that it'll make sense to implement a more idiomatic/ergonomic interface for Ruby at some point.

Worth just writting all examples in psuedo code?

Possibly! Or maybe we can go with one of the languages included in this repo (Go or JS).

coderdan

Nice job! I like the approach. Left some musings on Ruby DSL improvements but not needed for the moment.

tobyhede · 2024-10-17T03:04:48Z

I would make the q field be a valid index/encoding type and add the json path as a new accepted encoding.

"q": "m"

CDThomas · 2024-10-17T03:22:29Z

I would make the q field be a valid index/encoding type and add the json path as a new accepted encoding.
"q": "m"

I think this might not work for use cases like websearch_to_tsquery support, @tobyhede. Our websearch_to_tsquery implementation will use the same output type as a regular match index, but will have some extra tokenisation logic that only needs to happen for queries.

tobyhede · 2024-10-17T03:26:36Z

@CDThomas Do you have an example of what that means?

CDThomas · 2024-10-17T04:28:56Z

@CDThomas Do you have an example of what that means?

websearch_to_tsquery is a PG function that allows basic "web search" syntax on top of Postgres full text search. There are some docs here.

For example, the search 'cat or dog' could be used to search for documents that contain the words "cat" or "dog" in them.

In our implementation, the search 'cat or dog' will need to be handled differently to the text 'cat or dog' that's stored in a column (or as a logical operator vs as a word in the text itself). Both will use match indexes/bloom filters, but the search needs some extra parsing/tokenisation logic.

@tobyhede

calvinbrewer · 2024-10-17T10:22:02Z

README.md

 **Example:**

 ```rb
-Users.findAll(&:encrypted_email)


@CDThomas I was trying to show a basic SELECT statement here withour any WHERE clauses and highlighting that responses are simply decrypted. I like what you've done here but maybe for the cs_match_v1 example?

That makes sense @calvinbrewer. I reverted the changes in this section (and already updated the match example).

This change updates the EQL query docs to use the new field `"q"` (for query). This field tells proxy to perform encryption for a specific query operation (instead of performing source encryption and encryption for all indexes). This change also updates the JSON schema: - add new "for query" field - add missing STE vec index field - remove ".1" suffix from m, u, and o fields

This change adds variants for the encrypted payloads for queries to the EQL JSON schema. These variants are used internally by proxy for query-only encryption (for filters in `WHERE` clauses, for example).

Update the match filter in the select diagram to reflect actual usage.

These aren't actually needed since STE term plaintext queries will be encrypted as the same payload as ORE queries and websearch_to_match queries will be the same payload as match queries. Also add missing query variant for EJSON path queries.

CDThomas · 2024-10-18T05:13:13Z

Merging this one since the docs here reflect what we've already started to implement on the Proxy side, but we can adjust the docs if we end up simplifying more.

…tion Update docs and JSON schema with "query-only" encryption changes

CDThomas force-pushed the add-docs-on-query-only-encryption branch 5 times, most recently from 5afdfbd to b67e940 Compare October 17, 2024 01:23

CDThomas changed the title ~~Add docs on query only encryption~~ Update docs and JSON schema with "query-only" encryption changes Oct 17, 2024

CDThomas commented Oct 17, 2024

View reviewed changes

CDThomas marked this pull request as ready for review October 17, 2024 02:01

coderdan reviewed Oct 17, 2024

View reviewed changes

coderdan approved these changes Oct 17, 2024

View reviewed changes

cipherstash deleted a comment from CDThomas Oct 17, 2024

calvinbrewer reviewed Oct 17, 2024

View reviewed changes

CDThomas added 6 commits October 18, 2024 16:01

Add variants for encrypted queries to JSON schema

8c00927

This change adds variants for the encrypted payloads for queries to the EQL JSON schema. These variants are used internally by proxy for query-only encryption (for filters in `WHERE` clauses, for example).

Update match filter in select diagram

07e0e92

Update the match filter in the select diagram to reflect actual usage.

Fix up "k" (kind) property in match query example

18825fe

Revert to previous example in "Reading data" section

6122f41

CDThomas force-pushed the add-docs-on-query-only-encryption branch from bd33ebf to 6122f41 Compare October 18, 2024 05:07

CDThomas merged commit 59a6982 into main Oct 18, 2024

CDThomas deleted the add-docs-on-query-only-encryption branch October 18, 2024 05:13

tobyhede pushed a commit that referenced this pull request Oct 27, 2025

Merge pull request #18 from cipherstash/add-docs-on-query-only-encryp…

da62c01

…tion Update docs and JSON schema with "query-only" encryption changes

Update docs and JSON schema with "query-only" encryption changes #18

Update docs and JSON schema with "query-only" encryption changes #18

Uh oh!

Conversation

CDThomas commented Oct 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CDThomas Oct 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coderdan left a comment

Choose a reason for hiding this comment

Uh oh!

tobyhede commented Oct 17, 2024

Uh oh!

CDThomas commented Oct 17, 2024

Uh oh!

tobyhede commented Oct 17, 2024

Uh oh!

CDThomas commented Oct 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CDThomas commented Oct 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

CDThomas commented Oct 17, 2024 •

edited

Loading

CDThomas Oct 17, 2024 •

edited

Loading

CDThomas commented Oct 17, 2024 •

edited

Loading