Skip to content

Commit 57f7660

Browse files
EspressoTrip-v2EspressoTrip-v2
authored
Chore/general client analytics (#394)
* emiiter engine * emitter changes * chnages based on feedback * refactoring and some renaming * controller addition handler * type fix * dev packages * forgot lockfile * reverted dev packages * lockfile dammit * testing * test emitter * test * added emiiter to socket route * fixed binding * removed log * report storage initial * sdk scape * reworked the emitter engine * alterations to types eventemitter * list events chnage * changed document type mongo * changed document type mongo * changed document type mongo * changed document type mongo * chnaged date imps events * chnaged jwt exp * ffs * dev packages are desynced * test disconnect event * disconnect event * removed accidental key in event * changes to update logic sdk * chnageste bump * dev packages issue * dev packages * chnages to mongo storage and package bumps * unset disconnect on connect * change document sdk * deisconnect chnages * connection query * chnaged query for sdk data * removed the disconnect time * chnaged disconnect report to use userdata * fix * chnaged sdk scrape and added sdk delete event * change disconnect mongo update * wip * wip * create new doc only if older than a day * wip * wip * duh * scrape data * wip * timespan logging * fix day query bug * remove facet * dates * wip * wip * unique sdks * wip * sdk * sdk * wip * wip * list connections and some helpers * corrections timeframes * testing * scrapeSdk interval * change to response * change to response * testing date ranges * custom start range list connections * start date * custom date validation * date start validation * testing * test * refactoring * date bug * fixes krep in bugs * added seperate filter for delete * indexes * check document scan result * remove explain * end date to list connections * removed a log * postgres imp WIP * tests * timestamp casting * casting issues * fix incorrect type * iso issues * testing * testing * transaction * added logs * pg-wire frustrations * pg-wire frustrations * pg-wire frustrations * pg-wire frustrations * pg-wire frustrations * pg-wire frustrations * pg-wire frustrations * pg-wire frustrations * change the sql query to test * testing * testing * testing * testing * ts-codecs * ts-codecs * ts-codecs * ts-codecs * ts-codecs * ts-codecs * fixed sdk scrape query * checking delete old data * checking delete old data * deleted row logging * seperated migration for sdk * seperated migration for sdk * added redundancy for events * better log * oops * clean up * fixing a bug * fixing a bug * removed query * removed query * removed query * trying to simplify the update * trying to simplify the update * tests tests * postgres tests WIP * removed command * application name conflict report storage * checking concurrency issues in tests * test chnages to tests * completed postgres tests * mongo tests report * seperated the migrations for sdk, problems testing * okay then * mongo db tests report storage * changed the disconnect report to use connect_at time for long running connections * fixed connected at filter * cclean up some code for pr * version bump * chnages to date ranges on scrapes, simplified * fixed list current connections payload * fix migration indexes on down * model document naming * moved abstarct methods to the top * PR review chnages * PR review changes * emitter PR chnages * PR chnages * PR changes as requested * changed names to the agreed naming convention * chnages table names to match naming conventions * fixed tests * removed index drop migrations * removed unused type * changed date query * added comments * refactored test utils to its own file in postgres and mongo * oops * some fixes * better documentation * removed date range from get connected clients * refactored tests * date formats * removed as utc * DateTime chnages to postgres driver * added forced UTC for connection update * removed as utc * moved migration * renamed migration * if not exists to connecction migration * mock event engine to tests * mock token payload * Merge main * removed errand import merged from main * PR changes * removed duplicated type * jpgwire * jpgwire * removed jpgwire comment * added mongo query and tests * postgress general analytics pagination * package versions * Revert "package versions" This reverts commit 4c88e55. * version bumps * Revert "version bumps" This reverts commit 6be8b4c. * issues * changeset history issue dev packages * changesets added * renaming of functions and some comments * comments * fixed postgress date update utc * removed comment * removed document count from pagination * changed tests * tests for postgres query build and a small bug in mongo cursor * pr changes * remove total from paginated response
1 parent c656695 commit 57f7660

File tree

12 files changed

+831
-17
lines changed

12 files changed

+831
-17
lines changed

‎.changeset/khaki-toys-cheat.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
'@powersync/service-module-postgres-storage': patch
3+
'@powersync/service-module-mongodb-storage': patch
4+
'@powersync/service-core-tests': patch
5+
'@powersync/service-core': patch
6+
'@powersync/service-types': patch
7+
---
8+
9+
General client connections analytics added
10+

‎.github/workflows/packages_release.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,9 @@ jobs:
2222
steps:
2323
- name: Checkout Repo
2424
uses: actions/checkout@v5
25+
with:
26+
# check out full history. development packages need this for changesets
27+
fetch-depth: 0
2528
- name: Enable Corepack
2629
run: corepack enable
2730
- name: Setup Node.js

‎modules/module-mongodb-storage/src/storage/MongoReportStorage.ts‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ import { storage } from '@powersync/service-core';
22
import { event_types } from '@powersync/service-types';
33
import { PowerSyncMongo } from './implementation/db.js';
44
import { logger } from '@powersync/lib-services-framework';
5+
import { createPaginatedConnectionQuery } from '../utils/util.js';
56

67
export class MongoReportStorage implements storage.ReportStorage {
78
public readonly db: PowerSyncMongo;
@@ -43,6 +44,27 @@ export class MongoReportStorage implements storage.ReportStorage {
4344
return result[0];
4445
}
4546

47+
async getGeneralClientConnectionAnalytics(
48+
data: event_types.ClientConnectionAnalyticsRequest
49+
): Promise<event_types.PaginatedResponse<event_types.ClientConnection>> {
50+
const { cursor, date_range } = data;
51+
const limit = data?.limit || 100;
52+
53+
const connected_at = date_range ? { connected_at: { $lte: date_range.end, $gte: date_range.start } } : undefined;
54+
const user_id = data.user_id != null ? { user_id: data.user_id } : undefined;
55+
const client_id = data.client_id != null ? { client_id: data.client_id } : undefined;
56+
return (await createPaginatedConnectionQuery(
57+
{
58+
...client_id,
59+
...user_id,
60+
...connected_at
61+
},
62+
this.db.connection_report_events,
63+
limit,
64+
cursor
65+
)) as event_types.PaginatedResponse<event_types.ClientConnection>;
66+
}
67+
4668
async reportClientConnection(data: event_types.ClientConnectionBucketData): Promise<void> {
4769
const updateFilter = this.updateDocFilter(data.user_id, data.client_id!);
4870
await this.db.connection_report_events.findOneAndUpdate(

‎modules/module-mongodb-storage/src/utils/util.ts‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,3 +114,43 @@ export function setSessionSnapshotTime(session: mongo.ClientSession, time: bson.
114114
throw new ServiceAssertionError(`Session snapshotTime is already set`);
115115
}
116116
}
117+
118+
export const createPaginatedConnectionQuery = async <T extends mongo.Document>(
119+
query: mongo.Filter<T>,
120+
collection: mongo.Collection<T>,
121+
limit: number,
122+
cursor?: string
123+
) => {
124+
const createQuery = (cursor?: string) => {
125+
if (!cursor) {
126+
return query;
127+
}
128+
const connected_at = query.connected_at
129+
? { $lt: new Date(cursor), $gte: query.connected_at.$gte }
130+
: { $lt: new Date(cursor) };
131+
return {
132+
...query,
133+
connected_at
134+
} as mongo.Filter<T>;
135+
};
136+
137+
const findCursor = collection.find(createQuery(cursor), {
138+
sort: {
139+
/** We are sorting by connected at date descending to match cursor Postgres implementation */
140+
connected_at: -1
141+
}
142+
});
143+
144+
const items = await findCursor.limit(limit).toArray();
145+
const count = items.length;
146+
/** The returned total has been defaulted to 0 due to the overhead using documentCount from the mogo driver.
147+
* cursor.count has been deprecated.
148+
* */
149+
return {
150+
items,
151+
count,
152+
/** Setting the cursor to the connected at date of the last item in the list */
153+
cursor: count === limit ? items[items.length - 1].connected_at.toISOString() : undefined,
154+
more: !(count !== limit)
155+
};
156+
};

‎modules/module-mongodb-storage/test/src/__snapshots__/connection-report-storage.test.ts.snap‎

Lines changed: 238 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,13 @@
33
exports[`Connection reporting storage > Should create a connection report if its after a day 1`] = `
44
[
55
{
6-
"client_id": "client_week",
6+
"client_id": "client_one",
77
"sdk": "powersync-js/1.24.5",
88
"user_agent": "powersync-js/1.21.0 powersync-web Firefox/141 linux",
99
"user_id": "user_week",
1010
},
1111
{
12-
"client_id": "client_week",
12+
"client_id": "client_one",
1313
"sdk": "powersync-js/1.24.5",
1414
"user_agent": "powersync-js/1.21.0 powersync-web Firefox/141 linux",
1515
"user_id": "user_week",
@@ -213,3 +213,239 @@ exports[`Report storage tests > Should show currently connected users 1`] = `
213213
"users": 2,
214214
}
215215
`;
216+
217+
exports[`Report storage tests > Should show paginated response of all connections of specified client_id 1`] = `
218+
{
219+
"count": 1,
220+
"cursor": undefined,
221+
"items": [
222+
{
223+
"client_id": "client_two",
224+
"sdk": "powersync-js/1.21.1",
225+
"user_agent": "powersync-js/1.21.0 powersync-web Chromium/138 linux",
226+
"user_id": "user_two",
227+
},
228+
],
229+
"more": false,
230+
}
231+
`;
232+
233+
exports[`Report storage tests > Should show paginated response of all connections with a limit 1`] = `
234+
{
235+
"count": 4,
236+
"cursor": "<removed-for-snapshot>",
237+
"items": [
238+
{
239+
"client_id": "client_one",
240+
"sdk": "powersync-dart/1.6.4",
241+
"user_agent": "powersync-dart/1.6.4 Dart (flutter-web) Chrome/128 android",
242+
"user_id": "user_one",
243+
},
244+
{
245+
"client_id": "client_four",
246+
"sdk": "powersync-js/1.21.4",
247+
"user_agent": "powersync-js/1.21.0 powersync-web Firefox/141 linux",
248+
"user_id": "user_four",
249+
},
250+
{
251+
"client_id": "",
252+
"sdk": "unknown",
253+
"user_agent": "Dart (flutter-web) Chrome/128 android",
254+
"user_id": "user_one",
255+
},
256+
{
257+
"client_id": "client_two",
258+
"sdk": "powersync-js/1.21.1",
259+
"user_agent": "powersync-js/1.21.0 powersync-web Chromium/138 linux",
260+
"user_id": "user_two",
261+
},
262+
],
263+
"more": true,
264+
}
265+
`;
266+
267+
exports[`Report storage tests > Should show paginated response of all connections with a limit 2`] = `
268+
{
269+
"count": 4,
270+
"cursor": undefined,
271+
"items": [
272+
{
273+
"client_id": "client_three",
274+
"sdk": "powersync-js/1.21.2",
275+
"user_agent": "powersync-js/1.21.0 powersync-web Firefox/141 linux",
276+
"user_id": "user_three",
277+
},
278+
{
279+
"client_id": "client_one",
280+
"sdk": "powersync-js/1.24.5",
281+
"user_agent": "powersync-js/1.21.0 powersync-web Firefox/141 linux",
282+
"user_id": "user_week",
283+
},
284+
{
285+
"client_id": "client_month",
286+
"sdk": "powersync-js/1.23.6",
287+
"user_agent": "powersync-js/1.23.0 powersync-web Firefox/141 linux",
288+
"user_id": "user_month",
289+
},
290+
{
291+
"client_id": "client_expired",
292+
"sdk": "powersync-js/1.23.7",
293+
"user_agent": "powersync-js/1.23.0 powersync-web Firefox/141 linux",
294+
"user_id": "user_expired",
295+
},
296+
],
297+
"more": false,
298+
}
299+
`;
300+
301+
exports[`Report storage tests > Should show paginated response of all connections with a limit with date range 1`] = `
302+
{
303+
"count": 4,
304+
"cursor": "<removed-for-snapshot>",
305+
"items": [
306+
{
307+
"client_id": "",
308+
"sdk": "unknown",
309+
"user_agent": "Dart (flutter-web) Chrome/128 android",
310+
"user_id": "user_one",
311+
},
312+
{
313+
"client_id": "client_two",
314+
"sdk": "powersync-js/1.21.1",
315+
"user_agent": "powersync-js/1.21.0 powersync-web Chromium/138 linux",
316+
"user_id": "user_two",
317+
},
318+
{
319+
"client_id": "client_three",
320+
"sdk": "powersync-js/1.21.2",
321+
"user_agent": "powersync-js/1.21.0 powersync-web Firefox/141 linux",
322+
"user_id": "user_three",
323+
},
324+
{
325+
"client_id": "client_one",
326+
"sdk": "powersync-js/1.24.5",
327+
"user_agent": "powersync-js/1.21.0 powersync-web Firefox/141 linux",
328+
"user_id": "user_week",
329+
},
330+
],
331+
"more": true,
332+
}
333+
`;
334+
335+
exports[`Report storage tests > Should show paginated response of all connections with a limit with date range 2`] = `
336+
{
337+
"count": 2,
338+
"cursor": undefined,
339+
"items": [
340+
{
341+
"client_id": "",
342+
"sdk": "unknown",
343+
"user_agent": "Dart (flutter-web) Chrome/128 android",
344+
"user_id": "user_one",
345+
},
346+
{
347+
"client_id": "client_two",
348+
"sdk": "powersync-js/1.21.1",
349+
"user_agent": "powersync-js/1.21.0 powersync-web Chromium/138 linux",
350+
"user_id": "user_two",
351+
},
352+
{
353+
"client_id": "client_three",
354+
"sdk": "powersync-js/1.21.2",
355+
"user_agent": "powersync-js/1.21.0 powersync-web Firefox/141 linux",
356+
"user_id": "user_three",
357+
},
358+
{
359+
"client_id": "client_one",
360+
"sdk": "powersync-js/1.24.5",
361+
"user_agent": "powersync-js/1.21.0 powersync-web Firefox/141 linux",
362+
"user_id": "user_week",
363+
},
364+
],
365+
"more": false,
366+
}
367+
`;
368+
369+
exports[`Report storage tests > Should show paginated response of connections of specified user_id 1`] = `
370+
{
371+
"count": 2,
372+
"cursor": undefined,
373+
"items": [
374+
{
375+
"client_id": "client_one",
376+
"sdk": "powersync-dart/1.6.4",
377+
"user_agent": "powersync-dart/1.6.4 Dart (flutter-web) Chrome/128 android",
378+
"user_id": "user_one",
379+
},
380+
{
381+
"client_id": "",
382+
"sdk": "unknown",
383+
"user_agent": "Dart (flutter-web) Chrome/128 android",
384+
"user_id": "user_one",
385+
},
386+
],
387+
"more": false,
388+
}
389+
`;
390+
391+
exports[`Report storage tests > Should show paginated response of connections over a date range 1`] = `
392+
{
393+
"count": 6,
394+
"cursor": undefined,
395+
"items": [
396+
{
397+
"client_id": "client_one",
398+
"sdk": "powersync-dart/1.6.4",
399+
"user_agent": "powersync-dart/1.6.4 Dart (flutter-web) Chrome/128 android",
400+
"user_id": "user_one",
401+
},
402+
{
403+
"client_id": "client_four",
404+
"sdk": "powersync-js/1.21.4",
405+
"user_agent": "powersync-js/1.21.0 powersync-web Firefox/141 linux",
406+
"user_id": "user_four",
407+
},
408+
{
409+
"client_id": "client_two",
410+
"sdk": "powersync-js/1.21.1",
411+
"user_agent": "powersync-js/1.21.0 powersync-web Chromium/138 linux",
412+
"user_id": "user_two",
413+
},
414+
{
415+
"client_id": "",
416+
"sdk": "unknown",
417+
"user_agent": "Dart (flutter-web) Chrome/128 android",
418+
"user_id": "user_one",
419+
},
420+
{
421+
"client_id": "client_three",
422+
"sdk": "powersync-js/1.21.2",
423+
"user_agent": "powersync-js/1.21.0 powersync-web Firefox/141 linux",
424+
"user_id": "user_three",
425+
},
426+
{
427+
"client_id": "client_one",
428+
"sdk": "powersync-js/1.24.5",
429+
"user_agent": "powersync-js/1.21.0 powersync-web Firefox/141 linux",
430+
"user_id": "user_week",
431+
},
432+
],
433+
"more": false,
434+
}
435+
`;
436+
437+
exports[`Report storage tests > Should show paginated response of connections over a date range of specified client_id and user_id 1`] = `
438+
{
439+
"count": 1,
440+
"cursor": undefined,
441+
"items": [
442+
{
443+
"client_id": "client_one",
444+
"sdk": "powersync-dart/1.6.4",
445+
"user_agent": "powersync-dart/1.6.4 Dart (flutter-web) Chrome/128 android",
446+
"user_id": "user_one",
447+
},
448+
],
449+
"more": false,
450+
}
451+
`;

‎modules/module-postgres-storage/package.json‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,16 +17,16 @@
1717
},
1818
"exports": {
1919
".": {
20+
"types": "./dist/@types/index.d.ts",
2021
"import": "./dist/index.js",
2122
"require": "./dist/index.js",
22-
"default": "./dist/index.js",
23-
"types": "./dist/@types/index.d.ts"
23+
"default": "./dist/index.js"
2424
},
2525
"./types": {
26+
"types": "./dist/@types/index.d.ts",
2627
"import": "./dist/types/types.js",
2728
"require": "./dist/types/types.js",
28-
"default": "./dist/types/types.js",
29-
"types": "./dist/@types/index.d.ts"
29+
"default": "./dist/types/types.js"
3030
}
3131
},
3232
"dependencies": {

0 commit comments

Comments
 (0)