feat: segment telemetry #329

MQ37 · 2025-11-07T02:41:39Z

This PR implements the Segment telemetry.

Implemented based on https://www.notion.so/apify/Apify-MCP-server-analytics-2a1f39950a2280c4a9a5c8154dfc0f1a.

related to https://github.com/apify/ai-team/issues/56 (I would mark completed manually after we verify in prod).

This PR does not implement the anonymousId (Device ID) yet for unauth requests (coming soon) - I would implement this with the PR where we are going to implement the unauth requests for docs.

Other changes:

added TELEMETRY_ENABLED env var to doppler. Localhost=false, for staging and prod it is enabled.
added TELEMETRY_ENV with options dev and prod

Bonus features:

implemented reading Apify token from the ~/.apify/auth.json file that is created by Apify CLI so if you are a CLI user you can use the server without specifying the env var.

…v and prod settings

…empty string)

jirispilka

I was included as a reviewer before, somehow my comments are mixed as review comments. Hopefully it make sense

src/actor/server.ts

src/telemetry.ts

src/actor/server.ts

src/mcp/server.ts

src/const.ts

src/types.ts

tests/helpers.ts

src/telemetry.ts

jirispilka · 2025-11-24T12:55:49Z

I wouldn't personally refer to this initiative as telemetry but analytics instead. Or at least this is what we do in terms of segment usage across all our codebase.

@katzino oh, I missed this comment. Yes, it make sense.

There are two aspects:

external people (we use telemetry in apify-cli). I think we should keep the same naming in Apify MCP
internally, we can refactor it to analytics (I will do it after your review, not to make the review more difficult)

@MQ37 @jirimoravcik what do you think?

katzino

Pre-approving, just check out that one comment I have for userId

katzino · 2025-11-24T13:37:05Z

src/telemetry.ts

+ * @param properties - Event properties for the tool call
+ */
+export function trackToolCall(
+    userId: string,


I have seen you using crypto.randomUUID() as a fallback but that should be attributes to anonymousId instead. In theory it shouldn't make much of a difference in mixpanel, but Segment is treating userId and anonymousId differently. I would maybe double check with @mr-rajce about it.

Good catch, I was not aware of that. I've changed it to

client.track({ ...(userId ? { userId } : { anonymousId: crypto.randomUUID() }), event: SEGMENT_EVENTS.TOOL_CALL, properties, });

… userId is available, use it; otherwise use anonymousId

src/utils/userid-cache.ts

jirimoravcik · 2025-11-24T15:28:41Z

tests/unit/mcp.server.tool-call-counter.test.ts

+            // Default store should be available after construction
+            const store = server.options.telemetry!.toolCallCountStore!;


What does should mean? If we're certain, we can type it in a way that we don't need !. If we're not sure, we should do a check before accessing it.

I've changed this part a bit more. See this comment

In nutshell, there is now private toolCallCountStore: ToolCallCounterStore | undefined; which will be defined only if telemetry is enabled.

But I had to change the tests to

const toolCallCount = server.getToolCallCountStore(); expect(toolCallCount).toBeDefined(); if (!toolCallCount) { // TypeScript needs this for type narrowing expect.fail('toolCallCount should be defined'); }

jirimoravcik · 2025-11-24T16:23:59Z

src/types.ts

+         * Enable or disable telemetry tracking for tool calls.
+         * Defaults to true when not set.
+         */
+        enabled?: boolean;


Not sure if this should be optional, the parent object already is.

Yeah, it might be better to be explicit here, so I marked it as required

* Must be explicitly set when telemetry object is provided. * When telemetry object is omitted entirely, defaults to true (via env var or default). */

src/telemetry.ts

jirimoravcik · 2025-11-25T08:24:49Z

src/telemetry.ts

+/**
+ * Tracks a tool call event to Segment.
+ *
+ * @param userId - Apify user ID (null if not available)


null if not available

but the type is just string? Shouldn't it be string | null?

Yeah, my mistake, I've done some changes and forgot to update it, it is string | null now (to handle anonymousId correctly)

jirimoravcik · 2025-11-25T08:26:11Z

src/telemetry.ts

+            analyticsClient = new Analytics({
+                writeKey,
+                flushAt: 50,
+                flushInterval: 5000,


Can we extract this into a constant that would mention the units? I'd assume it's 5_000 ms, but not sure from this part of code.

Yeah, right, changed to:

flushAt: SEGMENT_FLUSH_AT_EVENTS, flushInterval: SEGMENT_FLUSH_INTERVAL_MS,

jirimoravcik · 2025-11-25T08:28:20Z

src/mcp/server.ts

    private currentLogLevel = 'info';
    public readonly options: ActorsMcpServerOptions;
+    // In-memory storage for tool call counters (used when toolCallCounterStore is not provided)
+    private sessionToolCallCounters = new Map<string, number>();


I think this can grow infinitely, we should implement some mechanism that would keep a limited number of items in the cache. Maybe a LRUCache would work?

Good catch. In practice, this shouldn’t happen. When running locally, you usually create a single session, since stdio can’t handle multiple concurrent sessions.

For a remote server, you should provide your own getAndIncrementToolCallCounter with a Redis backend.
BUT if that function isn’t provided, the counter would grow indefinitely

So fixed using LRUCache.

jirimoravcik · 2025-11-25T08:28:45Z

src/mcp/server.ts

+     * @returns Promise resolving to the new counter value (after increment)
+     */
+    private async getAndIncrementToolCallCounter(sessionId: string): Promise<number> {
+        return await this.options.telemetry!.toolCallCountStore!.getAndIncrement(sessionId);


Can we type it so that we get rid of !?

Well, this makes me think I should handle it in a better way (including telemetry envs). Previously, I was modifying input.options, but instead of doing that, I’ve now introduced vars

private telemetryEnabled: boolean | null = null; private telemetryEnv: TelemetryEnv = DEFAULT_TELEMETRY_ENV; private toolCallCountStore: ToolCallCounterStore | undefined;

IMO this makes code cleaner, as I don't have to write this.options.telemetry.env etc.

Co-authored-by: Jiří Moravčík <[email protected]>

…ounters` with LRUCache, add telemetry as class instance variables

MQ37

My main question is why the tests/integration/actor.server-sse.test.ts and tests/integration/actor.server-streamable.test.ts were removed? Then only minor comments, otherwise LGTM 👍

MQ37 · 2025-11-25T11:52:15Z

tests/unit/telemetry.test.ts

note: I don't think we need tests like this. But let's keep that since it is already there.

I've removed the Actor code in other PR. I did not realize that you were relying on these test to for HTTP and SSE.

MQ37 · 2025-11-25T11:56:18Z

src/mcp/server.ts

+    // Telemetry configuration (resolved from options and env vars in setupTelemetry)
+    private telemetryEnabled: boolean | null = null;
+    private telemetryEnv: TelemetryEnv = DEFAULT_TELEMETRY_ENV;
+    private toolCallCountStore: ToolCallCounterStore | undefined;


question: Do we actually need this tool call counter logic? Why not just use the timestamp from the analytics that is included automatically and if we are just interested in the total session tool calls count I guess some mixpanel query could easily handle that.

Just for the record: I check again with Kuba R. and it is not actually required.
Earlier it was proposed by Kuba to track user journey, so I'm going to remove it.

Thanks for pointing it out again!

feat: implement segment telemetry

5c11a41

github-actions bot assigned MQ37 Nov 7, 2025

github-actions bot added the t-ai Issues owned by the AI team. label Nov 7, 2025

MQ37 added the beta Create beta prereleases label Nov 7, 2025

MQ37 changed the title ~~feat: implement segment telemetry~~ feat: segment telemetry Nov 7, 2025

MQ37 requested a review from jirispilka November 7, 2025 02:55

add reason input schema field for telemetry

3e58be6

github-actions bot added the tested Temporary label used only programatically for some analytics. label Nov 7, 2025

MQ37 added 3 commits November 6, 2025 21:42

fix: reorder telemetry section in the table of contents

5eb7f70

fix: update default userId to an empty string in trackToolCall function

61480bc

fix typo

1b1215d

MQ37 marked this pull request as ready for review November 7, 2025 06:05

MQ37 and others added 16 commits November 7, 2025 12:24

Merge remote-tracking branch 'origin/master' into segment-telemetry

6cd4af3

Merge branch 'master' into segment-telemetry

0f1fbda

fix: add new variable telemetry-enabled instead of telemetry. Hide de…

2f5a4a3

…v and prod settings

fix: add constants instead of prod and dev

92e8024

fix: add capabilities to tracking

4016352

fix: remove unused ENVIROMENT variable

82eefe5

fix: add env variable to enable/disable telemetry

d779506

fix: add env variable to enable/disable telemetry

0e6cdfd

fix: update unit-tests

72b9ed6

fix: update telemetry properties

53bead4

fix: add status and exec time to telemetry

f46bff9

fix: add tool call number

f3fb01c

fix: remove reason from telemetry

3f0d09d

fix: change tool status to match Actor run status

a5fd735

fix: change event name

a551686

fix: tests

0bf506b

jirispilka added beta Create beta prereleases and removed beta Create beta prereleases labels Nov 20, 2025

jirispilka added 15 commits November 21, 2025 22:04

fix: in the ActorsMcpServer use nested telemetry object

4cc26d9

fix: in the ActorsMcpServer use nested telemetry object

ab7f161

fix: log errors in telemetry

9da99b0

fix: move ActorsMcpServerOptions and ToolCallCounterStore to types.ts

d6e7c6d

fix: Improve getAndIncrementToolCallCounter and add test

da55972

fix: Improve naming in setupTelemetry function

5ba5751

fix: comment

a651486

fix: refactor prepareTelemetryData

11e57f0

fix: simplify userId cache

571c087

fix: make telemetryEnv uppercase -> PROD, DEV

421f1a3

fix: handle telemetry-enabled param in better way (also for null and …

8c8721f

…empty string)

fix: simplify - get package version

209e41f

fix: rename app_name to app

a3c5ece

fix: save mcp_client_capabilities

5165cc0

fix: set tool-status failed

dd5291e

jirispilka reviewed Nov 24, 2025

View reviewed changes

jirispilka requested review from jirimoravcik and katzino November 24, 2025 12:59

katzino approved these changes Nov 24, 2025

View reviewed changes

fix: Segment requires either userId OR anonymousId, but not both When…

e3a4bc0

… userId is available, use it; otherwise use anonymousId

jirimoravcik approved these changes Nov 25, 2025

View reviewed changes

jirispilka and others added 3 commits November 25, 2025 09:58

Apply suggestions from code review

17c1860

Co-authored-by: Jiří Moravčík <[email protected]>

fix: add units to analyticsClient creation. Replace `sessionToolCallC…

8c05a44

…ounters` with LRUCache, add telemetry as class instance variables

fix: types, makes telemetry.enabled required

7bfe235

MQ37 commented Nov 25, 2025

View reviewed changes

jirispilka added 2 commits November 25, 2025 15:56

fix: disable telemetry for tests

0448298

fix: remove tool_count_number, it is not required

f703e90

jirispilka merged commit fa8f421 into master Nov 26, 2025
5 checks passed

jirispilka deleted the segment-telemetry branch November 26, 2025 11:04

		// Default store should be available after construction
		const store = server.options.telemetry!.toolCallCountStore!;

feat: segment telemetry #329

feat: segment telemetry #329

Uh oh!

Conversation

MQ37 commented Nov 7, 2025 • edited by jirispilka Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jirispilka left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jirispilka commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

katzino left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MQ37 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

MQ37 commented Nov 7, 2025 •

edited by jirispilka

Loading

jirispilka left a comment •

edited

Loading

jirispilka commented Nov 24, 2025 •

edited

Loading