StreamingFast Firehose node instrumentation

[AngeloCa]

Rationale

Why should this feature exist?

This feature will improve the data availability for EVM chains and L2s using go-ethereum.

Firehose is a new technology developed by StreamingFast working with The Graph Foundation. The product provides previously unseen capabilities and speeds for indexing blockchain data using a files-based and streaming-first approach.

Firehose is responsible for extracting data from blockchain nodes in a highly efficient manner. Firehose also consumes, processes, and streams blockchain data to consumers of nodes running Firehose-enabled, instrumented blockchain client software.

What are the use-cases?

From a blockchain core development team’s perspective:

• Immediate support for Substreams, and its high-speed parallel processing engine.
• Immediate integration into The Graph, through Substreams-based Subgraphs.
• Immediate integration into the latest graph node which can feed directly from the Firehose.
• Immediate support for dozens of sinks (think loading to PostgreSQL, MongoDB, CDC replication, Kafka, Flink, writing to buckets - for ingestion by BigQuery, AWS Redshift, Clickhouse, etc.).
• Activates a community of developers that already know how to index blockchain data, and are going to be happy to discover your blockchain’s data model and activity.

From users’ perspective:

• The capability to quickly index and sift through your favorite blockchain’s history.
• Stream its output with low latency.
• Hook it to any data systems you have, trading bots, cross-chain bridges, databases.
• All of it in an extremely reliable way.
• Benefit from the anti-fragility brought by The Graph network.
• Offer an army of indexers to service your network.

Implementation

Do you have ideas regarding the implementation of this feature?

Read StreamingFast Docs to integrate new chains.

[karalabe]

I've read through most of the docs site, but honestly I don't see much information on how this thing works of why it is useful or what integration really takes or any performance numbers. The docs mentions they have an Ethereum client already instrumented, so perhaps can we look at the code and some numbers as to what it generates, how fast, how much does it grow, etc?

[fjl]

We've looked through the firehose instrumentation at https://github.com/streamingfast/go-ethereum and it seems like this could be integrated using a 'native tracer'.

[paymog]

I've read through most of the docs site, but honestly I don't see much information on how this thing works of why it is useful or what integration really takes or any performance numbers. The docs mentions they have an Ethereum client already instrumented, so perhaps can we look at the code and some numbers as to what it generates, how fast, how much does it grow, etc?

Here's the firehose instrumented fork of geth. As far as I know, all the instrumented version does is emit messages to stdout which get consumed by another process. The instrumentation should have very little impact on performance and storage (and should be easy to hide behind a flag). Though, I'm not an expert in how the instrumentation works and @maoueh would be a much better resource.

[abourget]

I'm wondering if there is any progress on this one @fjl ? The uncertainty alone is holding us back from doing work in some of the forks. It would be helpful to have visibility at least of the intent of going forward. I propose again that we give a first dent into tweaking the Tracer as minimally as possible to achieve something that would work across the ecosystem from the geth codebase.

[abourget]

@s1na any progress that we could report here? Folks from Arbitrum asking if we'd take a dent into a FirehoseTracer for Nitro. If you've started something, perhaps we could start looking at how easily it would port over?

[fjl]

We are working on a API proposal. Sorry for the delay.

[holiman]

Fixed by #29189 (I think / hope!)

[maoueh]

I think we can say yes. We are going to explore some Go plugin system to see if it would be possible to register "external" entities to act as tracer and what you be the outcome.

We have seen https://github.com/hashicorp/go-plugin used in Cosmos land so we might make some tests to see how it would look like.

But anyway, all this is amazing, thanks to the Geth team and @s1na for the everything, amazing!