[DataCap Application] OpenStreetMap

[babores]

Data Owner Name

OpenStreetMap

Data Owner Country/Region

United States

Data Owner Industry

Information, Media & Telecommunications

Website

https://www.openstreetmap.org/#map=4/38.01/-95.84

Social Media Handle

https://www.openstreetmap.org/#map=4/38.01/-95.84

Social Media Type

Slack

What is your role related to the dataset

Dataset Owner

Total amount of DataCap being requested

1.2PiB

Expected size of single dataset (one copy)

120TiB

Number of replicas to store

10

Weekly allocation of DataCap requested

200TiB

On-chain address for first allocation

f1tgxrkf5oz5sc5fxo2mgzefnaj74i5fihe7degji

Data Type of Application

Public, Open Dataset (Research/Non-Profit)

Custom multisig

• Use Custom Multisig

Identifier

No response

Share a brief history of your project and organization

OpenStreetMap (OSM) is a global mapping project that aims to create a free, editable map. It is contributed by volunteers, and users can collect and update geographic information through GPS, aerial imagery, and other sources. Here are some important information about OpenStreetMap:

Features
Openness: OSM data is completely open and can be used, modified, and shared by anyone. This makes it a very flexible tool for a variety of applications.
Community-driven: OSM relies on contributions from volunteers around the world. These contributors can be anyone, from professional geographers to ordinary users.
Rich data: OSM contains a variety of geographic information such as roads, buildings, rivers, mountains, public transportation routes, etc., covering urban and rural areas.
Multiple uses: Developers can use OSM data to build map applications, perform geographic analysis, or use it for navigation and positioning services.
Multiple formats: Data can be downloaded in multiple formats, such as XML, PBF files, etc., to facilitate different usage scenarios.

Is this project associated with other projects/ecosystem stakeholders?

No

If answered yes, what are the other projects/ecosystem stakeholders

Describe the data being stored onto Filecoin

OpenStreetMap (OSM) is a global mapping project that aims to create a free, editable map. It is contributed by volunteers, and users can collect and update geographic information through GPS, aerial imagery, and other sources. Here are some important information about OpenStreetMap:

Features
Openness: OSM data is completely open and can be used, modified, and shared by anyone. This makes it a very flexible tool for a variety of applications.
Community-driven: OSM relies on contributions from volunteers around the world. These contributors can be anyone, from professional geographers to ordinary users.
Rich data: OSM contains a variety of geographic information such as roads, buildings, rivers, mountains, public transportation routes, etc., covering urban and rural areas.
Multiple uses: Developers can use OSM data to build map applications, perform geographic analysis, or use it for navigation and positioning services.
Multiple formats: Data can be downloaded in multiple formats, such as XML, PBF files, etc., to facilitate different usage scenarios.

Where was the data currently stored in this dataset sourced from

AWS Cloud

If you answered "Other" in the previous question, enter the details here

If you are a data preparer. What is your location (Country/Region)

HK

If you are a data preparer, how will the data be prepared? Please include tooling used and technical details?

We use the command line tools to download, we have the AWS CLI tools installed, download directly from AWS in the EU Central 1 and US West 2 regions. This is faster than downloading normally.
Command line:
aws s3 ls --no-sign-request s3://osm-planet-eu-central-1/
aws s3 ls --no-sign-request s3://osm-planet-us-west-2/

If you are not preparing the data, who will prepare the data? (Provide name and business)

Has this dataset been stored on the Filecoin network before? If so, please explain and make the case why you would like to store this dataset again to the network. Provide details on preparation and/or SP distribution.

no

Please share a sample of the data

https://www.openstreetmap.org/#map=4/38.01/-95.84

Confirm that this is a public dataset that can be retrieved by anyone on the Network

• [ yes] I confirm

If you chose not to confirm, what was the reason

What is the expected retrieval frequency for this data

Yearly

For how long do you plan to keep this dataset stored on Filecoin

1-2 year

In which geographies do you plan on making storage deals

Greater China, Asia other than Greater China, Europe, South America

How will you be distributing your data to storage providers

HTTP or FTP server

How did you find your storage providers

Slack、Partners

If you answered "Others" in the previous question, what is the tool or platform you used

Please list the provider IDs and location of the storage providers you will be working with.

f03519438  HK
f03312989   HK
f02826126  HK
f02827880  HK
f02827288  ShangHai
f02827045   ShangHai
f09693  SiChuan
f03358608 HK
f03233304 United States
f02924003 United States

How do you plan to make deals to your storage providers

Boost client

If you answered "Others/custom tool" in the previous question, enter the details here

Can you confirm that you will follow the Fil+ guideline

Yes

[datacap-bot]

Application is waiting for allocator review

[datacap-bot]

Similarity Report

No similar applications found for the issue

[dampud]

@babores
Thank you for your application. After reviewing it, we will need a few additional pieces of information:

• Will you be able to provide detailed information on the dataset preparation process in a way that community members can reproduce it? The field describing the data preparation was omitted.
• The dataset description indicates that the data is open and available to everyone; however, in the application you did not specify that the data will be available for download by other users. Could you please clarify? This field was omitted.
• Are you able to confirm in any way the dataset size (1.5 PiB) and the need for as many as 10 replicas?

[babores]

https://download.geofabrik.de/index-v1.json
Here is the complete master data index file. We will first download and save the corresponding source data in this file. The size of the entire data set is estimated based on the official project introduction. Currently, about 2000G of OSM XML files are generated every week. According to the 52 weeks in a year, 2000*52=104000G/1024=101.54T can be calculated. The current data set stores all historical data from 2013 to date, so the total data set is about 1.2P. Considering that the same-category related data will be packaged in a car file for easy recording and retrieval during the source data packaging process, each car file will be kept between 20-32G, so the size of 1.5P for a single node is reasonable.

[babores]

https://planet.openstreetmap.org/planet/

[babores]

Planet.osm (also known as The planet file) is the OpenStreetMap data in one file: all the nodes, ways and relations that make up our map. A new version is released every week. It's a big file (on 2025-03-01, the plain OSM XML variant takes over 2032.4 GB when uncompressed from the 147.5 GB bzip2-compressed or 79.4 GB PBF-compressed downloaded data file).

https://wiki.openstreetmap.org/wiki/Planet.osm#Downloading

[babores]

We have improved the data list.Updated the cooperating SP. And reduced the total data application from 15PiB to 12PiB.
We sincerely hope to get your support, thank you.

[dampud]

@babores
The claim of needing 1.2–1.5 PiB to store the OpenStreetMap dataset is highly overestimated. OSM's entire data archive, including weekly planet snapshots and historical dumps, fits comfortably within 120 TB (not petabytes). Weekly snapshots are around 80 GB (PBF), and storing a decade’s worth of data would total 10s of TB, not 1.2 PiB. Your estimate assumes unrealistic data sizes (e.g., uncompressed XML) and duplicates data across regions. A more reasonable storage requirement for OSM data is in the low hundreds of TB, not petabytes.
Full aws bucket osm-planet-eu-central-1 contains 24228257 objects and total Size: 116.5 TiB
If no other source of data can be provided, DC that could be requested in view of this information is maximum 120 TiB(one data set) x 10 = 1.2 PiB in total.