OpenAPI Web Search

[jansche]

Summary: Develop an open-source approach to finding Swagger and OpenAPI definitions on the open web, crawling web pages looking for API definitions, validating them, and them consuming and indexing them as part of an ongoing search. Providing a simple way that developers can find APIs that exist by finding documentation, repositories, and other common aspects of running an API.

Skills: Knowledge of the web, and how to crawl web pages, follow URLs, or utilize an existing solution like Common Crawl.

Expected Outcomes: Provide a simple open-source API that abstracts away the complexity of searching the web for specific terms, helping identify APIs in a sea of web pages. Providing a simple interface that will set in motion an asynchronous searching of the web or corpus of web content looking for APIs. Allowing users to initiate a search, but then return regularly to see the results of the search over time, building up results, but then aggregating them for each pulling via simple API.

Possible mentors: @vinitshahdeo + 1-2 additional mentors

Project Repo: https://github.com/postman-open-technologies/openapi-web-search

Size of Project: 175h

Rating: Medium skills level

[jansche]

Could this be merged with #6 and become a 175h project? @kinlane

[BabyElias]

Hey!
So I have been using Postman for quite some time now (always my go-to for visualising API outputs in the best way possible) and I find this idea really exciting.
Quick Question: This requires knowledge of web-scraping, right? How can I go about discussing about this task with potential mentors and seek guidance regarding the same?

[Prajwalprakash3722]

Hello @jansche, This Project actually looks interesting, please correct me If i understood the statement incorrectly;

I am comparing this with API Marketplaces, the proposed solution aims to help developers find APIs that may not be available on existing API marketplaces like RapidAPI by crawling the web and looking for Swagger and OpenAPI definitions, indexing them, and providing access through a simple API interface. This can make it easier for developers to discover and use APIs that are not part of any marketplace and could be relevant for their specific use case.

The idea of crawling the web to find all the Swagger and OpenAPI definitions out there sounds like a Herculean task. Can you tell me more about how can we plan on making it happen? Are we talking about building an army of web crawlers independently or do you have something else in mind?

Nevertheless Exciting stuff!

[jansche]

Hi folks,
we're currently coordinating mentors and will provide more details as well as answer questions beginning of next week (week of February 27). Please bear with us.
Best regards
Jan

[Prajwalprakash3722]

Cool :)