[docs] Basic link verification at PR level

.circleci/config.yml

@@ -240,6 +240,11 @@ jobs:
           command: |
             yarn extract-error-codes
             git diff --exit-code
+      - run:
+          name: '`yarn docs:link-check` changes committed?'
+          command: |
+            yarn docs:link-check
+            git diff --exit-code
   test_types:
     <<: *defaults
     resource_class: 'medium+'

docs/.broken-links-false-detect.txt

@@ -0,0 +1,4 @@
+Broken links found by `yarn docs:link-check` that exist:
+
+- https://mui.com/blog/material-ui-v4-is-out/#premium-themes-store-✨
+- https://mui.com/size-snapshot

docs/data/material/experimental-api/css-theme-variables/customization.md

@@ -180,7 +180,7 @@ Or use `var()` to refer to the CSS variable directly:
 ```
 
 :::info
-💡 If you're using a [custom prefix](/material-ui/experimental-api/css-theme-variables/customization/#changing-variable-prefix), make sure to replace the default `--mui`.
+💡 If you're using a [custom prefix](/material-ui/experimental-api/css-theme-variables/customization/#changing-variable-prefixes), make sure to replace the default `--mui`.
 :::
 
 ### TypeScript

docs/data/material/experimental-api/css-theme-variables/usage.md

@@ -111,7 +111,7 @@ function App() {
   ```
 
   :::info
-  💡 If you have set up a [custom prefix](/material-ui/experimental-api/css-theme-variables/customization/#changing-variable-prefix), make sure to replace the default `--mui`.
+  💡 If you have set up a [custom prefix](/material-ui/experimental-api/css-theme-variables/customization/#changing-variable-prefixes), make sure to replace the default `--mui`.
   :::
 
 ## Server-side rendering

docs/package.json

@@ -15,7 +15,8 @@
     "start": "next start",
     "typescript": "tsc -p tsconfig.json && tsc -p scripts/tsconfig.json",
     "typescript:transpile": "echo 'Use `yarn docs:typescript:formatted'` instead && exit 1",
-    "typescript:transpile:dev": "echo 'Use `yarn docs:typescript'` instead && exit 1"
+    "typescript:transpile:dev": "echo 'Use `yarn docs:typescript'` instead && exit 1",
+    "link-check": "node ./scripts/reportBrokenLinks.js"
   },
   "dependencies": {
     "@babel/core": "^7.19.3",

docs/pages/blog/2020-q3-update.md

@@ -19,7 +19,7 @@ Here are the most significant improvements since June 2020. This was a dense qua
 - 🧪 We have promoted 7 components from the lab to the core: Alert, Autocomplete, Pagination, Rating, Skeleton, SpeedDial, and ToggleButton.
   Thank you for all your feedback on these components.
   While we still plan a couple of breaking changes on them, we are confident that they have reached the same level of quality as the other core components.
-- 👮 We have introduced a new component in the lab, the [TrapFocus](https://mui.com/base/react-trap-focus/). It manages focus for its descendants. This is useful when implementing overlays such as modal dialogs, which should not allow the focus to escape while open:
+- 👮 We have introduced a new component in the lab, the [TrapFocus](https://mui.com/base/react-focus-trap/). It manages focus for its descendants. This is useful when implementing overlays such as modal dialogs, which should not allow the focus to escape while open:
 
   <video style="max-height: 416px; margin-bottom: 24px;" autoplay muted loop playsinline>
     <source src="/static/blog/2020-q3-update/trap-focus.mp4" type="video/mp4" />

docs/pages/blog/2020.md

@@ -48,7 +48,7 @@ We have achieved most of what we could have hoped for.
   - [DatePicker](https://v4.mui.com/components/pickers/)
   - [LoadingButton](https://mui.com/material-ui/react-button/#loading-button)
   - [Timeline](https://v4.mui.com/components/timeline/)
-  - [TrapFocus](https://mui.com/base/react-trap-focus/)
+  - [TrapFocus](https://mui.com/base/react-focus-trap/)
 - We have fixed most of the issues with the [Autocomplete](https://v4.mui.com/components/autocomplete/). We have received an overwhelming interest in the component. It was impressive to see.
 - We have completed the work for [strict mode](https://reactjs.org/docs/strict-mode.html) support.
 - We have increased the adoption of TypeScript in the codebase. We don't plan a dedicated migration but to write new code in TypeScript, as we go.

docs/pages/blog/2021-q3-update.md

@@ -88,7 +88,7 @@ Here are the most significant improvements since early July 2021.
   import Portal from '@mui/base/Portal';
   ```
 
-  - [TrapFocus](/base/react-trap-focus/)
+  - [TrapFocus](/base/react-focus-trap/)
 
   ```jsx
   import TrapFocus from '@mui/base/TrapFocus';

docs/pages/blog/2021.md

@@ -77,7 +77,7 @@ We have achieved most of what we could have hoped for.
 
   - [Masonry](/material-ui/react-masonry/)
   - [Stack](/material-ui/react-stack/)
-  - [Trap Focus](/base/react-trap-focus/)
+  - [Trap Focus](/base/react-focus-trap/)
   - [Unstyled Button](/base/react-button/)
   - [Unstyled Slider](/base/react-slider/)
   - [Unstyled Modal](/base/react-modal/)

docs/pages/blog/mui-core-v5.md

@@ -593,7 +593,7 @@ Having a separate lab package allows us to release breaking changes when necessa
 
 The following components are now available in the lab:
 
-- [LoadingButton](/material-ui/react-button/#loading-buttons). It does what you would expect. It renders the `Button` with a configurable loading/pending state.
+- [LoadingButton](/material-ui/react-button/#loading-button). It does what you would expect. It renders the `Button` with a configurable loading/pending state.
 - [FocusTrap](/base/react-focus-trap/). This component traps the keyboard focus within a DOM node. For example, it's used by the Modal to prevent tabbing out of the component for accessibility reasons.
 - [Masonry](/material-ui/react-masonry/). One great use case for this component is when using the `Grid` component leads to wasted space. It's frequently used in dashboards.
 

docs/scripts/reportBrokenLinks.js

@@ -0,0 +1,212 @@
+/* eslint-disable no-console */
+const path = require('path');
+const fse = require('fs-extra');
+const { createRender } = require('@mui/markdown');
+const { marked } = require('marked');
+
+const UNSUPPORTED_PATHS = ['/api/', '/careers/', '/store/', '/x/'];
+
+const docsSpaceRoot = path.join(__dirname, '../');
+
+const buffer = [];
+
+function write(text) {
+  buffer.push(text);
+}
+
+function save() {
+  const fileContents = [...buffer, ''].join('\n');
+  fse.writeFileSync(path.join(docsSpaceRoot, '.link-check-errors.txt'), fileContents);
+}
+
+// Use renderer to extract all links into a markdown document
+const getPageLinks = (markdown) => {
+  const hrefs = [];
+
+  const renderer = new marked.Renderer();
+  renderer.link = (href) => {
+    if (href[0] === '/') {
+      hrefs.push(href);
+    }
+  };
+  marked(markdown, { renderer });
+  return hrefs;
+};
+
+// List all .js files in a folder
+const getJsFilesInFolder = (folderPath) => {
+  const files = fse.readdirSync(folderPath, { withFileTypes: true });
+  return files.reduce((acc, file) => {
+    if (file.isDirectory()) {
+      const filesInFolder = getJsFilesInFolder(path.join(folderPath, file.name));
+      return [...acc, ...filesInFolder];
+    }
+    if (file.name.endsWith('.js') || file.name.endsWith('.tsx')) {
+      return [...acc, path.join(folderPath, file.name)];
+    }
+    return acc;
+  }, []);
+};
+
+// Returns url assuming it's "./docs/pages/x/..." becomes  "mui.com/x/..."
+const jsFilePathToUrl = (jsFilePath) => {
+  const folder = path.dirname(jsFilePath);
+  const file = path.basename(jsFilePath);
+
+  const root = folder.slice(jsFilePath.indexOf('/pages') + '/pages'.length);
+  const suffix = path.extname(file);
+  let page = `/${file.slice(0, file.length - suffix.length)}`;
+
+  if (page === '/index') {
+    page = '';
+  }
+
+  return `${root}${page}`;
+};
+
+function cleanLink(link) {
+  const startQueryIndex = link.indexOf('?');
+  const endQueryIndex = link.indexOf('#', startQueryIndex);
+
+  if (startQueryIndex === -1) {
+    return link;
+  }
+  if (endQueryIndex === -1) {
+    return link.slice(0, startQueryIndex);
+  }
+  return `${link.slice(0, startQueryIndex)}${link.slice(endQueryIndex)}`;
+}
+
+function getLinksAndAnchors(fileName) {
+  const toc = [];
+  const headingHashes = {};
+  const userLanguage = 'en';
+  const render = createRender({ headingHashes, toc, userLanguage });
+
+  const data = fse.readFileSync(fileName, { encoding: 'utf-8' });
+  render(data);
+
+  const links = getPageLinks(data).map(cleanLink);
+
+  return {
+    hashes: Object.keys(headingHashes),
+    links,
+  };
+}
+
+const getMdFilesImported = (jsPageFile) => {
+  // For each JS file extract the markdown rendered if it exists
+  const fileContent = fse.readFileSync(jsPageFile, 'utf8');
+  /**
+   * Content files can be represented by either:
+   * - 'docsx/data/advanced-components/overview.md?@mui/markdown'; (for mui-x)
+   * - 'docs/data/advanced-components/overview.md?@mui/markdown';
+   * - './index.md?@mui/markdown';
+   */
+  const importPaths = fileContent.match(/'.*\?@mui\/markdown'/g);
+
+  if (importPaths === null) {
+    return [];
+  }
+  return importPaths.map((importPath) => {
+    let cleanImportPath = importPath.slice(1, importPath.length - "?@mui/markdown'".length);
+    if (cleanImportPath.startsWith('.')) {
+      cleanImportPath = path.join(path.dirname(jsPageFile), cleanImportPath);
+    } else if (cleanImportPath.startsWith('docs/')) {
+      cleanImportPath = path.join(
+        jsPageFile.slice(0, jsPageFile.indexOf('docs/')),
+        cleanImportPath,
+      );
+    } else if (cleanImportPath.startsWith('docsx/')) {
+      cleanImportPath = path.join(
+        jsPageFile.slice(0, jsPageFile.indexOf('docs/')),
+        cleanImportPath.replace('docsx', 'docs'),
+      );
+    } else {
+      console.error(`unable to deal with import path: ${cleanImportPath}`);
+    }
+
+    return cleanImportPath;
+  });
+};
+
+const parseDocFolder = (folderPath, availableLinks = {}, usedLinks = {}) => {
+  const jsPageFiles = getJsFilesInFolder(folderPath);
+
+  const mdFiles = jsPageFiles.flatMap((jsPageFile) => {
+    const pageUrl = jsFilePathToUrl(jsPageFile);
+    const importedMds = getMdFilesImported(jsPageFile);
+
+    return importedMds.map((fileName) => ({ fileName, url: pageUrl }));
+  });
+
+  // Mark all the existing page as available
+  jsPageFiles.forEach((jsFilePath) => {
+    const url = jsFilePathToUrl(jsFilePath);
+    availableLinks[url] = true;
+  });
+
+  // For each markdown file, extract links
+  mdFiles.forEach(({ fileName, url }) => {
+    const { hashes, links } = getLinksAndAnchors(fileName);
+
+    links
+      .map((link) => (link[link.length - 1] === '/' ? link.slice(0, link.length - 1) : link))
+      .forEach((link) => {
+        if (usedLinks[link] === undefined) {
+          usedLinks[link] = [fileName];
+        } else {
+          usedLinks[link].push(fileName);
+        }
+      });
+
+    hashes.forEach((hash) => {
+      availableLinks[`${url}/#${hash}`] = true;
+    });
+  });
+};
+
+const getAnchor = (link) => {
+  const splittedPath = link.split('/');
+  return splittedPath[splittedPath.length - 1];
+};
+
+// {[url with hash]: true}
+const availableLinks = {};
+
+// {[url with hash]: list of files using this link}
+const usedLinks = {};
+
+parseDocFolder(path.join(docsSpaceRoot, './pages/'), availableLinks, usedLinks);
+// TODO: Allows to run on documents of /mui/material-ui from /mui/mui-x
+// parseDocFolder(path.join(ROOT, process.env.MUI_X_PATH, "docs/pages/"), availableLinks, usedLinks);
+
+function getPageUrlFromLink(link) {
+  const [rep] = link.split('/#');
+  return rep;
+}
+
+write('Broken links found by `yarn docs:link-check` that exist:\n');
+Object.keys(usedLinks)
+  .filter((link) => link.startsWith('/'))
+  .filter((link) => !availableLinks[link])
+  // unstyled sections are added by scripts (can not be found in markdown)
+  .filter((link) => !link.includes('#unstyled'))
+  .filter((link) => UNSUPPORTED_PATHS.every((unsupportedPath) => !link.includes(unsupportedPath)))
+  .sort()
+  .forEach((linkKey) => {
+    write(`- https://mui.com${linkKey}`);
+    console.log(`https://mui.com${linkKey}`);
+    console.log(`used in`);
+    usedLinks[linkKey].forEach((f) => console.log(`- ${path.relative(docsSpaceRoot, f)}`));
+    console.log('available anchors on the same page:');
+    console.log(
+      Object.keys(availableLinks)
+        .filter((link) => getPageUrlFromLink(link) === getPageUrlFromLink(linkKey))
+        .sort()
+        .map(getAnchor)
+        .join('\n'),
+    );
+    console.log('\n\n');
+  });
+save();

package.json

@@ -25,6 +25,7 @@
     "docs:size-why": "cross-env DOCS_STATS_ENABLED=true yarn docs:build",
     "docs:start": "yarn workspace docs start",
     "docs:i18n": "cross-env BABEL_ENV=development babel-node --extensions \".tsx,.ts,.js\" ./docs/scripts/i18n.js",
+    "docs:link-check": "yarn workspace docs link-check",
     "docs:typescript": "yarn docs:typescript:formatted --watch",
     "docs:typescript:check": "yarn workspace docs typescript",
     "docs:typescript:formatted": "cross-env BABEL_ENV=development babel-node --extensions \".tsx,.ts,.js\" ./docs/scripts/formattedTSDemos",