Implement migrate sub-command (#314)

* Add migrate subcommand

* Change migrated file extensions to `.md.tmpl`

* Finish `migrate` subcommand implementation

* Remove `-help` output check

* Add changelog entry

* Update changelog entry

Co-authored-by: Brian Flad <[email protected]>

* fix changelog entry

* Remove `--old-website-source-dir` flag and support migrating `docs/` subdirectory

* Support converting files with any file extension to `.md.tmpl`

* Refactor `Migrate()` method

* Skip `layout` front matter

* Fix linting errors

* Add comment to generated templates explaining functionality

* Remove `website/` directory at the end of migration

* Fix `ineffassign` lint error

* Update README.md

Co-authored-by: Austin Valle <[email protected]>

* Resolve comments from code review

---------

Co-authored-by: Brian Flad <[email protected]>
Co-authored-by: Austin Valle <[email protected]>

Author: 3 people

.changes/unreleased/FEATURES-20231220-141244.yaml

@@ -0,0 +1,7 @@
+kind: FEATURES
+body: 'migrate: Added new `migrate` subcommand that migrates existing provider docs using the
+  rendered website source directories (`website/docs/` or `/docs/`) to a `terraform-plugin-docs`-supported
+  templates directory.'
+time: 2023-12-20T14:12:44.820323-05:00
+custom:
+  Issue: "314"

README.md

@@ -5,7 +5,7 @@ The primary way users will interact with this is the `tfplugindocs` CLI tool to
 
 ## `tfplugindocs`
 
-The `tfplugindocs` CLI has two main commands, `validate` and `generate` (`generate` is the default).
+The `tfplugindocs` CLI has three main commands, `migrate`, `validate` and `generate` (`generate` is the default).
 This tool will let you generate documentation for your provider from live example `.tf` files and markdown templates.
 It will also export schema information from the provider (using `terraform providers schema -json`),
 and sync the schema with the reference documents.
@@ -28,6 +28,7 @@ Usage: tfplugindocs [--version] [--help] <command> [<args>]
 Available commands are:
                 the generate command is run by default
     generate    generates a plugin website from code, templates, and examples
+    migrate     migrates website files from either the legacy rendered website directory (`website/docs/r`) or the docs rendered website directory (`docs/resources`) to the tfplugindocs supported structure (`templates/`).
     validate    validates a plugin website for the current directory
 
 ```
@@ -59,6 +60,18 @@ $ tfplugindocs validate --help
 Usage: tfplugindocs validate [<args>]
 ```
 
+`migrate` command:
+
+```shell
+$ tfplugindocs migrate --help
+
+Usage: tfplugindocs migrate [<args>]
+
+    --examples-dir <ARG>             examples directory based on provider-dir                                                                                           (default: "examples")
+    --provider-dir <ARG>             relative or absolute path to the root provider code directory when running the command outside the root provider code directory
+    --templates-dir <ARG>            new website templates directory based on provider-dir; files will be migrated to this directory                                    (default: "templates")
+```
+
 ### How it Works
 
 When you run `tfplugindocs`, by default from the root directory of a provider codebase, the tool takes the following actions:
@@ -100,6 +113,23 @@ Otherwise, the provider developer can set an arbitrary description like this:
     // ...
 ```
 
+#### Migrate subcommand
+
+The `migrate` subcommand can be used to migrate website files from either the legacy rendered website directory (`website/docs/r`) or the docs 
+rendered website directory (`docs/resources`) to the `tfplugindocs` supported structure (`templates/`). Markdown files in the rendered website 
+directory will be converted to `tfplugindocs` templates. The legacy `website/` directory will be removed after migration to avoid Terraform Registry 
+ingress issues.
+
+The `migrate` subcommand takes the following actions:
+1. Determines the rendered website directory based on the `--provider-dir` argument
+2. Copies the contents of the rendered website directory to the `--templates-dir` folder (will create this folder if it doesn't exist)
+3. (if the rendered website is using legacy format) Renames `docs/d/` and `docs/r/` subdirectories to `data-sources/` and `resources/` respectively
+4. Change file suffixes for Markdown files to `.md.tmpl` to create website templates
+5. Extracts code blocks from website docs to create individual example files in `--examples-dir` (will create this folder if it doesn't exist)
+6. Replace extracted example code in website templates with `codefile`/`tffile` template functions referencing the example files.
+7. Copies non-template files to `--templates-dir` folder
+8. Removes the `website/` directory
+
 ### Conventional Paths
 
 The generation of missing documentation is based on a number of assumptions / conventional paths.
@@ -130,6 +160,37 @@ For examples:
 | `examples/resources/<resource name>/resource.tf`          | Resource example config         |
 | `examples/resources/<resource name>/import.sh`            | Resource example import command |
 
+#### Migration
+
+The `migrate` subcommand assumes the following conventional paths for the rendered website directory:
+
+Legacy website directory structure:
+
+| Path                                              | Description                 |
+|---------------------------------------------------|-----------------------------|
+| `website/`                                        | Root of website docs        |
+| `website/docs/guides`                             | Root of guides subdirectory |
+| `website/docs/index.html.markdown`                | Docs index page             |
+| `website/docs/d/<data source name>.html.markdown` | Data source page            |
+| `website/docs/r/<resource name>.html.markdown`    | Resource page               |
+
+Docs website directory structure:
+
+| Path                                                 | Description                 |
+|------------------------------------------------------|-----------------------------|
+| `docs/`                                              | Root of website docs        |
+| `docs/guides`                                        | Root of guides subdirectory |
+| `docs/index.html.markdown`                           | Docs index page             |
+| `docs/data-sources/<data source name>.html.markdown` | Data source page            |
+| `docs/resources/<resource name>.html.markdown`       | Resource page               |
+
+Files named `index` (before the first `.`) in the website docs root directory and files in the `website/docs/d/`, `website/docs/r/`, `docs/data-sources/`, 
+and `docs/resources/` subdirectories will be converted to `tfplugindocs` templates. 
+
+The `website/docs/guides/` and `docs/guides/` subdirectories will be copied as-is to the `--templates-dir` folder. 
+
+All other files in the conventional paths will be ignored.
+
 ### Templates
 
 The templates are implemented with Go [`text/template`](https://golang.org/pkg/text/template/)
@@ -179,7 +240,7 @@ using the following data fields and functions:
 | `tffile`        | A special case of the `codefile` function, designed for Terraform files (i.e. `.tf`).             |
 | `trimspace`     | Equivalent to [`strings.TrimSpace`](https://pkg.go.dev/strings#TrimSpace).                        |
 | `upper`         | Equivalent to [`strings.ToUpper`](https://pkg.go.dev/strings#ToUpper).                            |
-
+ 
 ## Disclaimer
 
 This is still under development: while it's being used for production-ready providers, you might still find bugs

cmd/tfplugindocs/main_test.go

@@ -46,3 +46,11 @@ func Test_SchemaJson_GenerateAcceptanceTests(t *testing.T) {
 		Dir: "testdata/scripts/schema-json/generate",
 	})
 }
+
+func Test_SchemaJson_MigrateAcceptanceTests(t *testing.T) {
+	t.Parallel()
+
+	testscript.Run(t, testscript.Params{
+		Dir: "testdata/scripts/schema-json/migrate",
+	})
+}