cmd/fix: automate migrations for simple deprecations

[bcmills]

(I've mentioned this in passing on #32014 and #27248, but I don't see an actual proposal filed yet — so here it is!)

Over time, the author of a package may notice certain patterns in the usage of their API that they want to make simpler or clearer for users. When that happens, they may suggest that users update their own code to use the new API.

In many cases, the process of updating is nearly trivial, so it would be nice if some tool in the standard distribution could apply those trivial updates mechanically. The go fix subcommand was used for similar updates to the language itself, so perhaps it could be adapted to work for at least the simple cases.

The very simplest sort of update rewrites a call to some function or use of some variable or constant to instead call some other function or use some other variable, constant, or expression.

To draw some examples from the standard library:

• The comment for archive/tar.TypeRegA says to use TypeReg instead.
• The comments for archive/zip.FileHeader.{Compressed,Uncompressed}Size say to use the corresponding *64 fields instead.
• The comments for image.ZR and image.ZP say to use zero struct literals instead.
• The comment for go/importer.For says to use ForCompiler, and is implemented as a trivial call to ForCompiler with an additional argument.
• The comment for the io.SEEK_* constants says to use corresponding io.Seek* constants instead.
• The comments for syscall.StringByte{Slice,Ptr} say to use the corresponding Byte*FromString functions, and are implemented as short wrappers around those functions.

Outside the standard library, this pattern can also occur when a package makes a breaking change: if the v3 API can express all of the same capabilities as v2, then v2 can be implemented as a wrapper around v3 — and references to the v2 identifiers can be replaced with direct references to the underlying v3 identifiers.

---

See the current proposal details in #32816 (comment) below.

Previous draft (prior to #32816 (comment)):

I propose that we establish a consistent convention to mark these kinds of mechanical replacements, and improve cmd/fix to be able to apply them.

I'm not particularly tied to any given convention, but in the spirit of having something concrete to discuss, I propose two tokens, //go:fix-to and //go:forward, as follows:

• //go:fix-to EXPR on a constant or variable indicates that go fix should replace occurrences of that constant or variable with the expression EXPR. Free variables in the expression are interpreted as package names, and go fix may rewrite them to avoid collisions with non-packages.
• //go:fix-to .EXPR on struct field indicates that go fix should replace references to that field with the given selector expression (which may be a field access OR a method call) on the same value. Free variables are again interpreted as package names. If the expression is a method call, it replaces only reads of the field.
• //go:forward on an individual constant or variable indicates that go fix should replace the constant or variable with the expression from which it is initialized. If a constant is declared with a type but initialized from an untyped expression, the replacement is wrapped in a conversion to that type.
• //go:forward on a const or var block indicates that every identifier within the block should be replaced by the expression from which it is initialized.
• //go:forward on a method or function indicates that go fix should inline its body at the call site, renaming any local variables and labels as needed to avoid collisions. The function or method must have at most one return statement and no defer statements.
• //go:forward on a type alias indicates that go fix should replace references to the alias name with the (immediate) type it aliases.

All of these rewrites must be acyclic: the result of a (transitively applied) rewrite must not refer to the thing being rewritten.

Note that there is a prototype example-based refactoring tool for Go in golang.org/x/tools/cmd/eg that supported a broader set of refactoring rules. It might be useful as a starting point.

---

For the above examples, the resulting declarations might look like:

package image

// Deprecated: Use a literal image.Point{} instead.
var ZP = Point{} //go:forward

// Deprecated: Use a literal image.Rectangle{} instead.
var ZR = Rectangle{} //go:forward

package importer

// Deprecated: Use ForCompiler, which populates a FileSet
// with the positions of objects created by the importer.
//
//go:forward
func For(compiler string, lookup Lookup) types.Importer {
	return ForCompiler(token.NewFileSet(), compiler, lookup)
}

package io

// Deprecated: Use io.SeekStart, io.SeekCurrent, and io.SeekEnd.
const (
	SEEK_SET int = 0 //go:fix-to io.SeekStart
	SEEK_CUR int = 1 //go:fix-to io.SeekCur
	SEEK_END int = 2 //go:fix-to io.SeekEnd
)

package syscall

// Deprecated: Use ByteSliceFromString instead.
//
//go:forward
func StringByteSlice(s string) []byte {
	a, err := ByteSliceFromString(s)
	if err != nil {
		panic("string with NUL passed to syscall.ByteSliceFromString")
	}
	return a
}

[…]

---

This proposal does not address the archive/tar use-case: callers producing archives might be able to transparently migrate from TypeRegA to TypeReg, but it isn't obvious that callers consuming archives can safely do so.

Nor does this proposal does not address the archive/zip.FileHeader use-case: the deprecation there indicates a loss of precision, and the caller should probably be prompted to make related fixes in the surrounding code.

Both of those cases might be addressable with a deeper example-based refactoring tool, but (I claim) are too subtle to handle directly in go fix.

---

CC @marwan-at-work @thepudds @jayconrod @ianthehat @myitcv @cep21

[mvdan]

How would this deal with Go versions? For example, importer.ForCompiler was added in 1.12, so go fix shouldn't make the change if a module still supports Go 1.11.

Perhaps it can grab the oldest supported Go version from the optional line in go.mod, defaulting to a version extremely conservative (such as 1.0) if it's missing.

[bcmills]

@mvdan, good question. I'd say that it should not attempt to do anything special with Go versions. go fix is an explicit operation, and folks should look at the diffs (and run tests) after applying it to ensure that they're happy with the results.

For stuff in the standard library in particular, perhaps that implies that we shouldn't add //go:fix or //go:forward comments until two releases after the target of the fix, since that's our recommended support window.

[mvdan]

until two releases after the target of the fix, since that's our recommended support window.

I personally think this is a losing battle. Certain libraries support older Go versions, for their own reasons. You can say "just don't use go fix", but that seems unnecessary if go.mod can give us that information.

An alternative is to use "two Go releases ago" as the fallback version, instead of 1.0. Then a module can specify an older or newer version as needed.

[thepudds]

The "Automatic API Updates" section of the initial vgo blog series sketched out an alternative. It might be interesting to contrast this proposal with that earlier sketch.

One difference between that earlier sketch and the proposal here is that the earlier sketch I think had fewer comment variations, but I think relied on the v1 code being implemented as a shim around v2 (and perhaps it could in theory also support the reverse if v2 is implemented as a wrapper around v1).

Example snippet from that older sketch:

For example, suppose v1 of an API has functions EnableFoo and DisableFoo and v2 replaces 
the pair with a single SetFoo(enabled bool). After v2 is released, v1 can be implemented 
as a wrapper around v2:

package p // v1

import v2 "p/v2"

func EnableFoo() {
	//go:fix
	v2.SetFoo(true)
}

func DisableFoo() {
	//go:fix
	v2.SetFoo(false)
}

The fact that actual Go code is used and the earlier sketch might "use symbolic execution" might mean the earlier sketch is more expressive than this newer proposal. (edit: not sure if this is true; need to digest the new proposal more).

In contrast, this newer proposal also addresses deprecation use cases (without a need for a major version change).

[bcmills]

@mvdan, I suppose we could use the annotations in api/go*.txt to figure out when various identifiers were added, but note that the go directive is more of a maximum than a minimum. (For example, you can write a go 1.13 module to get access to the features added in 1.13, but use build tags to notch out those source files when building with 1.12 and earlier.)

[bcmills]

@thepudds, note that @rsc's sketch there is substantially more powerful than what I'm proposing here.

In particular, I'm not assuming that go fix can do any sort of control-flow analysis.

One of the examples there is:

func SetFoo(enabled bool) {
	if enabled {
		//go:fix
		v2.EnableFoo()
	} else {
		//go:fix
		v2.DisableFoo()
	}
}

but under this proposal that would be expressed as:

//go:forward
func SetFoo(enabled bool) {
	if enabled {
		v2.EnableFoo()
	} else {
		v2.DisableFoo()
	}
}

Of course, once go fix does the inlining, you can always apply other (more powerful!) analyzers to prune out the inlined paths that are provably unreachable.

[mvdan]

note that the go directive is more of a maximum than a minimum

Ah, noted.

[cep21]

A few questions:

• Does go:forward imply deprecated? Is the intent to place both comments in my code?
• Can you give an example of go:fix-to .EXPR
• The go:forward inline case seems the most magical of the bunch. Probably worth extra consideration.

Trying to get a scope of what you're proposing.

• Fixing net.Dialer.Cancel is way out of scope (?)
• Fixing net.Dialer.DualStack ... does that work somehow with .Expr?
• Can net.Dialer.Dial be replaced with net.Dialer.DialContext(context.TODO(), ...) ?
• There's probably a whole class of current and future context API migrations

[bcmills]

Does go:forward imply deprecated? Is the intent to place both comments in my code?

I would generally expect so, yes: go:forward implies that there is a simpler, more direct, or more robust alternative.

However, I think we still want both comments in general. //go:forward tells go fix that the fix should be automated, and // Deprecated: tells humans why the alternative is preferred.

Can you give an example of go:fix-to .EXPR

Sure: you could envision using it for (*net.URL).RawPath, since “In general, code should call EscapedPath instead of reading u.RawPath directly.”

package url

type URL struct {
	[…]
	// encoded path hint
	RawPath string //go:fix-to .EscapedPath()
	[…]
}

The go:forward inline case seems the most magical of the bunch. Probably worth extra consideration.

Agreed. It's the most magical, but also arguably the most useful. 🙂

I think it will require particular care around how we map := and return, since we have to make the return statement mesh with the caller code. Perhaps as a first past we should restrict it to functions whose single return statement is at the end of the function.

Fixing net.Dialer.Cancel is way out of scope (?)

Correct. I could envision some ways to use example-based refactoring to convert uses of the Cancel field to use DialContext, but they would require a much more sophisticated algorithm to match the call site against the refactoring pattern.

Fixing net.Dialer.DualStack ... does that work somehow with .Expr?

Sadly, no: .EXPR works for reads, but the thing we want to match there is the write operation.

I could imagine a //go:fix-write-to extension of this proposal that would handle it, though. Let _ stand for the field being fixed, and let //go:fix-write-to .EXPR be the rewrite rule for writes to a field. Then we can combine a //go:fix-write-to with a //go:forward to produce the intended refactoring:

package net

type Dialer {
	[…]
	DualStack bool //go:fix-write-to .setDualStack(_)
	[…]
}

//go:forward
func (d *Dialer) setDualStack(dualStack bool) {
	if !dualStack {
		d.FallbackDelay = -1
	}
}

Can net.Dialer.Dial be replaced with net.Dialer.DialContext(context.TODO(), ...) ?

Yes:

package net

//go:forward
func (d *Dialer) Dial(network, address string) (Conn, error) {
	return d.DialContext(context.TODO(), network, address)
}

There's probably a whole class of current and future context API migrations

Yep. Note that it's easy to introduce context.TODO() using this proposal, but converting those context.TODO() calls to use the actual context in the current scope is beyond its capability.

[cep21]

Sure: you could envision using it for (*net.URL).RawPath, since “In general, code should call EscapedPath instead of reading u.RawPath directly.”

Is go-fix intended for changes that aren't strictly backwards compatible?

[bcmills]

I think that's an open question. This proposal leaves that decision up to package authors: nothing stops them from applying a //go:fix-to or //go:forward that isn't strictly backward compatible, although I would expect its use for incompatible changes to be rare in practice.