LineWriter is broken

[ijackson]

Summary

std::io::LineWriter presents the inner writer with partial lines.

The behaviour is bizarre. It is also contrary to a careful reading of the (rather poor) documentation, and makes it unsuitable for most intended use cases.

Steps

https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=ad71a92706e77761dcefef3658e6b12c

use std::io::{self, LineWriter, Write as _};

struct LineCallWriter(LineWriter<EprintlnDebugWriter>);
impl LineCallWriter {
    pub fn new() -> Self {
        LineCallWriter(LineWriter::new(EprintlnDebugWriter))
    }
}
struct EprintlnDebugWriter;
impl io::Write for LineCallWriter {
    fn write(&mut self, i: &[u8]) -> io::Result<usize> {
        eprintln!("I {:?}", String::from_utf8_lossy(i));
        self.0.write(i)
    }
    fn flush(&mut self) -> io::Result<()> {
        eprintln!("I FLUSH");
        Ok(())
    }
}
impl Drop for LineCallWriter {
    fn drop(&mut self) {
        eprintln!("W DROP");
    }
}
impl io::Write for EprintlnDebugWriter {
    fn write(&mut self, i: &[u8]) -> io::Result<usize> {
        eprintln!("W {:?}", String::from_utf8_lossy(i));
        Ok(i.len())
    }
    fn flush(&mut self) -> io::Result<()> {
        eprintln!("W FLUSH");
        Ok(())
    }
}

fn main() -> Result<(), io::Error> {
    let mut lcw = LineCallWriter::new();
    write!(lcw, "one line at a go, ok\n")?;
    write!(lcw, "in parts, ")?;
    write!(lcw, "bad.\n")?;
    write!(lcw, "multiple lines\nall in one go\n")?;
    write!(lcw, "multiple lines\nwith last one\nsplit, ")?;
    write!(lcw, "wow!\n")?;
    Ok(())
}

Actual output

I "one line at a go, ok\n"
W "one line at a go, ok\n"
I "in parts, "
I "bad.\n"
W "in parts, "
W "bad.\n"
I "multiple lines\nall in one go\n"
W "multiple lines\nall in one go\n"
I "multiple lines\nwith last one\nsplit, "
W "multiple lines\nwith last one\n"
I "wow!\n"
W "split, "
W "wow!\n"
W DROP

Expected output

I "one line at a go, ok\n"
W "one line at a go, ok\n"
I "in parts, "
I "bad.\n"
W "in parts, bad.\n"
I "multiple lines\nall in one go\n"
W "multiple lines\nall in one go\n"
I "multiple lines\nwith last one\nsplit, "
W "multiple lines\nwith last one\n"
I "wow!\n"
W "split, wow!\n"
W DROP

Discussion

The documentation is rather vague, but it does seem to promise that the inner writer will only see whole lines (except at the end). "This batched write" implies a single write to the inner writer.

IMO:

• Writing multiple lines in one write to the inner writer is perhaps OK. An inner writer that wants to split them can do so easily enough. On the other hand, some use cases would find it more helpful to receive one line at a time. Anyway, the behaviour should be documented.
• Each write should end with a newline, except during drop and maybe flush. The documentation should promise this explicitly.
• The behaviour on flush should be documented.

Meta

I saw this with this compiler, but the latest stable on the playground does just the same.

rustc --version --verbose:

rustc 1.85.0 (4d91de4e4 2025-02-17)
binary: rustc
commit-hash: 4d91de4e48198da2e33413efdcd9cd2cc0c46688
commit-date: 2025-02-17
host: x86_64-unknown-linux-gnu
release: 1.85.0
LLVM version: 19.1.7

[hanna-kruppe]

The documentation could be improved, yeah. But I don’t think the behavior you expect is the right one. As the docs say (emphasis mine):

Like BufWriter, a LineWriter’s buffer will also be flushed when the LineWriter goes out of scope or when its internal buffer is full.

Unlimited buffering has undesirable effects when there’s very long lines in the output. In particular, consider that io::stdout() is line buffered when connected to a terminal (and there’s still no way to opt out of that).

So I don’t think the purpose of LineWriter is to ensure the underlying writer sees only full lines. Like BufWriter, it’s a performance optimization for coalescing many small writes. As you note, flushing prevents this from being true in any case. So I struggle to imagine how a writer could correctly make use of the “expected” behavior. If writing full lines matters for correctness (e.g., to exploit write system calls being atomic), then I suggest buffering manually in a String or Vec<u8>, or using vectored writes where available.

[ChrisDenton]

In 1.47 the behaviour was different:

I "one line at a go, ok\n"
W "one line at a go, ok\n"
W FLUSH
I "in parts, "
I "bad.\n"
W "in parts, bad.\n"
W FLUSH
I "multiple lines\nall in one go\n"
W "multiple lines\nall in one go\n"
W FLUSH
I "multiple lines\nwith last one\nsplit, "
W "multiple lines\nwith last one\n"
W FLUSH
I "wow!\n"
W "split, wow!\n"
W FLUSH
W DROP

This changed in 1.48 to what it is today.

[hanna-kruppe]

Probably changed in #72808 FWIW

[bjorn3]

Yeah, the PR description for that even acknowledges this behavior:

Does our input data contain a newline?

• If no:
  • simply use the regular BufWriter::write to write it; this will append it to the buffer and/or flush it as necessary based on how full the buffer is and how much input data there is.

[the8472]

LineWriter already contains a BufWriter, so small writes without a newline should end up in its buffer, not immediately visible to EprintlnDebugWriter.

We even have a test for this

rust/library/std/src/io/buffered/tests.rs

Lines 408 to 423 in 04ff05c

	#[test]
	fn test_line_buffer() {
	let mut writer = LineWriter::new(Vec::new());
	writer.write(&[0]).unwrap();
	assert_eq!(*writer.get_ref(), []);
	writer.write(&[1]).unwrap();
	assert_eq!(*writer.get_ref(), []);
	writer.flush().unwrap();
	assert_eq!(*writer.get_ref(), [0, 1]);
	writer.write(&[0, b'\n', 1, b'\n', 2]).unwrap();
	assert_eq!(*writer.get_ref(), [0, 1, 0, b'\n', 1, b'\n']);
	writer.flush().unwrap();
	assert_eq!(*writer.get_ref(), [0, 1, 0, b'\n', 1, b'\n', 2]);
	writer.write(&[3, b'\n']).unwrap();
	assert_eq!(*writer.get_ref(), [0, 1, 0, b'\n', 1, b'\n', 2, 3, b'\n']);
	}

Maybe it only works for writers that support vectored writes?

[hanna-kruppe]

That test checks what ends up in the underlying writer, but not by how many write calls it gets delivered. @ijackson's example exercises this part of the LineWriter logic:

rust/library/std/src/io/buffered/linewritershim.rs

Lines 83 to 88 in 04ff05c

	// Flush existing content to prepare for our write. We have to do this
	// before attempting to write `buf` in order to maintain consistency;
	// if we add `buf` to the buffer then try to flush it all at once,
	// we're obligated to return Ok(), which would mean suppressing any
	// errors that occur during flush.
	self.buffer.flush_buf()?;

i.e., a partial line shorter than the buffer size does get buffered until the line is completed, but when the line is completed that's delivered to the underlying writer in two steps (flush buffer; then write all or parts of the new data).

[the8472]

Ah right, I even had a discussion about that in the PR, but that was resolved without actually fixing the issue 😞.