EcmaScript lexer is inefficient

[kdy1]

• It currently pushes one char at a time. It should chunk to amortize allocations.

Current code is similar to

let mut buf = String::new();
loop {
   // xxx
   buf.push(c);
}

Obviously, this almost always allocate and reallocates quite frequently.

---

• It allocate needlessly while tokenizing numeric literals.

Noticeable parts are

let s = format!("{}", val);
// if it contains '8' or '9', it's decimal.
if s.contains('8') || s.contains('9') {}

// It's Legacy octal, and we should reinterpret value.
let val = u64::from_str_radix(&format!("{}", val), 8)
    .expect("Does this can really happen?");
let val = format!("{}", val)
    .parse()
     .expect("failed to parse numeric value as f64");

val = format!("{}.{}", val, minority).parse();

[nicoburns]

Would it makes sense for lexing to generate &strs which point into the original buffer and avoid allocating altogether (except once per file)?

[kdy1]

Holding reference to a original source makes all ast nodes have lifetime parameter.
Performance gain from it would not worth work to do it.
Also, it would make compile time longer.

Now we can utilize uncons_while()

swc/ecmascript/parser/src/lexer/input.rs

Lines 103 to 140 in eb2a2a7

	impl<'a> Input for SourceFileInput<'a> {
	fn peek(&mut self) -> Option<(BytePos, char)> {
	self.clone().nth(0)
	}
	fn peek_ahead(&mut self) -> Option<(BytePos, char)> {
	self.clone().nth(1)
	}
	fn uncons_while<F>(&mut self, f: F) -> Option<&str>
	where
	F: FnMut(char) -> bool,
	{
	//TODO?
	None
	}
	fn record_new_line(&self, pos: BytePos) {
	// FIXME: Handling of lb in js and rust is different.
	// self.fm.next_line(pos)
	}
	fn start_pos(&self) -> BytePos {
	self.fm.start_pos
	}
	}
	pub trait Input: Iterator<Item = (BytePos, char)> {
	fn peek(&mut self) -> Option<(BytePos, char)>;
	fn peek_ahead(&mut self) -> Option<(BytePos, char)>;
	fn record_new_line(&self, _pos: BytePos) {}
	fn start_pos(&self) -> BytePos;
	///Takes items from stream, testing each one with predicate. returns the
	/// range of items which passed predicate.
	fn uncons_while<F>(&mut self, f: F) -> Option<&str>
	where
	F: FnMut(char) -> bool;
	}

SourceFileInput stores source code as String so we can slice and copy it instead of extending String byte by byte.

I expect this task to be easy.

[swc-bot]

This closed issue has been automatically locked because it had no new activity for a month. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.