Beating the fastest lexer generator in Rust
The article shows the author's doubts about the efficiency and generality of lexer generators. The author decides to implement their own lexer to compare it with the performance of existing lexer generators. The hand-rolled implementation uses state transitions and terminal states to identify tokens. The author also mentions the use of a perfect hash function for keyword matching. The comparison between the author's implementation and the lexer generator "logos" shows that "logos" performs significantly better, especially on x86_64 systems. The author speculates that this performance difference is due to differences in speculative execution architecture. The article provides insights into the performance considerations of lexers and the impact of different implementations.