lex and yacc tutorial

Lex and Yacc Tutorial: Build Parsers and Compilers Step by Step

By Andrew Walker
AD

Building a compiler, interpreter, or any tool that processes structured text is a rite of passage for many programmers. The classic Unix utilities lex and yacc have powered this journey for decades, forming the backbone of countless parsers, config readers, and domain-specific languages. This lex and yacc tutorial walks you through both tools from the ground up, showing how they split the monumental task of language processing into two manageable halves: lexical analysis and syntactic analysis.

If terms like "tokenizer," "grammar rules," or "shift-reduce conflict" sound intimidating, don't worry. By the end of this guide, you will understand not just what these tools do, but how to write your own lexer and parser pair, compile them, and run a working calculator program. The concepts here transfer directly to modern tools like Flex, Bison, ANTLR, and PLY, so the time invested pays dividends far beyond these two legacy programs.

Understanding the Lex and Yacc Pipeline

At its core, a compiler or interpreter translates source text into something meaningful, whether that means machine code, an abstract syntax tree, or a calculated result. Doing this in one monolithic function is a nightmare of nested conditionals. Lex and yacc solve the problem by enforcing a clean separation of concerns: lex handles character-level pattern matching, and yacc handles the structural relationships between the resulting pieces.

AD
Lexi and Onya
Lexi and Onya
pricklyjokester canon 🐇🔪
pricklyjokester canon 🐇🔪
big lex baddies usa
big lex baddies usa

The workflow is elegantly circular. You write a lex specification that defines tokens, then feed those token definitions into a yacc grammar that defines how tokens combine into valid statements. Yacc generates a parser that calls the lexer whenever it needs the next token, creating a cooperative loop that processes input stream by stream, rule by rule.

The Role of Lexical Analysis

Lexical analysis, or scanning, is the process of chopping a raw character stream into meaningful chunks called tokens. The lex tool takes a specification file containing regular expressions and associated actions, then generates a C function called yylex() that performs this scanning. Each time the parser requests a token, yylex() consumes characters until it matches a pattern and returns a token identifier.

Hi
Hi
💜🦇ʅҽxყƈαƚ ‧͙🕷️~ʂԋαԃσɯҽԃSσυʅ.ԃʅʅ 魂ཀ+ ϝαɳαɾƚ !! 💙🪽
💜🦇ʅҽxყƈαƚ ‧͙🕷️~ʂԋαԃσɯҽԃSσυʅ.ԃʅʅ 魂ཀ+ ϝαɳαɾƚ !! 💙🪽
larx and jstn vc gang
larx and jstn vc gang

For example, when scanning the input total = 42 + price, a lexer might emit the token sequence IDENTIFIER, ASSIGN, NUMBER, PLUS, IDENTIFIER. Notice that whitespace simply vanishes, as most lexers silently discard it. The lexer does not care whether the sequence forms a valid statement; it only cares whether each individual piece matches a known pattern like an identifier or a number.

The Role of Syntax Analysis

Syntax analysis, or parsing, takes the token stream and determines whether it conforms to the grammar of the language. Yacc accepts a specification containing grammar productions in a notation resembling Backus-Naur Form, then generates a C function called yyparse(). This function repeatedly asks the lexer for tokens and shifts or reduces them according to the generated parsing tables.

AD

Where the lexer sees a flat list of tokens, the parser sees hierarchy and structure. It knows that a multiplication binds tighter than an addition, that parentheses group expressions, and that every statement must end with a semicolon. When the token stream violates these rules, the parser reports a syntax error rather than blindly producing garbage output.

Lexi and Kg
Lexi and Kg
big lex baddies midwest
big lex baddies midwest
Lexshua?!
Lexshua?!
big lex baddies usa
big lex baddies usa
Big Lex | Getting Irritated With Kold Killa’s Commentary.
Big Lex | Getting Irritated With Kold Killa’s Commentary.
divaa
divaa

Writing Your First Lex Specification

A lex file divides into three sections separated by double percent signs. The definitions section holds C code, macro definitions, and start conditions. The rules section holds pattern-action pairs. The user subroutines section holds additional C functions that get copied verbatim into the generated scanner. Understanding this layout is the key to writing maintainable lexers.

Patterns are regular expressions, and actions are C statements enclosed in braces. When the generated scanner finds a match, it executes the corresponding action. A common beginner mistake is forgetting that the longest match wins, not the first match listed, so ordering rules only breaks ties among equally long matches.

AD

A Minimal Lex Example

Consider a tiny lexer that recognizes integers and operators. The definitions section includes the standard C headers and declares a variable to hold numeric values. The rules section maps digit sequences to an action that converts the text to an integer using atoi(), stores it in a global, and returns a NUMBER token to the parser.

JCC (Elise + Bones + Vick) + Penguins of Madagasgar
JCC (Elise + Bones + Vick) + Penguins of Madagasgar
hi aubree hi why isn’t she saying hi back
hi aubree hi why isn’t she saying hi back
literally the best
literally the best

The final section defines a main() function that simply calls yylex() in a loop, printing each recognized token until end of input. Compiling this with the lex command and a C compiler produces a standalone tokenizer in seconds, demonstrating how little code is needed to get started.

Handling Tokens and Return Values

Each action in a lexer typically returns an integer token code. These codes are either single characters, like the plus sign returned as its ASCII value, or symbolic constants defined in the yacc grammar file. To keep both files in sync, programmers usually create a shared header or let yacc generate one with the -d flag.

AD

Sometimes a lexer needs to pass more than just a token type. It can store additional information, such as a number's value or an identifier's name, in a global variable that the parser reads after receiving the token. This side-channel communication is the standard mechanism for transferring semantic values in the lex and yacc world.

Designing Grammar Rules with Yacc

Yacc grammars describe languages using productions, which are rules of the form nonterminal : sequence of symbols. Terminals are the tokens returned by the lexer, while nonterminals are intermediate symbols defined by other productions. The parser's job is to discover a derivation from the start symbol down to the actual token stream it received.

Actions attached to productions execute when the parser reduces that rule. These actions can build syntax trees, evaluate expressions on the fly, or emit intermediate code. The special variables $$, $1, $2, and so on refer to the value of the left-hand side and the values of the right-hand side symbols, respectively.

Writing Productions and Actions

A classic example is an expression grammar supporting addition, subtraction, multiplication, and division. Left-recursive rules like expr : expr PLUS term are preferred in yacc because they keep the parser stack shallow and avoid conflicts. The action for such a rule typically computes the sum and assigns it to $$, propagating the result upward.

Operator precedence matters enormously. Without guidance, the grammar is ambiguous, and yacc will report shift-reduce conflicts. Declaring precedence and associativity with %left and %right directives resolves these conflicts cleanly, letting you write natural-looking rules without sprinkling extra nonterminals everywhere.

Resolving Common Conflicts

Shift-reduce conflicts occur when the parser cannot decide whether to shift the next token or reduce the current rule. Reduce-reduce conflicts occur when two rules could both reduce the same symbol sequence. Yacc reports both in the generated y.output file, which lists every parser state and the conflicting actions.

The dangling-else problem is the textbook shift-reduce example, and yacc resolves it by default in favor of shifting, which implements the expected "else binds to nearest if" behavior. Precedence declarations handle most arithmetic ambiguities, while grammar restructuring handles the rest. Always investigate conflicts rather than silencing them, because they often reveal genuine design flaws.

Connecting Lex and Yacc into a Working Program

The two tools cooperate through a shared interface. The yacc-generated parser calls yylex() whenever it needs input, and the lex-generated scanner returns token codes defined in the yacc file. The magic happens at link time: compile the lex output and the yacc output together, and the resulting executable performs full lexing and parsing without any manual glue code.

The conventional build sequence is to run yacc first with the -d flag to produce a header of token definitions, then run lex, then compile both C files and link them. Makefiles capture this dependency chain, ensuring that changes to the grammar trigger regeneration of the header, which in turn triggers recompilation of the scanner.

Build Steps and Makefile Basics

The commands themselves are refreshingly short. Running yacc -d calc.y produces y.tab.c and y.tab.h. Running lex calc.l produces lex.yy.c. Compiling with cc -o calc y.tab.c lex.yy.c -ly -ll links everything against the lex and yacc runtime libraries on older Unix systems.

On modern Linux distributions, flex and bison replace the originals but remain command-line compatible under the classic names in most cases. A simple Makefile with targets for the grammar, the scanner, and the final binary keeps the workflow reproducible and makes experimenting with grammar changes painless.

Testing and Debugging the Parser

Debugging generated parsers can feel opaque at first. Both lex and yacc offer trace flags. Setting yydebug to a nonzero value makes the yacc parser print every shift and reduce action, revealing exactly how it interprets your input. Similarly, a fallback rule in the lexer that echoes unmatched text helps identify scanning bugs.

A practical strategy is to test the lexer independently by writing a temporary main() that prints each token, then test the parser with hand-crafted token streams. Once each half works in isolation, combine them and run realistic inputs. This divide-and-conquer approach shortens the feedback loop dramatically.

Real-World Applications and Modern Alternatives

Lex and yacc have quietly shaped modern computing. The original Portable C Compiler, early versions of GCC, SQL parsers, configuration file readers, and countless domain-specific languages owe their front ends to these tools. Their design influenced parser generators across every programming ecosystem, from Java's CUP to Python's PLY and beyond.

Today, developers often reach for flex and bison, which are backward-compatible descendants with reentrant scanners, better error messages, and improved memory management. ANTLR takes a different approach with LL parsing and automatic tree generation, while hand-written recursive descent parsers dominate in languages like Go and Rust where performance and error recovery are paramount.

When to Use Parser Generators

Parser generators shine when the grammar is complex, subject to change, or documented as a formal specification. Building a parser for a standardized language like SQL or a configuration format with dozens of keywords by hand is tedious and error-prone, whereas a grammar file expresses the same rules compactly and readably.

They are less appealing for tiny, performance-critical parsers where the generated tables add overhead, or for languages with context-sensitive quirks that require manual lookahead hacks. Knowing both approaches lets you choose the right tool rather than defaulting to one out of habit.

Learning Resources and Next Steps

The classic text remains lex & yacc by Levine, Mason, and Brown, which is compact, example-driven, and still remarkably relevant. The GNU Bison manual and the Flex manual are freely available online and cover the modern dialects thoroughly, including reentrant and push parsers.

For hands-on practice, build progressively harder projects: a calculator, then a JSON parser, then a small programming language interpreter. Each step reinforces the same concepts while introducing new challenges like error recovery, symbol tables, and semantic actions. The skills compound quickly once the initial syntax feels natural.

The best way to internalize lex and yacc is to stop reading and start generating. Pick a small language you use daily, define its tokens, sketch its grammar, and watch the generated parser come alive. The first successful parse of a nontrivial input feels like unlocking a superpower, and from there, the entire landscape of compilers and language tools opens up in ways that hand-written string manipulation never could.

AD