Coursify

Compiler Phase That Converts Source Code into Tokens

Compiler Phase That Converts Source Code into Tokens

Verified Sources
May 20, 2026

The correct answer is (ii) Lexical analysis. In a compiler frontend, the lexical analysis phase reads the raw character stream of a program and groups it into tokens, such as keywords, identifiers, literals, operators, and punctuation.2 This phase is also called scanning or tokenization, and it occurs before parsing, which uses the token stream to verify grammatical structure.2

For example, the statement int x = 10; is converted during lexical analysis into a token sequence like:

LexemeToken Class
intkeyword
xidentifier
=operator
10literal
;separator

This transformation is not the job of code generation or optimization. Code generation happens much later, after the program has already been analyzed.2

Footnotes

  1. Lexical analysis - Wikipedia - Defines lexical analysis as tokenization and explains its role before parsing. ↩ ↩2

  2. Compiler Design - Lexical Analysis - TutorialsPoint - Describes lexical analysis as the first compiler phase that breaks source code into tokens. ↩

  3. Lexical Analysis and Syntax Analysis - GeeksforGeeks - Compares lexical analysis with syntax analysis and explains that parsing consumes tokens. ↩

  4. 6 Phases Of Compiler | A Detailed Explanation - Summarizes compiler phases and distinguishes lexical analysis from optimization and code generation. ↩

  5. Compilation Phases Explained - Analysis and Synthesis - Explains the sequence of compilation stages, including optimization and final code generation. ↩

Lexical Analyzer – Tokenization

Direct Answer

Among the options, lexical analysis is the compiler phase that converts source code into tokens.2

Footnotes

  1. Lexical analysis - Wikipedia - Defines lexical analysis as tokenization and explains its role before parsing. ↩

  2. Compiler Design - Lexical Analysis - TutorialsPoint - Describes lexical analysis as the first compiler phase that breaks source code into tokens. ↩

To understand why option (ii) is correct, it helps to distinguish the major compiler phases. A compiler does not transform a program in one step; instead, it performs a sequence of analyses and transformations.2

A simplified pipeline is:

  1. Lexical analysis: converts characters into tokens.2
  2. Syntax analysis (parsing): checks whether the token sequence follows the language grammar and often builds a parse tree or AST.2
  3. Semantic analysis: verifies meaning, such as type compatibility and declarations.2
  4. Intermediate code generation: produces a machine-independent representation.2
  5. Optimization: improves efficiency of the intermediate representation.2
  6. Code generation: emits target machine code or assembly.2

The key distinction is that lexical analysis focuses on individual lexical units, while parsing focuses on relationships among tokens in accordance with grammar rules.2

Footnotes

  1. 6 Phases Of Compiler | A Detailed Explanation - Summarizes compiler phases and distinguishes lexical analysis from optimization and code generation. ↩ ↩2 ↩3 ↩4 ↩5

  2. Compilation Phases Explained - Analysis and Synthesis - Explains the sequence of compilation stages, including optimization and final code generation. ↩ ↩2 ↩3 ↩4 ↩5

  3. Lexical analysis - Wikipedia - Defines lexical analysis as tokenization and explains its role before parsing. ↩

  4. Compiler Design - Lexical Analysis - TutorialsPoint - Describes lexical analysis as the first compiler phase that breaks source code into tokens. ↩

  5. Lexical Analysis and Syntax Analysis - GeeksforGeeks - Compares lexical analysis with syntax analysis and explains that parsing consumes tokens. ↩ ↩2

  6. Lexical and Syntax Analysis (PDF) - Virginia Tech - States that lexical analysis outputs tokens and syntactic analysis outputs a parse tree. ↩ ↩2

Conceptual Role of Compiler Phases

Relative emphasis on token formation versus structural analysis

How Source Code Becomes Tokens

  1. 1
    Step 1

    The scanner reads source code one character at a time from left to right, treating the program initially as raw text rather than as grammatical structure.2

    Footnotes

    1. Lexical analysis - Wikipedia - Defines lexical analysis as tokenization and explains its role before parsing. ↩

    2. Compiler Design - Lexical Analysis - TutorialsPoint - Describes lexical analysis as the first compiler phase that breaks source code into tokens. ↩

  2. 2
    Step 2

    It identifies meaningful character sequences called lexemes, such as while, count, 42, or +.2

    Footnotes

    1. Lexical analysis - Wikipedia - Defines lexical analysis as tokenization and explains its role before parsing. ↩

    2. Compiler Design - Lexical Analysis - TutorialsPoint - Describes lexical analysis as the first compiler phase that breaks source code into tokens. ↩

  3. 3
    Step 3

    Each lexeme is assigned to a token class such as keyword, identifier, operator, literal, or punctuation.2

    Footnotes

    1. Lexical analysis - Wikipedia - Defines lexical analysis as tokenization and explains its role before parsing. ↩

    2. Compiler Design - Lexical Analysis - TutorialsPoint - Describes lexical analysis as the first compiler phase that breaks source code into tokens. ↩

  4. 4
    Step 4

    Whitespace and comments are usually ignored or handled specially so they do not clutter later grammatical processing.2

    Footnotes

    1. Lexical analysis - Wikipedia - Defines lexical analysis as tokenization and explains its role before parsing. ↩

    2. Analysis of the source program, Phases of a compiler (PDF) - Explains lexeme, token, and pattern, and notes that whitespace is discarded during lexical analysis. ↩

  5. 5
    Step 5

    The resulting token stream is passed to syntax analysis, which checks whether the tokens form valid language constructs.2

    Footnotes

    1. Lexical Analysis and Syntax Analysis - GeeksforGeeks - Compares lexical analysis with syntax analysis and explains that parsing consumes tokens. ↩

    2. Lexical and Syntax Analysis (PDF) - Virginia Tech - States that lexical analysis outputs tokens and syntactic analysis outputs a parse tree. ↩

A common examination trap is confusing lexical analysis with parsing. Both occur early in compilation, but they do different jobs.2

Why parsing is not the correct answer

Parsing takes the token sequence produced by the lexical analyzer and checks it against the language grammar. It is responsible for building structures such as parse trees and syntax trees, not for splitting the raw source characters into tokens.2

Why optimization is not the correct answer

Optimization happens after the program has already been represented in a more structured form, usually in intermediate code or another internal representation. It may remove redundant computations or improve execution efficiency, but it does not tokenize source text.2

Why code generation is not the correct answer

Code generation is the phase that converts the compiler’s internal representation into assembly or machine code for a target architecture. By that point, tokenization has long been completed.2

Footnotes

  1. Lexical Analysis and Syntax Analysis - GeeksforGeeks - Compares lexical analysis with syntax analysis and explains that parsing consumes tokens. ↩ ↩2

  2. Lexical and Syntax Analysis (PDF) - Virginia Tech - States that lexical analysis outputs tokens and syntactic analysis outputs a parse tree. ↩ ↩2

  3. 6 Phases Of Compiler | A Detailed Explanation - Summarizes compiler phases and distinguishes lexical analysis from optimization and code generation. ↩ ↩2

  4. Compilation Phases Explained - Analysis and Synthesis - Explains the sequence of compilation stages, including optimization and final code generation. ↩ ↩2

(ii) Lexical analysis — this phase converts source code into tokens such as identifiers, keywords, operators, and literals.2

Footnotes

  1. Lexical analysis - Wikipedia - Defines lexical analysis as tokenization and explains its role before parsing. ↩

  2. Compiler Design - Lexical Analysis - TutorialsPoint - Describes lexical analysis as the first compiler phase that breaks source code into tokens. ↩

Exam Strategy

If a question asks about converting characters or source text into keywords, identifiers, literals, and symbols, think lexical analysis or tokenization.2

Footnotes

  1. Lexical analysis - Wikipedia - Defines lexical analysis as tokenization and explains its role before parsing. ↩

  2. Compiler Design - Lexical Analysis - TutorialsPoint - Describes lexical analysis as the first compiler phase that breaks source code into tokens. ↩

Common Confusion

Do not confuse token creation with grammar checking. Lexical analysis creates tokens; parsing checks token order and structure.2

Footnotes

  1. Lexical Analysis and Syntax Analysis - GeeksforGeeks - Compares lexical analysis with syntax analysis and explains that parsing consumes tokens. ↩

  2. Lexical and Syntax Analysis (PDF) - Virginia Tech - States that lexical analysis outputs tokens and syntactic analysis outputs a parse tree. ↩

Frequently Tested Clarifications

Consider the multiple-choice question directly:

Which phase of compiler converts source code into tokens?

  • (i) Code generation
  • (ii) Lexical analysis
  • (iii) Optimization
  • (iv) Parsing

The academically correct choice is (ii) Lexical analysis because the lexer is explicitly designed to scan source characters, recognize lexemes using patterns such as regular expressions, and emit tokens for subsequent compilation phases.3

This is a foundational concept in compiler design and often appears in introductory questions because it tests whether the learner can distinguish tokenization from syntax checking and from later machine-oriented phases.2

Footnotes

  1. Lexical analysis - Wikipedia - Defines lexical analysis as tokenization and explains its role before parsing. ↩

  2. Compiler Design - Lexical Analysis - TutorialsPoint - Describes lexical analysis as the first compiler phase that breaks source code into tokens. ↩

  3. Analysis of the source program, Phases of a compiler (PDF) - Explains lexeme, token, and pattern, and notes that whitespace is discarded during lexical analysis. ↩

  4. Lexical Analysis and Syntax Analysis - GeeksforGeeks - Compares lexical analysis with syntax analysis and explains that parsing consumes tokens. ↩

  5. 6 Phases Of Compiler | A Detailed Explanation - Summarizes compiler phases and distinguishes lexical analysis from optimization and code generation. ↩

Knowledge Check

Question 1 of 4
Q1Single choice

Which compiler phase converts raw source code characters into tokens?

Explore Related Topics

1

Introduction to Compiler Design and Architecture

The course introduces the fundamental structure and operation of modern compilers, describing how source code is transformed through front‑end analysis, intermediate representation, and back‑end generation.

  • Front‑end performs lexical, syntax, and semantic analysis, building a symbol table and an AST independent of the target.
  • An intermediate representation (IR) like three‑address code lets language‑independent optimizations run before back‑end register and instruction selection.
  • Optimization passes (e.g., dead‑code elimination, loop unrolling) on the IR consume about 50 % of compilation CPU time.
  • Top‑down parsers fail on left‑recursive grammars; they are fixed by rewriting A → Aα | β as A → β A' and A' → α A' | ε.
2

Which Component Is the Brain of a Microcomputer System?

The microprocessor is the brain of a microcomputer because it executes instructions, controls operations, and incorporates the ALU, control unit, and registers.

  • It combines all CPU functions on one chip, unlike RAM (volatile workspace) or ROM (permanent storage).
  • The ALU only performs arithmetic/logic and cannot direct the whole system.
  • Formula: Microcomputer Brain=Instruction Control+Execution+Coordination\text{Microcomputer Brain} = \text{Instruction Control} + \text{Execution} + \text{Coordination}
  • Exam strategy: discard memory components and sub‑units, leaving the microprocessor as the correct choice.
3

Lexical Analysis and the Main Structure Used: Finite Automata

Lexical analysis relies on finite automata—typically deterministic finite automata (DFA)—to recognize token patterns defined by regular expressions.

  • Regular expressions for identifiers, numbers, etc., are converted to NFAs then to a DFA for fast scanning.
  • The DFA processes the source character by character, tracking a single current state and emitting a token at each accepting state.
  • Queues, stacks, and trees support other compiler phases (parsing, AST construction) but are not the primary model for token recognition.
  • Lexers output a stream of tokens that the parser consumes for syntax analysis.
Browse all research articles