Lexemes vs Tokens in Lexical Analysis: Which Option Represents a Lexeme?

Verified Sources

Oct 28, 2026

Lexeme and Token | Lexical Analysis (Compiler Design)

In compiler design, a lexical analyzer (lexer) scans the source text left-to-right and groups the character stream into lexemes and then categorizes them as tokens. A keyword is a kind of token, while its actual spelling in the source code is the corresponding lexeme. Likewise, an integer literal token is formed from the actual digit sequence in the program as a lexeme.

Formally:

A lexeme is the actual sequence of characters from the source program that matches a token’s pattern. 2
A token represents the meaning class sent to the parser, typically as (token name + optional value). 2
A pattern describes how lexemes are recognized. 2

So, in the statement:

1int total = 25;

the lexemes include the actual character sequences int, total, and 25.

Mermaid (flow of lexical analysis):

Compilers (Lexical Analyzer: Tokens, Patterns, Lexemes) - Defines token as a category, lexeme as an actual character sequence instance, and pattern as a rule/pattern. ↩ ↩² ↩³
Lexeme and Token | Lexical Analysis (DFA for different program elements) - Educational explanation of lexeme/token distinction in lexical analysis. ↩ ↩²
Lexical Analysis - Wikipedia - States that a lexeme is the string of characters of a certain kind; the lexeme’s type/value constitutes a token representation for the parser. ↩

Answer to the MCQ

Given: (i) int
(ii) 25
(iii) total
(iv) All of the above

The correct choice is (iv) All of the Above, because each of int, total, and 25 is a lexeme: each is the exact sequence of characters in the source text that the lexer recognizes as matching patterns for some token types (keyword, identifier, integer literal). 2

To make the token/lexeme distinction explicit:

Source substring	Lexeme (exact text)	Likely token category (example)
`int`	`int`	keyword token
`total`	`total`	identifier token
`25`	`25`	integer constant token

This matches the definitions that:

a lexeme is an “actual character sequence” forming a token instance, and
tokens are the categories passed onward to the parser. 2

Compilers (Lexical Analyzer: Tokens, Patterns, Lexemes) - Defines token as a category, lexeme as an actual character sequence instance, and pattern as a rule/pattern. ↩ ↩²
Lexeme and Token | Lexical Analysis (DFA for different program elements) - Educational explanation of lexeme/token distinction in lexical analysis. ↩ ↩²

Memory aid

If you can put quotes around the exact characters you see in the code (e.g., "25"), that’s a strong hint it’s a lexeme.

Common misconception

A token is NOT the same thing as the lexeme. Tokens are categories like "IF" or "INT_LITERAL"; the lexeme is the actual text (like "if" or "25") that produced that token.

How the lexer identifies lexemes (using this line as an example)

1
Step 1
The lexer processes the source stream for potential token boundaries (whitespace and punctuation help delimiting lexemes).
2
Step 2
It finds the exact substrings int, total, and 25 that match the language’s token patterns for keywords, identifiers, and integer literals.
3
Step 3
Each lexeme is categorized into a token name (e.g., keyword vs identifier vs integer literal).
4
Step 4
For identifier and numeric literals, the lexer may store an associated attribute (e.g., identifier table reference, literal numeric value).
5
Step 5
The parser receives tokens (categories + values), not raw character lexemes.

Lexemes in `int total = 25;`

The exact substrings are lexemes; they map to token categories.

FAQ: Lexeme vs token in quick examples

Lexical Analysis Core Terms

1 / 4

Question · Term

Lexeme

Click to reveal

Answer · Definition

Exact matched substring in source code that matches a token’s pattern.

Knowledge Check

Question 1 of 3

Q1Single choice

In int total = 25;, which options represent lexemes?

(i) and (ii)

(ii) and (iii)

(i), (ii), and (iii)

Only (iii)

Explore Related Topics

Quadruple Record Structure in Compiler Intermediate Code

Regular Languages Among Given Options: A Formal Language Analysis

Compiler Phase That Converts Source Code into Tokens

Lexical analysis, also called scanning or tokenization, is the compiler front‑end phase that reads raw source characters and groups them into tokens such as keywords, identifiers, literals, operators, and punctuation.

It converts each lexeme (e.g., int, x, =) into a token class, producing a token stream for the parser.
This phase runs before syntax analysis, which checks token order against the language grammar.
Whitespace and comments are typically ignored, and lexical errors (invalid characters) are reported here.
Lexical analysis is distinct from later phases like parsing, optimization, and code generation, which operate on already‑tokenized structures.

Browse all research articles