Recognising handwritten mathematical expressions is one of the hardest challenges in computer vision. Unlike printed text, handwritten maths involves nested structures, variable spacing, and ambiguous symbols. This post explores how we built an HMER (Handwritten Mathematical Expression Recognition) model using PyTorch with Attention and Coverage mechanisms.
Why HMER Is Challenging
- Two-dimensional structure: fractions, superscripts, and subscripts require spatial reasoning
- Ambiguous symbols: ‘1’, ‘l’, ‘I’, and ‘|’ can look identical in handwriting
- Variable sequence lengths: expressions range from simple single symbols to multi-line equations
- Lack of large labelled datasets compared to standard OCR tasks
The Attention + Coverage Architecture
Our PyTorch implementation uses an encoder-decoder architecture. The CNN encoder extracts visual features from the input image. The GRU decoder generates LaTeX tokens one at a time, guided by an attention mechanism that focuses on relevant image regions.
The Coverage mechanism addresses a key weakness of standard attention: it tracks which parts of the image have already been attended to, preventing the decoder from repeatedly focusing on the same region and missing symbols.
Training the Model
The model was trained on the CROHME dataset (Competition on Recognition of Online Handwritten Mathematical Expressions). Key training details include teacher forcing during training, cross-entropy loss on LaTeX token sequences, and Adam optimiser with learning rate scheduling.
Output: LaTeX Notation
The model outputs LaTeX code that can be rendered directly in scientific documents, web pages, and education platforms. For example, a handwritten fraction becomes \frac{a}{b} automatically.
Applications
- Digitising handwritten notes for students and researchers
- Automated marking systems for mathematics assignments
- Accessibility tools for converting whiteboard equations to digital format
- STEM education platforms
Explore the full project on GitHub. Ozlin Info builds custom AI/ML solutions for education and enterprise. Contact us to discuss your project.