InkFM: A Foundational Model for Full-Page Online Handwritten Note Understanding
Abstract
Tablets and styluses are increasingly popular for taking notes. To optimize this experience and ensure a smooth and efficient workflow, it's important to develop methods for accurately interpreting and understanding the content of handwritten digital notes. We introduce a foundational model called InkFM for analyzing full pages of handwritten content. Trained on a diverse mixture of tasks, this model offers a unique combination of capabilities: recognizing text in 28 different scripts, mathematical expressions recognition, and segmenting pages into distinct elements like text and drawings. Our results demonstrate that these tasks can be effectively unified within a single model, achieving SoTA text line segmentation out-of-the-box quality surpassing public baselines like docTR. Fine- or LoRA-tuning our base model on public datasets further improves the quality of page segmentation, achieves state-of the art text recognition (DeepWriting, CASIA, SCUT, and Mathwriting datasets) and sketch classification (QuickDraw). This adaptability of InkFM provides a powerful starting point for developing applications with handwritten input.
Cite
@article{arxiv.2503.23081,
title = {InkFM: A Foundational Model for Full-Page Online Handwritten Note Understanding},
author = {Anastasiia Fadeeva and Vincent Coriou and Diego Antognini and Claudiu Musat and Andrii Maksai},
journal= {arXiv preprint arXiv:2503.23081},
year = {2025}
}