An Enhanced Note Assistance Application with Handwriting Recognition Leveraging LLM

Introduction

In today’s world, many of us rely on handwritten notes, whether on paper or digital devices like iPads. However, interpreting these notes can be challenging. Although there are some existing note-taking applications, most of them focus on basic functionalities like text transcription and keyword search without effectively handling diverse inputs such as handwritten notes, sketches, and complex visual data. Additionally, they lack context-aware querying and intelligent content categorization, making it difficult for users to organize and interact with their notes, especially when dealing with a combination of text, diagrams, and annotations.

Therefore, we would like to introduce an innovative note assistant application that combines the strengths of general-purpose and specialized LLMs to offer a more interactive and structured note-management experience. This project presents a tool to transform the note-taking process through handwriting recognition, sketch conversion, and note question-answering (QA) capabilities. Utilizing large language models (LLM) and optical character recognition (OCR), the application converts handwritten drafts into organized, searchable, and insightful digital notes. It transforms rough sketches into clean formats like Markdown and enables users to query their notes for deeper insights.