Booth Id:
ROBO044T
Category:
Year:
2016
Finalist Names:
Bolya, Daniel
McLeod, Dylan
Abstract:
Neural networks have existed for a relatively short period of time. Thus, many of the best techniques in this field are only now being discovered. In this project, we discuss how a pipeline of convolutional and recurrent neural networks combined with image processing can be used to parse offline handwritten and typed math homework and generate online handwritten answers—two problems that were previously very difficult, but have been made easier with the advent of neural networks. In the process, we discuss a novel way to parse data in the form of areas of interest. We also show how one can generate training data to mimic handwriting without needing to collect handwritten data, both by generating handwritten equations from already-collected data and distorting typed problems to look more like handwriting—a process desirable because data collection can often be very time-consuming and costly. The pipeline process an image of a worksheet taken from a webcam in a series of five steps: division, identification, parsing, solving, and finally writing the answer. We found the accuracy of our pipeline to be difficult to measure, as even though all the components are above 90% accurate on tests sets, the actual input of the pipeline is a real-world image which can vary greatly in quality, readability, and style. Thus, further testing is required to analyze its specific accuracy. While the scope and breadth of our pipeline does not greatly improve on previous equation recognizers—especially online ones—our applications are not deeply rooted in equation parsing, and thus many of the same concepts can be used to solve other difficult tasks such as grading simple free-response tests.