The recent trends in NLP has shown great interest in understanding text to perform various tasks. Understanding
text to perform mathematical reasoning have focused
on automatically solving school level
math word problems. Advancement in this area has great
potential to be used as automatic tutoring service
for school students.
This blog focuses on a web based tool named ILLINOIS MATH SOLVER that supports performing mathematical reasoning [1]. The solver can answer a wide range of mathematics questions, ranging from operation questions like “What is the result when 6 is divided by the sum of 7 and 5 ?” to elementary school level math word problems, like “I bought 6 apples. I ate 3 of them. How many do I have left ?”. ILLINOIS MATH SOLVER provides an easy way to test the robustness of the system, and a tool for crowd based data acquisition.
Working Description
The whole system comprises of two different modules, firstly a Context Free Grammar(CFG) based semantic parser to handle queries for operation between numbers (addition, difference, fraction). The parser creates a list of number and mathematical terms using some derivation rules. For example, if the question is "What is the result when 15 is multiplied to difference of 5 and 12 ?", it first creates a list as {15,multiplied, difference,5,12}. Here, the word "multiplied" helps to parse an expression into multiple sub-expressions. 26 such derivation rules are used and Cocke-Younger-Kasami(CYK) algorithm is used for parsing. Secondly, a Arithmetic Problem Solver [2] is used to handle arithmetic problems with multiple steps and operations which decomposes an input arithmetic problem into several decision problems, and learns predictors for these decision problems resulting in generation of a binary expression tree for the solution mathematical expression.
A classifier is learnt to predict a math operation along with its order. It finds the lowest common ancestor (LCA) node in the expression tree. In the above figure, this multi-class classifier task is to first perform the addition and then the multiplication operation. Also, number "2" in the above figure is irrelevant for the solution. Another classifier is trained to predict such irrelevant quantities in the problem.
Evaluation
The system was evaluated on union of three datasets (addition subtraction problems from AI2 dataset (AI2) [3], single operation problems from Illinois dataset (IL) [4] and multi-step problems from commoncore dataset (CC) [2]) and found to achieve state-of-the-art performance on all these datasets.
Schematic Diagram of Illinois Math Solver
Limitations
Currently the system works by combining the numbers mentioned in the next, but is unable to perform its task for strings like "1 day" as the prior knowledge of 1 day -> 24 hours is required for the same. It is also unable to handle algebra word problems involving multiple equations with one or more variables.
References
[1] Subhro Roy and Dan Roth. 2016 "Illinois Math Solver: Math Reasoning on the Web" Proceedings of NAACL-HLT (Demonstrations), Association for Computational Linguistics.
[2] Subhro Roy and Dan Roth. 2015. "Solving general arithmetic word problems" In Proc. of the Conference on Empirical Methods in Natural Language Processing (EMNLP)
[3] M. J. Hosseini, H. Hajishirzi, O. Etzioni, and N. Kushman. 2014. "Learning to solve arithmetic word problems with verb categorization" In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014
[4] S. Roy, T. Vieira, and D. Roth. 2015. "Reasoning about quantities in natural language". Transactions of the Association for Computational Linguistics, 3
This blog focuses on a web based tool named ILLINOIS MATH SOLVER that supports performing mathematical reasoning [1]. The solver can answer a wide range of mathematics questions, ranging from operation questions like “What is the result when 6 is divided by the sum of 7 and 5 ?” to elementary school level math word problems, like “I bought 6 apples. I ate 3 of them. How many do I have left ?”. ILLINOIS MATH SOLVER provides an easy way to test the robustness of the system, and a tool for crowd based data acquisition.
 |
| Fig. Screenshot of Illinois Math Solver |
The whole system comprises of two different modules, firstly a Context Free Grammar(CFG) based semantic parser to handle queries for operation between numbers (addition, difference, fraction). The parser creates a list of number and mathematical terms using some derivation rules. For example, if the question is "What is the result when 15 is multiplied to difference of 5 and 12 ?", it first creates a list as {15,multiplied, difference,5,12}. Here, the word "multiplied" helps to parse an expression into multiple sub-expressions. 26 such derivation rules are used and Cocke-Younger-Kasami(CYK) algorithm is used for parsing. Secondly, a Arithmetic Problem Solver [2] is used to handle arithmetic problems with multiple steps and operations which decomposes an input arithmetic problem into several decision problems, and learns predictors for these decision problems resulting in generation of a binary expression tree for the solution mathematical expression.
 |
| An example of arithmetic word problem with its solution and expression tree |
Evaluation
The system was evaluated on union of three datasets (addition subtraction problems from AI2 dataset (AI2) [3], single operation problems from Illinois dataset (IL) [4] and multi-step problems from commoncore dataset (CC) [2]) and found to achieve state-of-the-art performance on all these datasets.
Schematic Diagram of Illinois Math Solver
Limitations
Currently the system works by combining the numbers mentioned in the next, but is unable to perform its task for strings like "1 day" as the prior knowledge of 1 day -> 24 hours is required for the same. It is also unable to handle algebra word problems involving multiple equations with one or more variables.
References
[1] Subhro Roy and Dan Roth. 2016 "Illinois Math Solver: Math Reasoning on the Web" Proceedings of NAACL-HLT (Demonstrations), Association for Computational Linguistics.
[2] Subhro Roy and Dan Roth. 2015. "Solving general arithmetic word problems" In Proc. of the Conference on Empirical Methods in Natural Language Processing (EMNLP)
[3] M. J. Hosseini, H. Hajishirzi, O. Etzioni, and N. Kushman. 2014. "Learning to solve arithmetic word problems with verb categorization" In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014
[4] S. Roy, T. Vieira, and D. Roth. 2015. "Reasoning about quantities in natural language". Transactions of the Association for Computational Linguistics, 3
Comments
Post a Comment