Difficulty-level Modeling of Ontology-based Factual Questions

Tracking #: 1712-2924

This paper is currently under review
Authors: 
Vinu E. V
P Sreenivasa Kumar

Responsible editor: 
Michel Dumontier

Submission type: 
Full Paper
Abstract: 
Semantics-based knowledge representations such as ontologies are found to be very useful in automatically generating meaningful factual questions. Determining the difficulty-level of these system generated questions is helpful to effectively utilize them in various educational and professional applications. The existing approaches for finding the difficulty-level of factual questions are very simple and are limited to a few basic principles. We propose a new methodology for this problem by considering an educational theory called Item Response Theory (IRT). In the IRT, knowledge proficiency of end users (learners) are considered for assigning difficulty-levels, because of the assumptions that a given question is perceived differently by learners of various proficiencies. We have done a detailed study on the features/factors of a question statement which could possibly determine its difficulty-level for three learner categories (experts, intermediates, and beginners). We formulate ontology-based metrics for the same. We then train three logistic regression models to predict the difficulty-level corresponding to the three learner categories. The output of these models is interpreted using the IRT to find the question’s overall difficulty-level. The performance of the models based on cross-validation is found to be satisfactory and, the predicted difficulty-levels of questions (chosen from four domains) were found to be close to their actual difficulty-levels determined by domain experts. Comparison with the state-of-the-art method shows an improvement of 8.5% in correctly predicating the difficulty-levels of benchmark questions.
Full PDF Version: 
Tags: 
Under Review

Comments

Dear Reviewers,
I have identified a typo in the manuscript. I kindly request you to take the following minor change into consideration while reviewing the paper.

In the Abstract, instead of "8.5% improved" it should have been "20.5%" (from 67% to 87.5%) -- the same mistake has happened at the conclusion section as well.

Thanking you.