Review Comment:
The paper presents Canali, a system for answering NL questions over KBs, supporting auto completion. The overall approach is based on automata, trying to determine user intentions and suggest relevant content, guiding the formulation of valid questions that match the underlying KBs.
Abstract
“This novel feature enhances the interaction with and the usability of the CANaLI which also delivers a high level of accuracy and precision” => This novel feature enhances the interaction with and the usability of CANaLI, demonstrating at the same time high accuracy and precision.
Introduction
“In this paper, we provide a detailed presentation of the controlled NL parsing techniques that led that success and also enabled support for real-time question completion idea greatly improves the usability of CANaLI and the interactive experience users have with the system” => In this paper, we provide a detailed presentation of the controlled NL parsing techniques that led that success and also enabled real-time question completion, greatly improving the usability of CANaLI and the interactive experience users have with the system.
“CNL approach” ?? Canali Natural Language approach ?
“After a short description of SWiPE, ClioPedia and SWiPE” => After a short description of SWiPE, ClioPedia and Xser?
Introduction needs to be restructured. In page two, the paper suddenly starts describing related work, which I think it is better to include it in section 6.
Overall
One of the positive aspects of the paper is that it includes many examples that help the reader understand the overall concept and capabilities of the system (e.g. in section 2 and section 5).
However, the paper lacks a formal description of the approach in section 3. The presentation is quite descriptive/narrative, mixing long paragraphs of examples with definitions and explanations that hamper comprehensibility. It is more like a story than formally defining, describing, explaining and exemplifying. I would suggest that the authors revise section 3 and include a formal and well-defined definition of the several concepts proposed in the paper, reducing the size of the paragraphs (for example, the paragraph starting with S1: in page 7 is more than one column in length), and also include algorithms, functions, etc. that will help readers better understand the way that all the proposed concepts, elements, properties, etc. are connected. The examples should complement the formal descriptions, and should not be used as the only means to present the overall concept.
Moreover, the contribution of the paper is not clear. It is mentioned that this paper presents extended evaluation results, compared to the ones presented in [15]. This is OK, but the paper needs to present something more than plain results. Earlier, it is mentioned that this paper presents a detailed presentation of the controlled NL parsing that led the success of a previous (demo) paper in QALD6. Later in the same section, it is mentioned that even without the autocompletion feature, Canali achieves high precision, referencing again QALD6. So, it is not clear what aspects of the presented paper were also part of the demo in QALD6 and what parts are new.
Regarding the experimental evaluation, I have some questions that are mainly relevant to the auto completion feature.
It seems that the experimental evaluation has two parts: the first part is described at the beginning of section 5 and the second part in section 5.1. The difference between these two sections is not clear to me. It seems that the first part uses the auto completion feature, while the second part evaluates only the CNL subsystem. Is this true?
In any case, I guess that the auto completion feature needs someone to type the queries at runtime, right? Therefore, the system gets some help from the user for disambiguating sentences, since the user selects the suggestion that best matches her/his needs. Although this is perfectly fine, how can the system be compared to other systems that get as input questions in textual form, without further “assistance” from the user, and try to automatically disambiguate?
In section 5.1, several examples are presented on how some questions have been reformed in order to be handled by Canali. In one of these questions (what is the official web site of Tom Cruise), the reformulation is quite weird. For example, the revised question is “what is the web site of Tom Cruise”, which is quite similar to the original one. This raises the question to what extend the system employs semantic QA capabilities, apart from retrieving answers from KBs beyond simple keyword-based matches. For example, the terms “official web site” and “web site” are quite similar and any semantic-based QA system should be able to make this connection. Canali seems to heavily based on keywords, rather on the semantics of the input. I guess vocabularies or dictionaries, like WordNet, BebelNet, etc. are not taken into account, right? What if the question of the user contains, for example, a property that is not part of the KB vocabulary, but instead an equivalent property is present, for example, “web site” instead of “web page”. Overall, is not clear the semantic capabilities of the system in terms of ontologies, reasoning, etc. at the NL level, e.g. at the level of processing/parsing the user input and semantically understanding what the user means, beyond simple lucene matching.
Finally, unless I have missed it, the paper does not include any result regarding the response time which involves:
1. Response time regarding the generation/presentation of suggestions, as the user is typing
2. Response time regarding SPARQL execution, which also encapsulates the structure/quality of the generated SPARQL queries.
Regarding 1, it would be nice to include some times in order to have a picture of the time the user needs to wait till some suggestions are presented. For the latter, the paper does not discuss the way SPARQL queries are generated. Is there an one-to-one, straightforward mapping of the automata to SPARQL structures? Is there any optimization taking place? In principle, there are more than one ways to write a SPARQL query and I was wondering what the approach of Canali is.
|