ClassRank: a Method to Measure Class Importance in Knowledge Graphs Applied to Wikidata

Tracking #: 1833-3046

Authors: 
Daniel Fernández-Álvarez
Jose Emilio Labra-Gayo
Daniel Gayo-Avello

Responsible editor: 
Oscar Corcho

Submission type: 
Full Paper
Abstract: 
The use of collaborative knowledge graphs such as Wikidata or DBpedia has increased in the last years. Organizations, universities and individuals have fed those graphs with their own knowledge, producing massive stores of general-purpose data. There are many approaches using the information contained in those initiatives in order to develop applications or to enrich their own data. Nevertheless, each source covers each one of its topics in different depth. This causes the graph to be a better candidate to be exploited in domains of application related to the most important topics rather than the ones with less available information. In order to discover which are the most addressed topics on each source, we propose ClassRank, an algorithm based on aggregated PageRank scores which measure class importance in RDF graphs. In this paper, we test out approach in Wikidata and discuss the collected results by comparing them with the metrics already proposed by Wikidata project. We have found that our approach is more precise than the baseline in high positions of the rankings and that it is able to capture the importance of classes with few but central instances.
Full PDF Version: 
Tags: 
Reviewed

Decision/Status: 
Reject (Two Strikes)

Solicited Reviews:
Click to Expand/Collapse
Review #1
By Giuseppe Rizzo submitted on 16/Apr/2018
Suggestion:
Major Revision
Review Comment:

Thanks for the revision and clarifications, the paper has improved significantly. Before accepting, another round of major revision is necessary, see below.

Point-wise recommendations:
- "Resource Description Language" it's instead Resource Description Framework (RDF)
- p and pi are ambiguous in terms of notation, please address this
- Algorithm 1, Q means? co-occurrence matrix of predicates/objects?
- Algorithm 2, does M still represent the co-occurrence matrix of predicates/objects? why not referring to it as Q likewise Algorithm 1?
- "θa and by θa + 1 are not too different", what does different mean? statistically different? 1-size fits all difference value? the rationale of the choice of θ=5 is extremely brittle and not convincing
>> general comment It is too much left to the interpretation and imagination of the reader the guessing of numerous empirical values used <<
- what about Pc ?
- in Sec. 4, it is often mentioned baseline however there isn't a formal definition. I acknowledge that the authors say that the baseline is what Wikidata uses to compute the class importance. But, how does the baseline work? Please provide it in order to have a self-contained paper
- the results need further discussions and analyses. Despite the efforts of the authors to introduce a comparison with an existing method, authors use precision and recall without setting up the scene of the gold standard used and their requirements that lead to its definition. As we can imagine, this may reflect significantly the ultimate measurements and thus conclusions that can be narrowed down

Still numerous typos such as "is an scenario", "The numbers of ... keeps", "is a clearly an instance of an award", "per each" and many more.
Once more, I also recommend to stick with scientific writing using capital letters when pointing to sections ("in the section X" -> "in Section X"), algorithms and tables.

Review #2
Anonymous submitted on 16/May/2018
Suggestion:
Minor Revision
Review Comment:

This paper presents, ClassRank, an algorithm for calculating the importance of a class in a given knowledge graph which is based on PageRank algorithm. In their work, the authors aim to answer two research questions: (a) Which is
the most accurate way to detect classes in an RDF graph?, and (b) how to measure the importance of those classes in a given KG? An experiment on Wikidata is used for evaluating the results.

I recommend a "Minor Revision" since I believe that the authors have answered most of the points that were made in the review satisfactorily. The readability of the paper is significantly improved. I have listed my comments for this version.

Few more comments:

Section 1. I believe the inclusion of where the class rank algorithm can be potentially used is a good addition to the paper. It would help the reader if real-world examples added to graph summarization, and IR similar to the ones provided in class prioritization. If possible, please add footnote link or reference when you mention the use of the algorithm such as the ontology alignment tool, etc..

4.4 It's a bit strange that baseline approach has low precision (a large number of false positives). I expected the baseline to have a high precision and a low recall as it depended rdf:type and rdfs:subClassOf assertions. Do you think this result is kind of specific to Wikidata? Specially because authors also mention that this is due to few common patterns such as countries. It is also interesting to know what entities are typed as France (i.e., x rdf:type France).

4.5 "Since our approach just considers objects in triples, those elements cannot be detected by ClassRank". If I understood correct, the baseline uses (s, instance of, o) and (child, subclass of, parent) relations, isn't it? So the only elements that can't be detected are the "child" elements?

4.5 Is it possible to check what is the optimal value for theta?

6.1 - I find some of the explanations hard to follow. For instance, it is not quite clear which method or approaches some pronouns (it, they) refers to.

* Styling and writing.
I would recommend to use capital letters when pointing to sections, algorithms and tables. For example,
In section 2 -> In Section 2

Review #3
Anonymous submitted on 06/Jun/2018
Suggestion:
Reject
Review Comment:

Dear authors, thanks for the responses provided and the changes in the manuscript. Now this paper is more attractive to me, although I have several additional questions divided into high-level and low-level:
High level questions:
- Still it is unclear to me to me the importance of classes importance. Sorry for the play on words, but I cannot read in the document examples or argumentation describing the utility of knowing the importance of types. I see the importance of resources/entities in a KG (PageRank), but the importance of the classes beyond the number of in/out links (the baseline) is unclear to me. Perhaps some examples could provide a better understanding.

- Evaluation. In section Conclusions you claim "proven" facts (Research Questions stated in intro section). However I do not see the prove. Section 4.4 describes the process, in which a threshold values of 5 is selected. For this threshold is computed ClassRank, obtaining a list of classes. This is compared to the list of classes provided by Wikidata (baseline), based in the number of in/out links. You select ranges and say for the top-100 range that "baseline has a 18% rate of false positives". How do you get this number? The only human-made evaluation is the list of class-pointer. You get a list of classes (with 68,897 elements) ranked by "importance". However, this list of classes has not been validated by humans.

- Skew due to non transitivity. In Section 3.5 you say that your approach does not consider the transitivity of type. For me this is a severe limitation. Removing this transitivity you remove a very important semantic relation between classes that probably affects the ranking of classes importance.

Low level questions:
- The evaluation, as I said before, is unclear to me. Here are more details: Figure 3 caption says |Pc|= 62, while the list of Pc accepted after revision (human-made) has 80 elements. Why you reduced the number of Pc's for the evaluation?. This caption also says "not found in other lists". What other lists you mean?.
- The URL provided in page 7 for accessing the obtained results (http://boa.weso.es/wikidata) does not work.