Automatic Detection of Relation Assertion Errors and Induction of Relation Constraints

The paper provides three tightly-related contributions in the domain of knowledge graph maintenance:
(1) a method, called PaTyBRED, for detection of erronous facts in knowledge graphs
(2) the CoCKG method to automatically fix those facts
(3) a method to induce relation constraints in SHACL.

The first two contributions has been already published, thus the paper positions them as subtasks for the greater goal of knowledge graph quality maintenance. The experimental evaluation shows that the presented techniques (except for CoCKG) outperform significantly the state of the art in their corresponding task. Besides, the presented techniques seem sound, and the paper's assumptions reasonable.

1) Legibility and presentation

The paper is well-written and fairly legible. Its structure is adequate in my opinion.

2) Scientific contribution

The paper tackles an important problem: knowledge base automatic correction. It first focuses on the detection of erronous facts with PaTyBRED. Once detected, those facts can be automatically corrected with CoCKG. Furthermore, the detected errors can be used to learn constraints that can help in the detection of problems a priori. While PaTyBRED and constraint induction show quite satisfactory results, the results of CoCKG's evaluation are rather not satisfactory. Nevertheless, I must acknowledge that automatic KG correction, when limited to internal dataset features, is a very hard task.

3) Detailed Review

Section 4.2.
- Q: Can two atoms in a path share both variables, i.e., residenceCountry(x1, y1), nationality(x1, y1)? If not, why? In any case, the paper should clarify via examples how a path looks like.
- I would appreciate more details about how the metrics inter(A, B), m1(A, B) and m2(A, B) are used to prune irrelevant paths. Is there any thresholding involved?

Section 4.3

- The examples "child(Trump, Ivanka), child(William, George), child(Kate, George), spouse(Trump, Melania)" overflow the column space.

Section 5.2

- The example after equations 4-7 is not really clear.

Section 5.4

- The authors claim to have done the evaluation on DBpedia and NELL, but Figure 3 shows results for YAGO too. It would be great to explain in a couple of sentences why PaTyBRED performs poorly on YAGO with more features (yago25).

Section 6.

- Given the low precision of CoCKG, I wonder why the authors have not considered to reduce the number of candidate corrections by using a link prediction approach, i.e, among the surviving correction subject/object candidates, take those that a link prediction approach would rate as likely to occur in the triple. Moreover, by adding link prediction to the formula, the authors could test the performance of CoCKG when dropping the assumption that confussions tend to use more general IRIs.

- While the authors have stated their preference for endogenous features, they could use external sources such as search engines or other KBs (via the co-occurrence of the entities) to test the viability of a correction candidate.

Section 8.2

- The example depicted in Figure 5 is not clear. I would recommend the authors to replace the labels c_i with actual examples.

- The a-posteriori pruning of the decision trees raises the question of why not applying parameter tuning when learning the tree.

- It would be great to publish all the learned SHACL constraints.

I would suggest a major revision for this paper. In particular the contributions of Section 6 are not convincing and need to be improved.

The paper targets the problem of error detection in relation assertions of knowledge graphs and proposes a method based on path and type features. The paper further complements its first part by proposing a methodology for correcting these errors and finally presents how SHACL relation constraints can be learnt.

The targeted problem is very important and interesting. The paper is very well written and very well organized. The problem is introduced very clearly and thoroughly with examples. The introduced approaches are very clearly explained. Thorough experiments have been conducted with very extensive related work and comparison with the state-of-the-art approaches.

I would like to add some of the studies to the related work of this paper. For example in rule based approaches there is an evolution of AMIE [1],
another approach for completing RDF Knowledge Bases based on association rule mining was introduced in [2]. I would like to see one or two line comparison for them.

One of my only concerns is that the first part i.e., error detection has been published in [3], which authors also mentioned and there is almost no difference, same for the error detection which is proposed in [4] and not much difference can be seen. However, starting from learning SHACL relation constraints the part is new to the paper.

It would be nice to know why there is a huge difference in the numbers in Table 4 for example other than PaTyBRED and PaBRED all the numbers range 0.0 to 0.4 for sembib.

On page 8 does p = r_1 o ... o r_n appear as a role/relation composition operator?

More detailed comments:

In the introduction add the references to DBpedia, NELL, Wikidata and YAGO when mentioned for the first time.

Typos:

page 2:
which a member of --> which is a member of
two classes, that is --> i.e.,

page 3:
sake fo completeness --> sake of ....
considered in this course of this work --> considered in the course ...

page 4:
erroneous relatoins --> ... relations
improving it at as a consequence --> improving it as a consequence

page 9:

Table 2 depicts provide --> Table 2 depicts
There is formatting error on spouse(Trump, Melania)

page 18:

used by to categorize --> used to categorize

[1] https://link.springer.com/article/10.1007%2Fs00778-015-0394-1
[2] http://ijcai.org/Proceedings/15/Papers/121.pdf
[3] http://www.heikopaulheim.com/docs/kcap2017.pdf
[4] http://ceur-ws.org/Vol-2065/paper12.pdf

The paper investigates the problem of error detection in relation assertions of knowledge graphs, where manual curation is prohibitive due to the sizes of the involved knowledge graphs. In particular, the paper addresses three research questions (RQ) with three different approaches.

RQ1 addresses the issue of how incorrect assertions can effectively be detected. For this task the authors have developed the PaTyBRED approach. Here features are derived from type information on the subject and the object. Additional features are derived from the path analysis (PRA). Different classifiers are trained on those features to predict if an assertion is incorrect.

RQ2 addresses the question of how the error detection process can be analyzed and integrated into the knowledge graph. In PaTyBRED To SHACL, logical expressions are extracted by learning of decision trees which are converted into SHACL constraints. For validating the results of error detection, a user can now inspect a small number of patterns instead of a large number of individual mistakes.

RQ3 investigates how errors in the KG can be automatically corrected by using PaTyBRED. In particular the issue of errors resulting from confusion of entities is addressed by a procedure, the authors named CoCKG

The paper contains valuable work and should be published. The addressed research questions are of great practical relevance. The experimental results are mostly convincing.

My main complaint is about the readability of the paper which could be improved greatly. The writing is not bad or inconsistent, but sometimes a bit long winding. First, some sections can be shortened and better structured. Second, the author’s define very clearly the three research questions (a great idea) but then rarely refer to them. The latter would make the structure of the paper much more easily accessible. Also, I would recommend joining Sections 8 and 9 into one section with two subsections. Similarly, 6/7 and 4/5. Finally, since PaTyBRED is so central to the paper, it needs to be described more precisely. The authors should consider also describing PRA and SDValidate in more detail. In general, the authors should try to make the paper more self-contained.

Some detailed comments
There are some spelling errors / grammatical errors in the paper
(fo instead of of, “relations” has been misspelled, add(s) up, given “by” a knowledge graph)

Abstract
- The first half of the abstract does not read very well. Please improve. The abstract should be improved in general

Chapter 1
- Can be shortened; I like the clear statements of RQ1-RQ3
- Explain what IRI stands for
- An overview over the paper at the end of the introduction: "Section xxx discusses yyy" might help

Chapter 2
- It describes the problem and introduces the three research questions RQ1, RQ2, and RQ3.
- Page 4: Center left column: … higher or lower …. (why both options?)

Chapter 3
- It covers related work
- It is very long and should be shortened
- On page 5 second columns: you mention inverse relations. Explain what they are and be more detailed on the consequences.

Chapter 4
- This addresses RQ1: state this clearly
- PaTyBRED should be described more clearly (more structured); also PRA and SDValidate, on which it is based, should be explained in more detail.
- In contrast: other discussions can be shortened
- Section 4 is very long but lacks a bit clarity; fewer words, better structure, and more precise descriptions! It should be more self-contained
- In general: better structured and more concise definitions; then the discussion!

Chapter 5
- This addresses RQ1: state this clearly
- Chapters 4 and 5 should be joined
- 5.1 is repetitive and should be stream-lined
- Page 10 top: ….1% of noise …: define what this means
- Second column on page to top: define “silver standard”
- Page 12 on top: sub- and superscripts on PatYBRED are difficult to read
- I have no idea how to interpret Fig 1
- TyBRED is mentioned in the left column of page 12 but then only defined later in the right column (bad style)
- Similar for PaBRED
- 5.4: please be more clear how KGC is used for error detection: also elaborate on the bad performance

Chapter 6:
- This addresses RQ3: state this clearly
- Seems a bit heuristic and unrelated to the rest of the paper, but since it contributes to the larger quest, ok

Chapter 7:
- This addresses RQ3: state this clearly
- Chapters 6 and 7 should be joined

Chapter 8:
- This addresses RQ2: state this clearly

Chapter 9:
- This addresses RQ2: state this clearly
- Chapters 8 and 9 should be joined
- Comparison with only one other method (SSI) which is not really explained is a bit weak
- I would expect a more detailed discussion is this more of an anecdotic result or do you expect that this is a serious competitor to other ILP approaches
- Please discuss in more detail, why an approach based on error detection has important advantages