Ontology Embeddings with ontowalk2vec: an Application to UI Personalisation

Tracking #: 2692-3906

Authors: 
Blerina Gkotse
Pierre Jouvelot
Federico Ravotti

Responsible editor: 
Guest Editors DeepL4KGs 2021

Submission type: 
Full Paper
Abstract: 
Within software applications, user experience is greatly improved when user interface (UI) personalisation is possible, and even more so when recommender systems can help users find the set of settings best suited for their skills and goals. In this paper, we suggest that such recommender systems should be based on ontologies dedicated to describing both software traits and user preferences, an example of which is the Ontology-based Web Application Generation ontology (OWAO) that specifies what web applications and their UI are. The key scientific contribution of our approach is ontowalk2vec, an algorithm that maps instances of ontologies to feature vectors (embeddings) that can be later on used for classification purposes, a process inherent to recommender systems. In addition to OWAO, we validate ontowalk2vec on two other significant ontologies, namely MUTAG and DBpedia, where we demonstrate it outperforms the existing techniques. We finally discuss how using ontowalk2vec on OWAO can form the basis of personalised UI recommender systems, stressing, in particular, the importance of properly setting the many hyperparameters that typically characterise embedding-generation algorithms.
Full PDF Version: 
Tags: 
Reviewed

Decision/Status: 
Reject

Solicited Reviews:
Click to Expand/Collapse
Review #1
By Heiko Paulheim submitted on 17/Mar/2021
Suggestion:
Major Revision
Review Comment:

The authors introduce ontowalk2vec, an algorithm which combines node2vec and RDF2vec. They show experiments on a few benchmark datasets and an own use case of UI personalization, and claim that their approach outperforms both node2vec and RDF2vec. The code and data are available.

While the idea of combining the advantages of node2vec and RDF2vec, I have a few concerns about this paper, concerning both the approach itself as well as the evaluations conducted.

First, the way the approach is formulated either misses a few details, or the approach has a few shortcomings. node2vec operates on an undirected graph, i.e., it ignores the edge labels and generates a sequence of nodes. Considering the following graph:

locatedIn(Berlin,Germany)
headOfState(Germany,AngelaMerkel)
would lead to the sequence
Berlin -> Germany -> AngelaMerkel
extracted using node2vec. In contrast, RDF2vec would use the edge labels and extract
Berlin -> locatedIn -> Germany -> headOfState -> AngelaMerkel

These two sequences encode the same information, however, they constitute different training instances when training the skip-gram model. For example, when training the SG model for Germany, the node2vec sequence yields w(t-1)=Berlin, w(t+1)=Germany as its training target, while the RDF2vec sequence yields w(t-2)=Berlin, w(t-1)=locatedIn, w(t+1)=headOfState, w(t+2)=AngelaMerkel. It is unclear to me how such different training signals should improve the embedding vector for the instance Germany. More general, the RDF2vec sequences will always have properties in w(t-1) and w(t+1), while the node2vec sequences will have instances.

Given that observation, it is unclear to me how the direct combination of walks would have an advantage here. I would rather expect an advantage from training two embeddings separately and concatenating the embedding vectors. At least, that concatenation should be used as a baseline for combining RDF2vec and node2vec.

There are also some statements in the description of ontowalk2vec that I find doubtful. The first is that the "node2vec model focuses on the structural part of the ontology", while "RDF2vec focuses on the RDF triples, which include the instances" - if they both operate on the same graph comprising ontology and instances, they should both learn representations for both alike. It is unclear to me why one should rather focus on the T-box, while the other focuses on the A-box.

The second remark is that "CBOW is not suitable for our purpose". The experiments in the original RDF2vec paper show that CBOW is sometimes (not always) a few percentage points worse than SG, but I would not go as far as calling it "not suitable". We could also phrase it differently: the direction of relations is often a convention which is arbitrary in knowledge graphs (in the above example, we could also use headOfState with the person in the subject and the state in the object position), and CBOW cancels out the pseudo-signal which this arbitrary convention introduces.

My other concerns are related to the evaluations. First of all, in all places where numbers for RDF2vec are shown, those numbers are significantly lower than in the original RDF2vec paper. According to the original numbers, ontowalk2vec is significantly superior to RDF2vec. The authors should explain those differences.

Second, I am not sure what the bottom line of the confusion matrices is. The authors claim that ontowalk2vec yields better true negative and false positive rates than its competitors. That is true, but on the other hand RDF2vec yields better true positive and false negative rates. Is that a random artificat, or is there something in the design of ontowalk2vec that makes it focus more on the negative class?

Third, the evaluation on OWAO left a bit puzzled. The authors report near-perfect results on a dataset with synthetic instances. First of all, no train/test split is reported in the paper (but apparently, there is a train and test split when looking at the data). If not, I would simply assume that they can perfectly learn the pattern of the synthesis method, but this does not provide any insights on how the approach would work in the wild. Actually, the experiment does not prove that ontowalk2vec is suitable for UI personalization; it rather shows that it is suitable to learn the behavior of a synthetic data generator.

Overall, my impression is that the paper has too many shortcomings to be publishable in its current form. The depiction of the algorithm should be clearer, vector concatenation should be used as a baseline, the deviations from the original RDF2vec results deserve an explanation, and the evaluation on the UI personalization task needs to be conducted in a more sound way.

Review #2
Anonymous submitted on 27/Jun/2021
Suggestion:
Reject
Review Comment:

In this paper, the authors suggest to use a combination of RDF2Vec and node2vec to create embedding vectors, which can later be used for UI personalisation.

The paper is very understandable and well written. A fair amount of detail is provided in the paper and the source code is made available for review.

My main concern with this paper is that it does not use the latest developments in the methods it is deriving from. Specifically, when combining with other techniques to extract walks in a different way, one would expect a comparison with https://madoc.bib.uni-mannheim.de/43114/1/164.pdf , where different ways of assigning probabilities to edges are used. Similarly, https://link.springer.com/chapter/10.1007/978-3-319-68288-4_12 also applied the same probability techniques.

Also, it the way RDF2Vec is applied seems questionable. Exact used parameters for the experiments in section 5.1 and 5.2 are missing, which makes it difficult to know what has happened in the experiment. Besides, the following quote worries me: "In the pyRDF2Vec python implementation of RDF2-Vec [22], SPARQL queries are used for extracting datafrom DBpedia where the subject of the RDF triple isthe instance for which a feature vector is sought. After some preliminary experiments, it was observed thatnode2vec is not able to compute meaningful vectors from such data, since the random walks are too short." I do not know the motives behind this implementation, but if that only used triples in which the entity is contained, them the implementation does not reflect what happened in the original RDF2Vec. Because of the way word2vec works, it is essential to also include longer sequences. I might recall wrong, but It think it was even shown in the RD2Vec papers that longer sequences give better results on several of the tasks.

This gets us to one more point of doubt: the experiments conducted are only on a small number of tasks and only use two classification methods. This is just too little to make overarching conclusions about he quality of the methods.

Finally, I really wonder whether the overall approach is really suitable for this kind of problem. While it seems feasible to create semi-supervised embeddings and then use them in a recommendation system, it seems more promising to directly train a system for this. For example this work would point in that direction: https://arxiv.org/abs/1706.02263

Some more minor issues:

* Section 3 has too much details. Most of what is described here is pretty standard and this could be summarized in one column or less. Citation 15-19 seem unnecessary as well.

* I am confused about why you call RDF2Vec BFS. In fact, the nature of the random walks is DFS, as far as I understand them.

* You claim CBOW is not suitable for your work. But, given you only use very short walks, the argument you use is invalidated. Hence, the only way to show that CBOW is not suitable is by providing experimental evidence.

* You make the statement that "By comparing feature vectors using the cosine similarity metric, one can get an approximation of the proximity and relatedness of the corresponding data instances." While this is sometimes true, that is not generally accepted and certainly depends on the embedding technique.

* In the confusion matrix in table 3, you do highlight your results as better, even if the differences are not statistically significant.

* Why is there no hyperparameter optimization for RDF2Vec and node2vec, similar to what you did for ontowalk2vec?

* For he UI experiment, it seems odd that you use preferred as a binary label. I am imagining a setup preferred by one user, vs one preferred by 1000 users. Getting the one with 1000 right seems much more important here. Actually, I would see this more as a regression problem rather than a binary classification.

Review #3
Anonymous submitted on 28/Jun/2021
Suggestion:
Reject
Review Comment:

The current study focuses on improving the existing Knowledge Graph Embedding method RDF2Vec by augmenting the features with the help of node2vec. The proposed approach is called Onto2Walk. The authors then show how using ontowalk2vec on OWAO can form the basis of personalized UI recommender systems.

First of all, it is not at all motivated why the authors are using Node2Vec in case if they want to embed the ontology? Node2Vec works on undirected graphs. It seems like an ad hoc combination of RDF2Vec and Node2Vec for improving the results.

Also the focus as well as the main contribution is unclear since it was hard to understand what the authors want to focus on, i.e, proposing a new algorithm (which is not new) or the application to recommender systems.

There are many unnecessary details in section 3 such as details about cosine similarity, f1 score, etc. Also the details of hyperparameters in section 6.1.

The connection between UI Personalization in Web apps and Ontowalk2vec is not clear, since it does not seem to be the right choice of method for the problem that authors are trying to discuss.

Section 5.3.1 is not actually providing a definition.

The used depth is very low, i.e., 1, which means the algorithm is operating on the level of a triple without considering the contextual information of a node which is one of the strengths of the walk-based algorithms. In such cases, other embedding approaches could also be used such as TransE [1], etc.

[1] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston and O. Yakhnenko, Translating Embeddings for Modeling MultiRelational Data, NIPS, 2013.