A Survey On Knowledge-Aware News Recommender Systems

This paper provides an overview of the current state-of-the-art in knowledge-aware news recommendation. This is a particularly challenging domain for recommendation in general, due to the rapidly changing nature of content, and of user interests. The aim of the models discussed is to include structured knowledge (such as that from a knowledge base), to enrich the textual content of the news, and to alleviate some of these issues.

## Main concerns

Recommender systems are not my main area of research, so I have a limited view of how complete the paper is. Nevertheless, it seems thoroughly researched, and presented with attention to detail.

My main concern is the lack of a strict methodology for finding relevant papers. The authors claim on page 2, to provide a comprehensive review, but no discussion seems to be provided for how papers were selected for potential inclusion, and what criteria were used. A standard methodology, like starting from a set of DBLP search keywords to find a set of seed papers and following citations to a certain depth, would help to convince the reader that every effort was made to find relevant work. Since such a methodology can help to surface relevant work that may not have received the attention it deserves when it was published, I think this is an especially worthwhile approach.

While I agree with the authors' approach of highlight only a selection of representative papers to illustrate the basic idea of each approach, I do feel that a comprehensive review ought to include all relevant papers. They need not all be discussed in detail, but they should all be cited (perhaps the authors did follow this approach, the citation list certainly seems long enough, but the way it is written down no makes it seem that a fairly arbitrary selection was made).

Finally, the paper lacks somewhat in a strong message. Perhaps there is simply no such message to be found, but I would expect that a higher degree of synthesis can be achieved. I think it should be possible to answer questions like "Do knowledge-aware RSs actually solve the fundamental problems highlighted in the introduction, and how?" or "What is currently the most promising of the directions proposed, and what are the pros and cons of each?" or "Is knowledge-aware news recommendation currently a promising technology for production systems?". Section 7 covers some of this ground to some extent, but it currently reads more like a lengthy brainstorm, somewhat disconnected from what came before than a strong conclusion, synthesizing and extrapolating the work that has been done.

I believe these issues can be easily solved with a modest amount of extra work, so I recommend an accept with minor revisions.

## Other suggestions for improvement

* In Section 2, the problems that make news recommendation challenging are explained, but the case is never explicitly made for why and _how_ the inclusion of structured knowledge can help solve these problems. for instance, the fact that users do no make accounts, and do not leave explicit feedback, seem at first sight to be entirely unrelated to whether or not the system has access to DBPedia. I can work out how these might relate, but it would be better if the paper made the case explicitly, ideally with a skeptical reader in mind (since many in the Recommender System community may need some persuading).

* It seems that almost all methods discussed require some form of entity linking/named entity recognition to map the concepts discussed in the news articles to the structured knowledge. As far as I can tell, this step is never explicitly discussed. It seems to me that this is a rather important dimension, and choices made here may lead to great differences in performance. There may be other steps like these (general text pre-processing for instance) that any knowledge-aware news recommender requires, that may differ between systems.

* Some statements,in particular in sections 6 and 7 should be made more strongly. For example p34l2: "It can be said that ..." If the authors of a survey are not going to say this, then who? I suggest that the authors be a little more bold and state explicitly that the field lacks reproducibility and strong evaluation. I think they have made that case well enough not to couch their words.

* The paper uses passive tense a little too often. For instance on p10l26 in the sentence reading "These systems are analysed..." it is genuinely unclear whether this analysing is done by the authors, or by the community in general. Active tense solves such problems.

* Some sections could be better structured and read a bit more like a loose assortment of ideas than a coherent narrative. For instance:
** In Section 2, the paragraph starting on p3l40, starts by discussing the difference between long and short-term preferences, and then jumps to the difficulty of tracking users without an account and then jumps to the lack of explicit feedback. These are related concerns, but they are better discussed in turn, each in their own paragraph.
** Section 7.1 is a really important section, containing one of the main takeaways, but it reads like a long list of loose recommendations. I think this section can be made stronger by introducing a little more structure. For instance, stating up front that the authors make N specific recommendations to improve evaluation, and then including these in a numbered list, with the topic sentences for each in bold. It may even be worthwhile to remove some less important recommendations to better emphasize the most important ones.

* Occasionally, references to the origin of methods outside the domain of knowledge-aware RSs is missing. For instance, the Bing similarity is presumably a variant of the Google distance and related methods. In this case it would be good to reference that method briefly to position the research better. Semantic similarity based on graph distance is another example.

## Typos and small mistakes

* "utilize" - > "use"
* "X is comprised of Y" -> "X comprises Y" or "X is composed of Y"
* "If several users read the same two news ..." -> "If several users read the same two news articles ..."
* "This phenomena" -> "This phenomenon"
* "A stricter criteria" - > "A stricter criterion"
* The mathematical typesetting need a little bit more work. For instance:
** Any names consisting of multiple characters ("agg", "softmax") should be set in text mode.
** Angle brackets should be set with \langle and \rangle rather than < and >
* The four different types of approach are hard to pick out as subsections. Perhaps it would be better to make each their own section.

The authors propose a comprehensive literature review of knowledge-aware news recommender systems.
They coarsely partition the contributions according to the underlying knowledge-aware recommendation model into four categories: vector space models, semantic similarity-based, distance-based, and knowledge graph embeddings.

The paper starts by focusing on the specific challenges of news recommendation.
This section is very interesting since it highlights how the news domain exhibits peculiar characteristics and poses additional challenges to the recommendation task.

Subsequently, the paper continues with a brief literature review.
The section has two pillars: news recommendation systems and Knowledge-Aware Recommender Systems.

For what regards the Knowledge-aware Recommender Systems subsection, I have found it quite limited.
In fact, it seems partially based on "A Survey on Knowledge Graph-Based Recommender Systems" that is limited for the following reasons. Several papers on Knowledge-aware Recommender Systems have been presented at ISWC in the last few years. The same consideration can be done for ESWC. There are works in the 2020 editions on recommender systems exploiting knowledge graphs to the best of my knowledge. Going backward, ESWC hosted a challenge on feeding Recommender Systems with Linked Data. A series of Workshops on Knowledge-aware Recommender Systems (hosted at the premier conference on Recommender Systems, Recsys, and CIKM) is not mentioned, nor its papers analyzed. To my knowledge, some of the contributions there focus on news recommendation systems.

After this section, the authors introduce some notation to ease understanding the remainder of the survey and provide the interpretation function to understand their categorization of the contributions.
I particularly appreciated Section 4.2.5, where the authors present the benefits of Knowledge-aware Recommender Systems for the news recommendation.

Next, Section 5 focuses on describing and detailing the knowledge-aware recommendation techniques that are reviewed.
Although I like the representation of Table 2, it makes quite clear that factorization models (that are in the state-of-art of Recommender Systems) are absent. Even here, to my knowledge, matrix factorization and factorization machines have been adapted to work with Knowledge graphs to build hybrid Knowledge-aware Recommendation engines.
The remainder of the section is really interesting and clearly describes the methods.

The sixth section also provides very valuable information and takes one more step towards the Reproducibility of experiments.
In this direction, an interesting paper has been published on this topic: "A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research" that outlines what a paper must detail to make the experiments reproducible.

Overall, the work presents several positive points and proposes a comprehensive picture of knowledge-aware recommender systems.
However, some important pieces of literature are missing, along with the entire recommendation family of factorization models that have been widely used to create interpretable models with knowledge graphs.

In an effort to survey techniques that utilize knowledge injection for news recommender systems, the authors of this article propose a new categorization: building on vector space models, semantic similarities, distance and knowledge graph embeddings. In addition to this taxonomy, a review of evaluation methods and open challenges is proposed.

From this survey the method of how the survey was conducted is missing. This should include the search strategy and platforms included in the search as well as the strategy and criteria to include/exclude papers, how many people were involved in this decision and the entire process for determining the final set of papers/approaches, e.g. PRISMA. This information should definitely be included in the survey. Furthermore, a new taxonomy for classifying approaches should not only be introduced but it should be explained in detail why these

The taxonomy currently proposed is not compared to existing taxonomies and not justified. To me personally, it is highly problematic since the categories are not mutually exclusive and it would be very hard to classify a novel technique based on this taxonomy. For instance, semantic similarity is a crucial aspect of utilizing vector space models in this context. In fact, it seems that all approaches classified as VSM apply vector-based variants of TF-IDF, which might be a better category than the generic VSM overlapping with the other categories. Then semantic similarity also depends on VSMs and the description of this category does not make it clear which approaches to classify into the first and into the second category. Distance is basically a category taking structural information into account, which should be made more explicit in the naming and then the categorized approaches might need to be re-evaluated regarding their classification. For instance, as the definition of categories stands now OBSM could be classified in any of the first three categories. In terms of category four, the name knowledge graph embeddings is considerably too restrictive for the approaches presented in this section. In general, I recommend revising the taxonomy as well as its justification and explanation, especially the delimitation between individual categories.

This recommendation is strongly supported by the imbalance of approaches represented for each category, with considerably more approaches in the fourth KGE category. Additionally, I am wondering which KGE methods take KGE beyond node embeddings as stated in the introduction to this section into account to consider structural information, since KGEs are not restricted to node embeddings. This further calls for a more fine-grained or a more balanced and improved categorization.

In Section 2 the challenges of News Recommendation are discussed, however, two important challenges are seemingly left out: fake news and bias (e.g. political polarization). Is there a specific reason that these rather central challenges nowadays are not discussed? If so, this should probably be included in the survey.

While I understand the necessity to realistically limit approaches covered by a survey, I am wondering whether it would not be interesting to provide one section on knowledge-aware recommender systems in general, which might provide insights and inspiration to news recommender systems. In spite of the challenges specific to the news domain and the focus of the survey, it might still be interesting to provide a general, broad overview of knowledge injection methods to recommender systems.

In terms of evaluation recommendations, shouldn't a standard benchmark dataset in this context also contain knowledge graphs and similar resources to truly allow for a full comparison of knowledge-aware news recommender techniques? Wasn't the manual creation of KGs without publishing them one of the problems for reproducibility? In addition, more specific future research direction also from the point of view of techniques would be interesting rather than generic neural-symbolic challenges such as scalability and explainability. It would be interesting to see a more extensive discussion on challenges and future directions specific to knowledge-aware news recommender systems.

Missing references:
The references included are comprehensive and timely. One reference that could potentially be added to compare the identified challenges in this paper to other works, could be:
- Shao, B., Li, X., & Bian, G. (2020). A survey of research hotspots and frontier trends of recommendation systems from the perspective of knowledge graph. Expert Systems with Applications, 113764.

Minor comments:
page 3, 38 he will be less likely => is "he" still the appropriate pronoun to refer to an anonymous person? Should not a more inclusive pronoun be preferred?
p. 7, 2 closes a parenthesis that has not been opened before: [99]) -> ( missing
p. 7, 28 each term in WordNet => "term" is a domain-specific designation; most words in WordNet are not domain-specific
p. 8, 27 facilitate readers => help/support
Table 2: While the abbreviation "H" for Hybrid is introduced in the caption, it is not utilized in the table itself
p. 10, 22 comprises of => consists of
p. 10, 37 based on the vector space model => based on a vector space model
p. 15, 33 value meant capture => meant to?
Figure 3 is considerably too small
p. 21, 20 a entity => an entity
p. 22, 23 fine-grained DKN with self-attention => ??? (improved? extended?)
p. 23, 26 and end-to-end => an
p. 29, 50 semantic aware => semantic-aware
p. 35, 3 ensure that models are not only