WYSIWYM -- Integrated Visualization, Exploration and Authoring of Un-structured and Semantic Content

Tracking #: 479-1674

Authors: 
Ali Khalili
Sören Auer

Responsible editor: 
Guest editors Semantic Web Interfaces

Submission type: 
Full Paper
Abstract: 
The Semantic Web and Linked Data gained traction in the last years. However, the majority of information still is contained in unstructured documents. This can also not be expected to change, since text, images and videos are the natural way how humans interact with information. Semantic structuring on the other hand enables the (semi-)automatic integration, repurposing, rearrangement of information. NLP technologies and formalisms for the integrated representation of unstructured and semantic content (such as RDFa and Microdata) aim at bridging this semantic gap. However, in order for humans to truly benefit from this integration, we need ways to author, visualize and explore unstructured and semantic information in a holistic manner. In this paper, we present the WYSIWYM (What You See is What You Mean) concept, which addresses this issue and formalizes the binding between semantic representation models and UI elements for authoring, visualizing and exploration. With RDFaCE and Pharmer we present and evaluate two complementary showcases implementing the WYSIWYM concept for different application domains.
Full PDF Version: 
Tags: 
Reviewed

Decision/Status: 
Major Revision

Solicited Reviews:
Click to Expand/Collapse
Review #1
Anonymous submitted on 20/Jun/2013
Suggestion:
Reject
Review Comment:

This article discusses how to author/visualize/explore the semantic information from the perspective of user interfaces. It first presents a formalization of WYSIWYM(What You See is What You Mean) including model, binding, interface, etc. Then, some details about WYSIWYM such as different models and different UI techniques for visualization/exploration/authoring are introduced. The possible bindings between models and UIs, i.e., how different model elements could be implemented using various UI techniques, are also discussed. As a proof of the WYSIWYM concept, two implementations (RDFaCe and Pharmer) are described and evaluated.

Although the topic of this article is interesting and it falls into the topics of interest for this special issue, the contribution of this work is weak.
(1) First, the formalization of WYSIWYM is too simple. In this article, a WYSIWYM model is simply represented as a quintuple consists of data model, visualization technique, exploration technique, authoring technique, and some helper components. A more sophisticated model is needed.
(2) Second, the summary of possible bindings between different representation models and UIs shown in Figure 4 needs more clarification. For instance, how different binding configurations are used to bind the models and UIs. Furthermore, what is the benefit of this summary? How to utilize it?
(3) Regarding the implementation and evaluation, it seems the RDFaCE is an extension of the existing editor(TinyMCE) with some capabilities of asserting subject types/relationships with the help of recommendations from external resources such as DBPedia. These newly added capabilities could be achieved by a lot of existing editors/tools such as the WordPress RDFa markup plugins and the RDF/RDFa functionality in Drupal 7. Although this article gives a small-scale evaluation involving 16 users, no comparison evaluation between RDFaC and other RDFa authoring tools has been included. It is hard to convince people to believe the advantages of this approach without a comprehensive evaluation.

Review #2
Anonymous submitted on 18/Jul/2013
Suggestion:
Minor Revision
Review Comment:

The paper introduces the WYSIWYM concept to bridge the gap between unstructured and semantic content. In their proposed WYSIWYM concept, the authors
the widely used semantic representation models ranging from tree-based to graph-based and even hypergraph-based. They also provide a comprehensive introduction
of the visualization, exploration and authoring. Then the paper discusses the bindings as well as the helper components like recommendation and multilinguality issues.
Based on the proposed concept, they implemented two applications RDFaCE and Pharmer (for electronic prescriptions). The user evaluation results show the usability of
their approaches. The paper is well-written and well-organized. The presented two applications are very impressive. So I suggest to accept the paper.
The only minor issues are:
1. I do not see how the performance of the helper components influence the user experience
2. I would like to see more details in the experiments especially whether the selection of different bindings between representation model and visualization will affect the user experience.

Review #3
By Heiko Hornung submitted on 25/Sep/2013
Suggestion:
Minor Revision
Review Comment:

The authors point out the "semantic gap" between unstructured and structured Web content They point out, that despite NLP technologies, formalisms, etc., the gap continues to exist and people still do not "truly" benefit from a possible integration. The authors believe one reason of this is a lack of "ways" to author, visualize and explore (semantically) structured and unstructured content in an integrated manner. The authors then describe an approach based on the WYSIWYM metaphor that addresses this issue. They present two cases that implemented the proposed concept and that provide evidence that the approach results in systems with good usability.

Originality and significance of results
---------------------------------------

The paper is interesting and original in the sense of proposing a formalization of the authors' WYSIWYM concept and a mapping of semantic model representation elements to UI techniques. It is relevant and adequate for this special issue since it is addressing questions that are related to both semantic web and user interaction. The proposed concept is an interesting approach to enable developers, designers and domain experts to exploit/apply Semantic Web concepts and techniques in order to create user interfaces that facilitate the creation of and interaction with integrated unstructured and structured content.

In the following I will discuss some points that in my opinion need further clarification.

Definition of Terms and Scope
-----------------------------

Regarding the used vocabulary the following terms appear in the paper: semantic content, structured content, semantic information, semantically enriched documents, semantic documents, semantically structured data. Are these all used synonymously or are there any relevant differences? I suggest clarifying the differences and reducing the number of synonyms. Furthermore, I would like a definition of what you mean with "semantic content" and to what kind of "content" it refers. You mention "text, images and videos", however, I got the impression that this does maybe not include every form of visual, auditory, static, animated, or interactive content. Two of the most important references you used for informing your approach [4, 10] are about linked data and textual content. Your two showcases seem to use only static textual content. To give and example of different kinds of content and different kinds of user interface techniques, I think this (http://worrydream.com/VisualizingEdgeWeights/ - bar chart and graph at the bottom of the page) might be more or less covered by Fig. 4. Is this (http://worrydream.com/Tangle/ - page bottom) also covered (I'm aware that the content of the particular example might be out of scope, but one could easily conceive of a domain like historical government budget planning and projection, etc. where LD from different providers might be analyzed/"understood" ("explored" in your terms?) in a similar way)? If you focus on (static) textual content, you should state this more explicitly, including in title, keywords and abstract. Otherwise

The impression I mentioned in the previous paragraph might be due to the fact that my understanding of "semantic content" differs from yours and that your table 4 is based on a literature review (in the area of semantic web/linked data?). Hence my suggestion to explain these terms and define your scope more explicitly. As it is, Fig. 4 seems like a catalog of the state of the art in an area that this special issue seeks to expand. If this is within the scope of your paper, it would be interesting to see a discussion of the gap between Fig. 4 and and "interaction with (semantic) data" how you think it could/should be (including already existing examples outside the area of the Semantic Web). If UI and user interaction techniques like in the two links above are within the scope of the paper, I think it would be pertinent to include "data/information visualization" into the body of "related work" and examine the "gaps" in Figure 4 that would open. On a side note, since you mention knowledge visualization/representation, depending on your definition of "knowledge" you might deem Data/InfoViz irrelevant for this paper. if so, please make this and the reason(s) explicit in the text.

WYSIWYM
-------

Regarding your conceptualization of WYSIWYM, who are the two "You"s? Is the user who visualizes/explores always the one who is also the author of (or who at least could further edit/annotate/...) the content? If there are two users (something like "What You See Is What Someone Else Might Have Meant), I think the acronym might be misleading. What kind of users do you target (only domain experts or any interested user)? Your two showcases include domain experts. What does "mean" mean to you? Does it refer to human cognitive processes? If the two "You"s are different, is it important to you how different people "interpret"/"understand" authored content and that the first and the second "You" gain some shared understanding? Or is it only important that the author (thinks s/he) expressed meaning (or intentions?) in a better way? How would you evaluate these points? Or is the "Mean" in WYSIWYM only an allusion to using _Semantic_ Web techniques? I would be interested in seeing these points made more explicit in the paper.

In the first paragraph of section 3 you claim that the "formalization serves the purpose of providing a terminology for software engineers, user interface and domain experts to communicate efficiently and effectively", however you don't explore this point further. Are "domain experts" any experts of any domain, or do you think that they at least require some basic Semantic Web skills? What about users if there are users that are not domain experts? You then state that your WYSIWYM concept "allows to evaluate and classify existing user interfaces according to the conceptual model in a defined way". Please elaborate/outline what kind of evaluations you think of.

UI categories and techniques
----------------------------

Regarding subsections 3.2, 3.3, and 3.4, where do the UI categories and techniques come from? I assume they come from the authors' literature review (of textual SW/LD content?). If so, this should be made explicit (the phrases "We divide existing techniques for visualization of knowledge" (p.7) and "Although we deem the bindings to be fairly complete, new UI elements might be developed" (p.10) suggest that these depict the state of the art in knowledge representation/visualization (I consider Data/InfoViz a part of this) in general and not knowledge(?) representation/visualization in the examined body of literature). If you describe the categories as general categories of "'knowledge' visualization/exploration/authoring", you need to either cite accepted references or explain/detail some items, e.g.:
* in design, "text formatting" can be a form of "framing and segmentation" (think of Gestalt principles/laws)
* regarding accessibility, e.g. the equivalent of the visualization->highlighting techniques would be audio formatting techniques (changing of pitch, amplitude, speed, filtering, ... --> aural style sheets); since in design there seems to be a consensus that "accessibility" is not something that can be "plugged in" after everything else is finished, it might be worthwile exploring "(re)presentation" instead of/additionally to "visualization"
* what about animation, an important technique for illustrating time dependent data; in this context, one can also conceive of V4 (marking) of audio/video content (e.g. timed popups/transitions/...)?
* what about association by highlighting associated items or links/anchors?
* visualization vs exploration: what do the terms mean to you? is it something like a "passive consuming" vs "interaction" dichotomy? do you mean "exploration" in the sense of "understanding"/"reasoning" or in the sense of "navigating"/"interacting". If it's "understanding", then visualization is an integral part of it. Further indications that there is something amiss with these two categories: you describe X1 zooming as "see more detail or less", and put "detail view" as a visualization subcategory. The "Detail view" subcategory has V9 callouts which also appears as X4 callouts. X3 Bar layouts appears more like a visualization technique than an exploration technique.
* exploration and authoring don't have categories, which makes Fig. 4 less "elegant". Suggestions (depending on how you define the answer to the exploration/visualization question): a possible category for X1 and X4 is "detail view", "filtering" could be a category for X2, "alternative presentation" or "summary"/"synthesis" for X3; another technique that comes to mind and that doesn't appear in Fig. 4 is "drill down" (might be assigned to the "detail view" category)
* T7 and T8 are input modalities that apply to all the other categories/techniques
* methods vs techniques: some items are very specific and near the technical level (e.g. ribbon editing, where a more abstract concept would be contex-sensitive toolbar), while others appear to be rather methods than techniques (zooming is a method that could be implemented by the technique of magnifying an area (visual zoom) or by providing additional information("semantic" zoom))
* T8: "gesture (a.k.a. sign language)": gestures might be part of a sign language or not. I suggest you remove the "a.k.a. sign language": at the end of the paragraph, you write "The main problem with gestures is their high level of abstraction which makes it hard to assert concrete property values" which seems to describe a gesture that is not part of a sign language (e.g. the aleatory gestures of touchscreen interfaces).

You cite some problems/challenges for some of the categories, but not for others. One can probably come up with challenges for most of the categories. Is there a special reason why you point out challenges for specific categories?

Bindings between UI and Semantic Representation Model Elements
--------------------------------------------------------------

Regarding Fig. 4:
* What is the difference between partial and full binding? Does "partial binding" refer to what you found in literature or to what you deem "possible"/"conceivable" technically or in terms of design?
* Does the asterisk mean "there is a partial binding if the value is available in the text, otherwise there is no binding"? Could you provide a short example?
* Does the absence of Cx mean that there exist no configurations for the binding/a configuration is not applicable or that you didn't find any in literature? Is there always only one kind of configuration?
* Fig. 4 could benefit from using the labels established in subsections 3.1 and 3.2 (e.g. use "V1: ..." in the first row of the "UI techniques" column of Fig. 4).
* From a graph-theoretical point of view, it would be interesting to discuss whether e.g. the tree-based part of the table/figure can be expressed as a special case of the graph-based part.
* authoring and exploration contain only techniques, not categories (see above)
* There are at least two different shades of gray for "partial binding".

Usability Evaluation
--------------------

Regarding the RDFaCE evaluation in section 4, what kind of survey did you use? Is it a survey described in [12]? I must confess I don't know this book. Does the approach to usability differ substantially from other works in the area? The reason I'm asking is that in the Pharmer case you used the well established and validated SUS, i.e. we can interpret the results and know of eventual restrictions/limitations. If the survey you applied in the RDFaCE case is a non-validated or self-created questionnaire, some of the results might not be statistically significantly above "neutral/average" as you claim (aside from the fact that we can only draw conclusions for the universe of ISS-LOD 2011 summer school participants). In the Pharmer case, on the other hand, you have a good user mix and a task that could be considered a realistic task, thus the SUS results look more convincing.

Minor details regarding use of expressions/orthography/etc.
-----------------------------------------------------------

Besides the observations regarding vocabulary definition, the paper is easy to read. One point that hampered the reading flow and understanding when reading the paper for the first time is the formalization of the WYSIWYM concept in section 4. Fig. 4 is referenced on p.5 while only appearing on p.11. When on p. 5 for the first time, it is difficult to understand Fig. 4 and thus it doesn't contribute much to illustrating/clarifying the model. If you want to provide an illustration/clarification/instantiation, a suggestion is using short examples for the items that need clarifying.

* The authors use the words "holistic" and "integrated" to describe their approach to authoring, visualizing, and exploring information. I think "integrated" is more adequate: in my opinion some aspects are missing to call the approach "holistic", e.g. user experience.
* Parts of the abstract contain verbatim copies of the text, cf. especially the first sentences of the introduction. While I don't think that this should negatively affect the evaluation of a paper (contrary to many reviewers I know), I think paraphrasing instead of copying sections from the text can help the reader to better understand the issues addressed in the paper.
* On p. 2, in the left column, the authors write "we present *the* WYSIWYM concept". I think WYSIWYM is a metaphor with some strengths and weaknesses that is used very differently by different authors. The authors even present different usages of the metaphor, and one might even discuss semantic and conceptual differences between WYSIWYMean and WYSIWYMeant as presented on p.3. Thus I suggest to use something along the lines of "we present our [understanding of the] WYSIWYM concept" or "we present an approach {inspired by|using} the WYSIWYM metaphor".
* On p. 2, left column, you promise a "comprehensive survey of semantic representation elements [...]". As far as I can tell, the topic is presented in subsection 3.1 and revisted in Fig. 4. The expression "comprehensive survey" creates a different expectation for me.
* On p. 3, right column, you write "Visual mapping techniques (a.k.a. knowledge representation techniques)". Unless you can provide a respective definition of "knowledge", I don't think a.k.a. is a good choice here; maybe: "Visual mapping techniques are knowledge representation techniques that..."
* P.10, left column: "refereed" -> "referred"
* P.12, left column: "vacabularies"
* Reference 10 is incomplete.

Summary
-------

Overall, I like the paper, and I think it might make a valuable contribution to the Special Issue. Since I think that this Special Issue might attract readers from different fields, it is especially important to make explicit the used terms and the scope of the paper, as well as to pay attention to some details of section 3.