SVG Pictograms with Natural Language Based and Semantic Information

Keywords: pictogram , liguistic information

Kazunari ITO
Assistant Professor
Aoyama Gakuin University
Fuchinobe 5-10-1
Sagamihara 229-8558
Japan
kaz@it.aoyama.ac.jp

Biography

Kaz received his Ph.D. Degree in Computer Science from Keio University in 2005. He is currently an Assistant Professor in the Department of Integrated Information Technology, College of Science and Engineering, Aoyama Gakuin University. His research interests include semantic web, natural language processing, multimedia information processing, e-learning and application of pictograms.

Motohiro Matsuda
Student
Aoyama Gakuin University
matsuda@sw.it.aoyama.ac.jp

Martin J. Dürst
Associate Professor
Aoyama Gakuin University
duerst@it.aoyama.ac.jp

Kōiti HASIDA

National Institute of Advanced Industrial Science and Technology
10F 1-18-13 Sotokanda
Chiyoda-ku 101-0021
Japan
hasida.k@aist.go.jp


Table of Contents


1. Introduction
2. Background
3. Pictogram Authoring
4. Prototype
5. Markup for Linguistic Information
6. Two kinds of XML structured data
7. Conclusions
Bibliography

1. Introduction

Pictograms are simple graphics very well suited for being represented with SVG. However, until now, both our group as well as similar research efforts have used raster data formats exclusively. In this paper, we report on our effort to convert our collection of pictograms [PICT] to SVG, the advantages gained from this conversion, the introduction of a prototype system for pictogram authoring, and future benefits we are looking forward to.

2. Background

A pictogram is an expressive form which can be understood intuitively without a grammar. All pictograms are composed of one or more components, starting with primitives such as circles, rectangles, lines, and curves. There are numerous reports and case studies on the successful use of pictograms in the fields of learning assistance for infants and livelihood assistance for persons with disabilities[NHK].

In Japan, design guidelines for pictograms have been established in 2005 as JIS T 0103 [JIS_T_0103]. These guidelines recommend that new pictograms be designed by consciously considering combinations of existing pictograms. As an example, Figure 1 shows the pictogram for a hospital given as an example in JIS T 0103. It can be understood as overlaying the pictogram for a cross (Figure 2) on the pictogram for a building (Figure 3), with the color of the cross changed to white.

Hospital.svg

Figure 1: Hospital

Cross.svg

Figure 2: Cross

Building.svg

Figure 3: Building

Our group has already published (see [PICT]) around 1000 pictograms in the GIF and PNG raster image formats (Figure 4), and a Java based application (Figure 5) to create new raster-based pictograms based on existing ones. Our research is focusing on expressing sentence-level concepts by arranging several pictograms in 2D space, and to study the relationship between 2D arrangements and semantic relationships[ITO][MATSUDA]. But we have found it very hard to proceed with our research using raster images.

imagelist.png

Figure 4: Pictogram List

picttool.png

Figure 5: Pictorial Authoring Tool

3. Pictogram Authoring

To support the creation of new pictograms and the collection of data, we have been researching pictogram authoring. In this paper, we use the term pictogram authoring for the creation of new pictograms by combining and modifying one or multiple existing pictograms. This means that we exclude drawing new basic pictograms from scratch. Pictograms can be modified in a number of ways: applying layout and form changes such as scaling, mirroring, and rotation, changing color, applying animation, and so on. As a result of experiments, we have established [FUJIMORI] that by applying such modifications, the range of concepts that can be expressed is greatly expanded, reusability is increased, new pictogram vocabulary can be created, issues of color blindness can be addressed, and the primary focus of a pictogram can be made easy to understand. Figure 6 shows a example.

School-hospital.svg

(This figure is an animation. Please reload the document if you cannot see a walking man, or use a viewing tool that supports SVG animation.)

Figure 6: I walk to a hospital from school.

4. Prototype

To explore the feasibility of using SVG for pictograms, we have created a web-based SVG pictogram authoring tool prototype. Figure 7 shows a screenshot. Using SVG has greatly simplified the implementation. The prototype works on any browser that supports SVG.

The prototype consists of four text areas on the right and a pictogram output area on the left. The topmost text area, labeled "Command", serves to input textual commands to create a pictogram. Commands are separated by vertical bars; each command places an existing pictogram according to a series of parameters. There are at most eight parameters, separated by a comma. The parameters represent label, left, top, width, height, angle of rotation, color, vertical mirroring and horizontal mirroring. The label parameter is a text string that identifies a previously registered pictogram. All other parameters can be omitted, in which case default values are used. For example, the following text command generates the pictogram in Figure 7:

School,0,150,100,100|Building_big,150,0,100,100|Cross,175,40,50,50,,FFFFFF|Walk,50,200,40,50
    

We have already registered about 1000 basic pictograms not only with English labels but also with Japanese labels, so the above command also can be written as:

学校,0,150,100,100|大きい建物,150,0,100,100|十字架,175,40,50,50,,FFFFFF|歩く,
    50,200,40,50

The text area labeled "Sentence" is used to input a piece of natural language text corresponding to the pictogram being created. For the above-mentioned command, this could be the sentence "I walk to a hospital from school". A remarkable point is that the registered sentence can be reused as a label in the text command to recall the newly created pictogram. The following text command makes sense after registration.

I walk to a hospital from school,100,100,200,200|Sun,0,0,100,100
    

The following sentence could describe the new pictogram: "I walk to a hospital from school on a sunny day".

Another way to use the Command area and the Sentence area is to input the label of an existing pictogram in the Command area and to input an alternative label in the same language or translation in a different language into the Sentence area. This works because a single label is the simplest command possible.

Furthermore, the SVG document generated by the text command can be modified manually in the text area labeled "SVG Source". As an example, a user can change the color of an object or add an animation effect to particular objects or shapes. The SVG Source area also makes it possible to create new pictograms from scratch. In the same way as for pictograms created with the Command area, pictograms created or modified in the SVG Source are can be recalled and reused with the associated label or sentence.

When the user pushes the "register" button to the right of the Sentence area to register SVG data and the corresponding natural language expression, the "URI" area displays a URI including a unique ID that can be used to access the registered SVG document. The point is that when one wants to get an ID for a pictogram, one needs to input a sentence which represent the pictogram. In this way, we encourage the association of metadata with the pictograms. Only the creator of a pictogram knows the ID and can embed the pictograms he created in any Web page.

screenshot.png

Figure 7: Screenshot of the prototype

5. Markup for Linguistic Information

To further progress towards the automatic creation of pictograms from natural language, we use linguistic information. Every English or Japanese sentence that is input is automatically marked up with the GDA (Global Document Annotation, [GDA]) XML tag set. GDA defines various pragmatics and semantics of linguistic information such as a part of speech, modification relation, anaphoric relation, etc. The marked-up sentence is stored in a database together with the corresponding SVG document.

Example 1 shows an example of the GDA data representing "I walk to a hospital from school" and Example 2 the GDA data corresponding to "私は学校から病院へ歩く", the Japanese equivalent of the above English sentence.

    
<?xml version="1.0" encoding="UTF-8" ?> 
<gda:gda xmlns:gda="http://i-content.org/gda/">
 <gda:su>
  <gda:np gda:eq="gda:p1" gda:mph=";PRP;;;">I</gda:np> 
  <gda:vp>
   <gda:vp gda:mph=";VBP;;;">walk</gda:vp> 
   <gda:adp>
    <gda:vp gda:mph=";TO;;;">to</gda:vp> 
    <gda:np>
     <gda:adp gda:mph=";DT;;;">a</gda:adp> 
     <gda:np gda:mph=";NN;;;">hospital</gda:np> 
    </gda:np>
   </gda:adp>
   <gda:adp>
    <gda:adp gda:mph=";IN;;;">from</gda:adp> 
    <gda:np gda:mph=";NN;;;">school</gda:np> 
   </gda:adp>
  </gda:vp>
 </gda:su>
</gda:gda>
    

Example 1: GDA Markup for an English Sentence

    
<?xml version="1.0" encoding="UTF-8" ?> 
<gda:gda xmlns:gda="http://i-content.org/gda/">
 <gda:su>
  <gda:adp>
   <gda:np gda:eq="gda:p1" mph ="juman;名詞+普通名詞;;私;わたくし">私</gda:np> 
   <gda:ad gda:mph="juman;助詞+副助詞;;は;は">は</gda:ad> 
  </gda:adp>
  <gda:adp>
   <gda:np gda:mph="juman;名詞+普通名詞;;学校;がっこう">学校</gda:np> 
   <gda:ad gda:mph="juman;助詞+格助詞;;から;から">から</gda:ad> 
  </gda:adp>
  <gda:adp>
   <gda:np gda:mph="juman;名詞+普通名詞;;病院;びょういん">病院</gda:np> 
   <gda:ad gda:mph="juman;助詞+格助詞;;へ;へ">へ</gda:ad> 
  </gda:adp>
  < gda:v mph='juman;動詞;子音動詞カ行+基本形;歩く;あるく'>歩く</gda:v>
 </gda:su>
</gda:gda>
    

Example 2: GDA Markup for a Japanese Sentence

This linguistic information can be embedded in the <metadata> element of SVG when an SVG document is generated from the database. Rather than managing metadata separately from the actual pictograms, metadata about pictogram components and their semantic relationships can be kept as part of the pictogram itself.

6. Two kinds of XML structured data

The final goal of our research is the automatic conversion from natural language to pictograms and back. For this, we proceed in steps. The structure of a sentence and the graphical structure of the corresponding pictogram, available from SVG, are compared and matched.

Based on a collection of such data, new pictograms can be created from natural language sentences that are augmented with GDA markup. A simple example is the replacement of a pictogram that is the object of an action or state by another pictogram. Any self-sufficient words, segments and subsentences can be extracted automatically from the original sentence with GDA markup. Table 1 shows a list of these extracted from the sentence "I walk to a hospital from school". Various calculations such as joint, difference and replacement rely on the GDA markup. Rather than simply using string matching and substitution, it has to be considered whether a calculation applies to a word, a segment, or a subsentence to obtain a linguistically adequate result. Sentences such as "I walk" (Figure 8), "I walk from school" (Figure 9), and "I walk from a mountain to a river" (Figure 10) can be converted. Alternative labels enable the conversion of more and more sentences. The system also infers new labels for combinations of pictograms. As an example, in Chapter 4, the sentence "I walk to a hospital from school" was associated with the pictograms of a school, a building, a white cross, and a walking man. From this data, the system can infer that the word "hospital" must correspond to a combination of a building and a white cross.

Extracting sentence fragments at various liguistic levels
self-sufficient word segment subsentence
English

I

walk

hospital

school

I

walk

to a hospital

from school

I walk

I walk to a hospital

I walk from school

Japanese

私 (I)

学校 (school)

病院 (hospital)

歩く (walk)

私は (I)

学校から (from school)

病院へ (to a hospital)

歩く (walk)

私は歩く (I walk)

私は学校から歩く(I walk from school)

私は病院へ歩く(I walk to a hospital)

Table 1

Walk.svg

(This figure is an animation.)

Figure 8: I walk

Walk-from.svg

(This figure is an animation.)

Figure 9: I walk from school

Mountain-river.svg

(This figure is an animation.)

Figure 10: I walk from a mountain to a river

Our system also is able to relate natural language words and phrases to various graphic effects in SVG drawings using the linguistic structure available in the GDA markup. As an example, a cellphone (Figure 11) can take some color (Figure 12), can disappear or be non-existing (Figure 13), or can be moving (Figure 14). Such effects can very easily be implemented for SVG, but would be very difficult to implement for raster images.

Cellphone.svg

Figure 11: Cellphone (携帯電話)

RedCellphone.svg

Figure 12: Red Cellphone (赤い携帯電話)

noCellphone.svg

Figure 13: No Cellphone (携帯電話がない ), Cellphone Disappears (携帯電話が消える)

CellphoneMove.svg

(This figure is an animation.)

Figure 14: Cellphone Moving to the Right (携帯電話が右に移動する)

7. Conclusions

At the moment, users have to supply quite a bit of information, including coordinates, to create a new pictogram, but we continue to work on a system that will suggest layouts and effects (semi-)automatically based only on natural language input. With more and more data as a base, we expect to discover even deeper relationships between natural language, semantics and graphical layout. We are also looking forward to further benefits from SVG when moving to more automatic pictogram generation.

Bibliography

[NHK]
NHK Emoji Chat homepage, http://www.nhk.or.jp/school/nankyoku-kids/ja/frame.html (in Japanese)
[PICT]
Project's homepage, http://sa.carc.jp/pict/ (in Japanese)
[ITO]
Kazunari Ito etc. Ontology Mapping to Promote Making and Understanding Pictograms, DBSJ Letters, Vol. 5, No. 2, pp. C93-96, 2006. http://www.dbsj.org/Japanese/DBSJLetters/vol5/no2/ito.pdf (in Japanese)
[MATSUDA]
Motohiro Matsuda etc. A Rule Extraction Method for Arranging Pictogram Components, DBSJ Letters, Vol. 6, No. 1, pp. C165-168, 2007. http://www.dbsj.org/Japanese/DBSJLetters/vol6/no1/thesis/matsuda.pdf (in Japanese)
[FUJIMORI]
Makoto Fujimori etc. Inspection of Emotional Awareness regarding Arrangements and Animations of Pictograms, International Conference on Kansei Engineering and Emotion Research 2007, 2007 (to appear).
[JIS_T_0103]
JIS T 0103, http://www.kyoyohin.org/06_accessible/060100_jis.php
[GDA]
GDA, http://i-content.org/GDA/

XHTML rendition made possible by SchemaSoft's Document Interpreter™ technology.