Figure 4: Pictogram List
Keywords: pictogram , liguistic information
Biography
Kaz received his Ph.D. Degree in Computer Science from Keio University in 2005. He is currently an Assistant Professor in the Department of Integrated Information Technology, College of Science and Engineering, Aoyama Gakuin University. His research interests include semantic web, natural language processing, multimedia information processing, e-learning and application of pictograms.
1. Introduction
2. Background
3. Pictogram Authoring
4. Prototype
5. Markup for Linguistic Information
6. Two kinds of XML structured data
7. Conclusions
Bibliography
Pictograms are simple graphics very well suited for being represented with SVG. However, until now, both our group as well as similar research efforts have used raster data formats exclusively. In this paper, we report on our effort to convert our collection of pictograms [PICT] to SVG, the advantages gained from this conversion, the introduction of a prototype system for pictogram authoring, and future benefits we are looking forward to.
A pictogram is an expressive form which can be understood intuitively without a grammar. All pictograms are composed of one or more components, starting with primitives such as circles, rectangles, lines, and curves. There are numerous reports and case studies on the successful use of pictograms in the fields of learning assistance for infants and livelihood assistance for persons with disabilities[NHK].
In Japan, design guidelines for pictograms have been established in 2005 as JIS T 0103 [JIS_T_0103]. These guidelines recommend that new pictograms be designed by consciously considering combinations of existing pictograms. As an example, Figure 1 shows the pictogram for a hospital given as an example in JIS T 0103. It can be understood as overlaying the pictogram for a cross (Figure 2) on the pictogram for a building (Figure 3), with the color of the cross changed to white.
Our group has already published (see [PICT]) around 1000 pictograms in the GIF and PNG raster image formats (Figure 4), and a Java based application (Figure 5) to create new raster-based pictograms based on existing ones. Our research is focusing on expressing sentence-level concepts by arranging several pictograms in 2D space, and to study the relationship between 2D arrangements and semantic relationships[ITO][MATSUDA]. But we have found it very hard to proceed with our research using raster images.
To support the creation of new pictograms and the collection of data, we have been researching pictogram authoring. In this paper, we use the term pictogram authoring for the creation of new pictograms by combining and modifying one or multiple existing pictograms. This means that we exclude drawing new basic pictograms from scratch. Pictograms can be modified in a number of ways: applying layout and form changes such as scaling, mirroring, and rotation, changing color, applying animation, and so on. As a result of experiments, we have established [FUJIMORI] that by applying such modifications, the range of concepts that can be expressed is greatly expanded, reusability is increased, new pictogram vocabulary can be created, issues of color blindness can be addressed, and the primary focus of a pictogram can be made easy to understand. Figure 6 shows a example.
(This figure is an animation. Please reload the document if you cannot see a walking man, or use a viewing tool that supports SVG animation.)
Figure 6: I walk to a hospital from school.
To explore the feasibility of using SVG for pictograms, we have created a web-based SVG pictogram authoring tool prototype. Figure 7 shows a screenshot. Using SVG has greatly simplified the implementation. The prototype works on any browser that supports SVG.
The prototype consists of four text areas on the right and a pictogram output area on the left. The topmost text area, labeled "Command", serves to input textual commands to create a pictogram. Commands are separated by vertical bars; each command places an existing pictogram according to a series of parameters. There are at most eight parameters, separated by a comma. The parameters represent label, left, top, width, height, angle of rotation, color, vertical mirroring and horizontal mirroring. The label parameter is a text string that identifies a previously registered pictogram. All other parameters can be omitted, in which case default values are used. For example, the following text command generates the pictogram in Figure 7:
School,0,150,100,100|Building_big,150,0,100,100|Cross,175,40,50,50,,FFFFFF|Walk,50,200,40,50
|
We have already registered about 1000 basic pictograms not only with English labels but also with Japanese labels, so the above command also can be written as:
学校,0,150,100,100|大きい建物,150,0,100,100|十字架,175,40,50,50,,FFFFFF|歩く,
50,200,40,50 |
The text area labeled "Sentence" is used to input a piece of natural language text corresponding to the pictogram being created. For the above-mentioned command, this could be the sentence "I walk to a hospital from school". A remarkable point is that the registered sentence can be reused as a label in the text command to recall the newly created pictogram. The following text command makes sense after registration.
I walk to a hospital from school,100,100,200,200|Sun,0,0,100,100
|
The following sentence could describe the new pictogram: "I walk to a hospital from school on a sunny day".
Another way to use the Command area and the Sentence area is to input the label of an existing pictogram in the Command area and to input an alternative label in the same language or translation in a different language into the Sentence area. This works because a single label is the simplest command possible.
Furthermore, the SVG document generated by the text command can be modified manually in the text area labeled "SVG Source". As an example, a user can change the color of an object or add an animation effect to particular objects or shapes. The SVG Source area also makes it possible to create new pictograms from scratch. In the same way as for pictograms created with the Command area, pictograms created or modified in the SVG Source are can be recalled and reused with the associated label or sentence.
When the user pushes the "register" button to the right of the Sentence area to register SVG data and the corresponding natural language expression, the "URI" area displays a URI including a unique ID that can be used to access the registered SVG document. The point is that when one wants to get an ID for a pictogram, one needs to input a sentence which represent the pictogram. In this way, we encourage the association of metadata with the pictograms. Only the creator of a pictogram knows the ID and can embed the pictograms he created in any Web page.
To further progress towards the automatic creation of pictograms from natural language, we use linguistic information. Every English or Japanese sentence that is input is automatically marked up with the GDA (Global Document Annotation, [GDA]) XML tag set. GDA defines various pragmatics and semantics of linguistic information such as a part of speech, modification relation, anaphoric relation, etc. The marked-up sentence is stored in a database together with the corresponding SVG document.
Example 1 shows an example of the GDA data representing "I walk to a hospital from school" and Example 2 the GDA data corresponding to "私は学校から病院へ歩く", the Japanese equivalent of the above English sentence.
<?xml version="1.0" encoding="UTF-8" ?>
<gda:gda xmlns:gda="http://i-content.org/gda/">
<gda:su>
<gda:np gda:eq="gda:p1" gda:mph=";PRP;;;">I</gda:np>
<gda:vp>
<gda:vp gda:mph=";VBP;;;">walk</gda:vp>
<gda:adp>
<gda:vp gda:mph=";TO;;;">to</gda:vp>
<gda:np>
<gda:adp gda:mph=";DT;;;">a</gda:adp>
<gda:np gda:mph=";NN;;;">hospital</gda:np>
</gda:np>
</gda:adp>
<gda:adp>
<gda:adp gda:mph=";IN;;;">from</gda:adp>
<gda:np gda:mph=";NN;;;">school</gda:np>
</gda:adp>
</gda:vp>
</gda:su>
</gda:gda>
|
Example 1: GDA Markup for an English Sentence
<?xml version="1.0" encoding="UTF-8" ?>
<gda:gda xmlns:gda="http://i-content.org/gda/">
<gda:su>
<gda:adp>
<gda:np gda:eq="gda:p1" mph ="juman;名詞+普通名詞;;私;わたくし">私</gda:np>
<gda:ad gda:mph="juman;助詞+副助詞;;は;は">は</gda:ad>
</gda:adp>
<gda:adp>
<gda:np gda:mph="juman;名詞+普通名詞;;学校;がっこう">学校</gda:np>
<gda:ad gda:mph="juman;助詞+格助詞;;から;から">から</gda:ad>
</gda:adp>
<gda:adp>
<gda:np gda:mph="juman;名詞+普通名詞;;病院;びょういん">病院</gda:np>
<gda:ad gda:mph="juman;助詞+格助詞;;へ;へ">へ</gda:ad>
</gda:adp>
< gda:v mph='juman;動詞;子音動詞カ行+基本形;歩く;あるく'>歩く</gda:v>
</gda:su>
</gda:gda>
|
Example 2: GDA Markup for a Japanese Sentence
This linguistic information can be embedded in the <metadata> element of SVG when an SVG document is generated from the database. Rather than managing metadata separately from the actual pictograms, metadata about pictogram components and their semantic relationships can be kept as part of the pictogram itself.
The final goal of our research is the automatic conversion from natural language to pictograms and back. For this, we proceed in steps. The structure of a sentence and the graphical structure of the corresponding pictogram, available from SVG, are compared and matched.
Based on a collection of such data, new pictograms can be created from natural language sentences that are augmented with GDA markup. A simple example is the replacement of a pictogram that is the object of an action or state by another pictogram. Any self-sufficient words, segments and subsentences can be extracted automatically from the original sentence with GDA markup. Table 1 shows a list of these extracted from the sentence "I walk to a hospital from school". Various calculations such as joint, difference and replacement rely on the GDA markup. Rather than simply using string matching and substitution, it has to be considered whether a calculation applies to a word, a segment, or a subsentence to obtain a linguistically adequate result. Sentences such as "I walk" (Figure 8), "I walk from school" (Figure 9), and "I walk from a mountain to a river" (Figure 10) can be converted. Alternative labels enable the conversion of more and more sentences. The system also infers new labels for combinations of pictograms. As an example, in Chapter 4, the sentence "I walk to a hospital from school" was associated with the pictograms of a school, a building, a white cross, and a walking man. From this data, the system can infer that the word "hospital" must correspond to a combination of a building and a white cross.
| self-sufficient word | segment | subsentence | |
| English | I walk hospital school |
I walk to a hospital from school |
I walk I walk to a hospital I walk from school |
| Japanese | 私 (I) 学校 (school) 病院 (hospital) 歩く (walk) |
私は (I) 学校から (from school) 病院へ (to a hospital) 歩く (walk) |
私は歩く (I walk) 私は学校から歩く(I walk from school) 私は病院へ歩く(I walk to a hospital) |
Table 1
Our system also is able to relate natural language words and phrases to various graphic effects in SVG drawings using the linguistic structure available in the GDA markup. As an example, a cellphone (Figure 11) can take some color (Figure 12), can disappear or be non-existing (Figure 13), or can be moving (Figure 14). Such effects can very easily be implemented for SVG, but would be very difficult to implement for raster images.
At the moment, users have to supply quite a bit of information, including coordinates, to create a new pictogram, but we continue to work on a system that will suggest layouts and effects (semi-)automatically based only on natural language input. With more and more data as a base, we expect to discover even deeper relationships between natural language, semantics and graphical layout. We are also looking forward to further benefits from SVG when moving to more automatic pictogram generation.
XHTML rendition made possible by SchemaSoft's Document Interpreter™ technology.
