Implementing an integrated SVG application for real time dynamically generated Internet mapping

Vincent T. Adcock
Systems Architect
Veterinary Laboratories Agency
New Haw
Surrey
England
v.adcock@vla.defra.gsi.gov.uk
http://www.defra.gov.uk/corporate/vla/default.htm

Biography

Vincent Adcock works for the VLA (Veterinary Laboratories Agency) . He works in the Centre for Epidemiology and Risk Research that performs epidemiological research and risk analysis for both the government and private external organisations. As a part of his most recent work, he has been heading the development of a series of web based mapping solutions each of which involve the use of SVG (Scalable Vector Graphics) in conjunction with ASP (Active Server Pages) .Net


Abstract


SVG has immense potential for Internet delivered maps, combining high quality visualisation and good user interactivity. These advantages make it overwhelmingly superior to raster based image maps, but the advantages of SVG are less when compared to "Internet-GIS" solutions such as ArcIMS and MapXtreme. These are also vector-based and scalable, and being relatively mature solution, provide a completely integrated technology. The relative benefit of SVG mapping compared to these products lies in its flexibility, low cost and potential to adapt easily and quickly solutions developed for other members of the XML (Extensible mark-up language) "family".

Nevertheless, practical implementation of SVG mapping poses many challenges. The most difficult of these is for the delivery of real-time dynamically generated maps, wherein data from an active database is rendered in the browser. This is the requirement when Internet mapping goes beyond the display of static maps, being used as a tool for data integration, visualisation and part of real-time decision support systems.

We faced such a requirement when asked to develop a system to assist the ministry of agriculture ( DEFRA (Department for the Environment, Farming and Rural Affairs) ) in Great Britain in the control of a disease of cattle (bovine tuberculosis), wherein data from a number of sources needs to be combined and delivered to field veterinarians. This requires a large number of maps to be generated, along with charts, meta-data, and guidance for interpretation. The solution we successfully developed uses a number of elements, of which while all are well known within the SVG community, less commonly have been successfully combined into a single complex application.

The development of this application will be described in detail in our session. In brief, it consists of

  1. A web application built using Microsoft ASP .Net and written in VB (Visual Basic) .Net
  2. A pair of underlying SQL Server 2000 databases. One used to house the spatial data and the other used to store the historical disease information.
  3. SVG compression to reduce the size of the files generated by the application. This means reduced load times for users, especially when dealing with dial up or slow network connections.
  4. Maps are pre-compiled on first use meaning that all processing overheads are only incurred once. All other requests summon the completed file.
  5. Integrates a charting engine capable of producing SVG based charts.
  6. Client side JavaScript to allow for rapid navigation of year variable data and to display additional data outside the SVG object itself.
  7. XML metadata to support the SVG images and explain how they were created. Uses XSL (Extensible Style sheet Language) to change the base XML files into a suitable format for presentation.

Currently, the only XML -based technology lacking is the use of XSLT (Extensible Style sheet Transformation Language) , which reflect the difficulties implementing this solution in a Microsoft environment. However, XSLT offers such overwhelming advantages for greater user inter-operability, that this will be the focus of our development effort in the coming months.


Table of Contents


Introduction
Overview of the Application Architecture
     Web Application
         Web Server Architecture
         Web Page Architecture
Databases
     Spida_v1_1 Database
     TBHistoric Database
SVG
Performance Issues
     Pre-compilation
     Compression
JavaScript
Charting
XML Metadata
Results
Acknowledgements
Bibliography

Introduction

The overall purpose of this paper is to provide an overview of the Spida v1.1 web application that was designed to display historical animal disease information, specifically Bovine Tuberculosis (BTB) in England and Wales.

More specifically the document aims to detail:

  1. Overview of the application architecture
  2. Development environment
  3. Data sources, access and storage
  4. Web page architecture
  5. Application custom-built class architecture
  6. Rendering issues
  7. Deployment

Further details of the scientific justification can be found in the complimentary paper, “Use of Scalable Vector Graphics For A Web-delivered Interactive Digital Atlas of Bovine Tuberculosis” which was published as a part of the proceedings of the GisVet Conference 2004, see bibliography for details.

Overview of the Application Architecture

Spida (SVG Presented Interactive Disease Atlas) v1.1 operates around a combination of two key servers. The first of these is a web server used to host the Spida v1.1 web application. In order to run the application the server has version 1.1 of the .Net Framework installed along side Windows 2000 Professional Edition and Internet Information Services (IIS) 5.0

The application is located in a physical directory on the server and is accessed through a virtual directory provided by IIS. In order to allow access to the application by users across Defranet the access permissions have been set to allow anonymous access. This means that users who do not have an account on the server are still able to access the application. If this option was not set only those who have a user account in IIS set up for that server could obtain access.

The second main server is the database server. This server is also running under Windows 2000 Professional and has SQL Server 7.0 installed acting as a data store and provider.

Within the SQL Server instance two databases have been created. The first of these, Spida_v1_1, is the database used to store all the application data. It contains information used in the rendering of data such as vertices for England and Wales at both county and parish level, Ordinance Survey (http://www.ordnancesurvey.co.uk/) grid data and centre points used for the placement of on map charts and figures. In addition, it also holds details of the user accounts for the application and the levels of access granted to each.

The second database, TBHistoric, contains the data that is rendered by the system. It is accessed through a series of views and returned to the application via cross-referencing implemented in a series of stored procedures located in the Spida_v1_1 database. It is worth noting that both databases share an additional user account that was created to allow the system access.

In this case, the account created is named Application and has been granted select permission to all the required views in the TBHistoric table and exec permission to all the required stored procedures in the Spida_v1_1 database.

Outside of these two servers, the TBHistoric database receives periodic updates from a number of external sources. These include the VETNET data, Census data, TB50 data, and BRO data. Each of these is stored in separate locations outside the scope of this document. However, a slightly more in-depth description will be provided in the Databases section of this paper.

The last component of the system is the client PC. The only requirements for it is are that it has Internet Explorer (IE) 5.0 or higher installed and also has the Adobe SVG Viewer 3.0 or higher installed.

Web Application

Figure1.jpg

In order to explain the structure of the application we have divided it into four distinct segments. These are as follows:

  1. Web Server Architecture – Details of the architecture of the top-level, the web server.
  2. Web Page Architecture – Details of the architecture of the next level down, the web pages.
  3. Web Page / Class Interoperability – Details of how the web pages interoperate with the underlying classes to produce dynamic and data driven content.
  4. Class Architecture – Details of the architecture of the lowest layer, the underlying custom built classes.

Web Server Architecture

There are three key sections to the web server:

Internet Information Services (IIS)

  1. This controls remote access to the server via the Intranet.
  2. Permissions are set for the application that allow or deny users access to applications held on the server.
  3. IIS permission are under the control of the server administrators.
  4. In order for the application to function correctly these must be set to allow anonymous access to the application.

.Net Framework 1.1

  1. This provides the Common Language Runtime (CLR) and class libraries that are used by .Net web applications.
  2. The CLR actively compiles code stored within pages at runtime an will process instructions accordingly.
  3. The class libraries provide a range of extended functionality and are referenced by code in the web applications.

Web Application – Spida v1.1

  1. This covers all the web pages, user controls, classes, scripts, images, etc. that make up the web application.
  2. Content is delivered through an interoperation between classes and pages.

When a request is sent to the web server, it is first authenticated by IIS before being allowed access to the application. As anonymous access has been set, the user will be moved onto the next phase.

Rather than accessing the CLR directly, the call from the client will reference an object in the web application. This will then be passed through the CLR and content will be returned to the client.

Depending on the object requested external data might be required. In this case, a call will be sent by the CLR out to an external data source specified in the application code and data will be retrieved. This will then be processed and the dynamic content will be returned to the user (Figure 2).

Figure2.jpg

Web Page Architecture

Figure3.jpg

Databases

As stated earlier, the data sources for the application fall to two main databases (Figure 4):

  1. Spida_v1_1 – The main application database. Contains rendering information and user data.
  2. TBHistoric – The database used to store the historical disease information. When combined with rendering data can be used to historical disease information in a number of different ways.
Figure4.jpg

Access to these databases is handled exclusively through stored procedures. The aim of this is to prevent potential security threats such as sql injection, which can happen when using direct or parameterised sql in the page code.

In order to reduce overheads data is retrieved from the database and held in memory resident datasets for the duration of the majority of rendering operations. This means that the number of open connections at any one time is kept to a minimum reducing the overall delay caused by network communication.

It would be possible for the datasets to be pre-cached before hand and updated on a fixed time basis, however due to memory restrictions on the server this is not presently an appropriate solution and is under consideration for future versions of the application.

Once the data has been retrieved files are generated and stored on the server. These are stored in a temporary directory created in the Application Root/Files directory and are kept so long as the user remains logged in. When they log out of the application the files and directory are removed. This is performed to minimize space consumption on the server.

Spida_v1_1 Database

The Spida_v1_1 database is divided into three distinct groups of tables, each group related to a different section of the application.

The first group are tables relating to the rendering functions of the application and hold both coordinate data as well as lookup tables with the available data types, legend text and display colours.

The second group of tables relate to the administration of the application. These control user access to the application and the recording and handling of issues reports.

The third groups cover the other miscellaneous tables that do not server any direct function. These are generally used in the creation of data and are not used directly by the application.

As stated above retrieval and insertion of data into the database is handled entirely by stored procedures. Each one of these serves a different purpose and many of them are used by different segments of the application.

TBHistoric Database

Due to the complexity of the TBHistoric database it is not appropriate to cover the entire table contained within here.

SVG

The requirement was to produce choropleth maps, proportional symbol maps, grid maps, and dot maps.

  1. A choropleth map represents data by the shading of predefined spatial areas according to the underlying attributes.
  2. A proportional symbol map is similar in nature to a choropleth map, however in a proportional symbol map rather than the areas themselves displaying the data, information is represented by a series of 'proportional symbols'. An example is a map of England divided into counties each with a pie chart demonstrating the values associated with that area for the current theme.
  3. A grid map works exactly like a choropleth map but rather than directly representing the spatial area it uses a series of grid squares. These are then coloured according to their value in the same way as with a choropleth map.
  4. A dot map is similar to a proportional symbol map, however in a dot map each 'dot' refers to both the spatial location and value of data.

In addition, there are some of hybrid types but they will not be covered in the scope of this paper.

Choropleth, dot, and proportional symbol maps can be displayed at both countrywide and countywide levels. A countrywide area is composed of a series of counties while a countywide area is composed of a series of parishes. Navigation from countrywide to countywide level was achieved by moving the mouse of the desired area and left clicking, this then caused the page to reload now displaying a countywide view of the selected area. This was achieved by wrapping the path objects in and a tag and using an xref to link back the parent page with differing query data.

Grid maps however are only displayed at a country wide level using a grid divided into 10km square areas. No second level is presently assigned although it may be possible to use a 3.163km square grid if wishing to view data at a countywide level. Due to the non-standard shapes of both counties and parishes it was decided to represent them as Path objects. Coordinates are assigned from an initial point using the M directive in the d attribute and then each of the following coordinates is plotted using the L directive. Finally the z directive is used to close the shape so that its fill attribute can be changed. It should be noted that in order for the coordinates to be drawn correctly the order in which they are to be drawn must be extracted from the original shape file.

Pie charts are also represented using paths; this is because a circle segmented by lines cannot have attributes of the individual segments changed. Using the arc directive A it is possible to draw an arc from one point to another forming the outside of the circle. Presently the girds are also represented using path although rect could also be used. Finally dots are represented using the circle tag.

Figure5.jpg

Performance Issues

Pre-compilation

Initially all the SVG files were created dynamically causing the files to be generated by the server each time they were requested. This produces an extended load time to account for the network overhead and the server processing time.

In order to avoid incurring the overhead every time a page is requested pre-compilation was employed. As the data used by the system is historic and therefore static in nature it was possible to create the files the first time they were requested, keeping them resident on the physical storage of the server. In the case that it was dealing with dynamically changing data code could be put in place to delete the files and update them as required.

Compression

In order to successfully render detailed maps together with the associated information the volumes of data to be transmitted to the client machine were so large that the response times, particularly over slower networks, were in the order of five minutes. This was deemed to be unacceptable and so a solution was sought that would reduce the times to within reasonable levels, approximately one minute at maximum. The initial method employed was to reduce the resolution of the spatial data; by reducing the number of vertices plotted, it was possible to greatly reduce the file size and consequently the time required for the SVG file to be displayed. However, this still did not achieve the desired sub minute times.

Further research identified the ability of SVG viewers to render SVG files even when they are in a binary compressed form without the need for decompression. Following searching on the Internet we were able to locate a binary compression library for .Net written in C# capable of performing GZip compression. Because of the common architecture of the framework, despite the fact that the library was compiled in C# as it is still a part of .Net it can be called normally as if it were written in VB. The resulting SVG files were found to be in most cases less than

vTempFileName.Append(_ThemeId)
vTempFileName.Append("_" & _VisualisationId)
vTempFileName.Append("_" & _AreaId)
vTempFileName.Append("_" & HttpContext.Current.Session("Greyscale"))
vTempFileName.Append(".svg")

vFileName.Append(_ThemeId)
vFileName.Append("_" & _VisualisationId)
vFileName.Append("_" & _AreaId)
vFileName.Append("_" & HttpContext.Current.Session("Greyscale"))
vFileName.Append(".svgz")

If Not File.Exists(vFilePath & vFileName.ToString) Then
	Dim MyStreamWriter As New StreamWriter(vFilePath & vTempFileName.ToString)
	MyStreamWriter.Write(Content)
	MyStreamWriter.Close()

	Dim MyZipStream As New GZipOutputStream(File.Create(vFilePath & vFileName.ToString))
	Dim MyFileStream As FileStream

	MyFileStream = File.OpenRead(vFilePath & vTempFileName.ToString)

	Dim vByte(MyFileStream.Length) As Byte
	MyFileStream.Read(vByte, 0, CInt(MyFileStream.Length))
	MyZipStream.Write(vByte, 0, vByte.Length)
	MyZipStream.Close()
	MyFileStream.Close()

	If File.Exists(vFilePath & vTempFileName.ToString) Then
		File.Delete(vFilePath & vTempFileName.ToString)
	End If
End If
				

JavaScript

In order to accommodate the fact that objects may need to be changed dynamically, and therefore need to be capable of being uniquely referenced, the id attribute is no longer suitable for storing supplementary data. In order to accommodate this need an additional attribute was added, name. In order to display this additional data JavaScript functions had to be employed. A mouse over event was added to each of the objects with a listener placed in the web page where the SVG object would be embedded. When the event handler is fired, it picks up the attributes of the object and displays them on screen. A second mouse out event was also added to hide the data when the object was no longer selected.

The non-spatial data associated with each object spans a number of years. In order to display this data initially separate maps were required. To eliminate the need for loading additional pages JavaScript was employed to allow navigation between years. In order to accommodate this function it was necessary to include all the data relating to the currently selected spatial areas for the entire time period. Initially replicating the spatial objects approached this and simply using JavaScript to turn their visibility on and off, this proved to be inefficient as it drastically increases the file size. In order to resolve this problem a different approach was required.

It was determined that it would be more appropriate to define the spatial objects once and then use the JavaScript to adjust their attributes accordingly. Text objects were chosen as a suitable data container. The text objects are grouped into years inside a g object identified. When the JavaScript is called it receives a string value for the year from the main page and locates the corresponding g object then using a loop updates the attributes of the spatial objects to those contained in the corresponding text object. The final result of this was a near instantaneous change to the visually represented spatial data.

function initSvg(evt) {
	SvgDoc=evt.target.parentNode;
};

function setYear(vYear) {
	var vValues;
	var vArray;
	var vRow;
	var vObject = SvgDoc.getElementById(vYear);        
	vValues = vObject.firstChild.data;
	vArray = vValues.split('_');
	for (i=0; i < vArray.length - 1; i++) {
		vRow = vArray[i].split('~');
		var vTempObject = SvgDoc.getElementById(vRow[0]);
		TempObject.setAttribute('name',vRow[1]);
		vTempObject.setAttribute('fill',vRow[2]);
	}
};

function data_default(id) {
	alert(id)
}
if(window.data==null) {
	window.data=data_default;
};
			

Charting

Another supplementary benefit of development environment was identified when addressing the additional requirement of producing chart-based summaries of the attribute data; the preferred solution was to make use of the capabilities of SVG to draw them. The initial forays into producing these were unsuccessful and following some research a suitable proprietary solution was found.

GraPL (See http://www.grapl.com/) is a charting tool that can produce various forms of chart in SVG format. Following discussion with the developers at GraPL it was possible to secure a working C# version of the core libraries running under version 1.1 of the .Net Framework. By integrating this into Spida it became possible to produce the charts required dynamically in a similar fashion to the maps.

Furthermore, as GraPL’s output was also in SVG it was possible to employ the same compression routine to reduce the file sizes and avoid an increase in load times.

Dim MyChart As New GraPL()

…

With MyChart
	.[New](0, 0, 400, 400)
	.FrameStyle = "Wiped,Boxed"
	.Style = "Relative,Forcezero,Stacked,Redraw"
	.Margins = New Double() {50, 50, 50, 50}
	.Heading = vTitle
	.XLabels = vGroups_Y
	.Axes = "Black,1,Solid"
	.XCaption = "Years"
	.YCaption = vYCaption
	.XStyle = "Angled"
	.YStyle = "Atend"
	.CategoriseBy = vCategories
	.CategoriseInto = vGroups_C
	.Colours = vColours
	.Patterns = "Solid"
	.GroupBy = vYears
	.GroupInto = vGroups_Y
	.KeyFont = "times new roman, 8"
	.Barchart(vData)
End With

Return CreateFile(Constants.SVGType.Thumbnail_Chart, MyChart.RenderSVG(True))
			

XML Metadata

In order to provide some idea of where the data was sourced from XML based metadata was included. The information itself is stored in a series of XML files and then processed by the Framework’s built in XSLT processor when requested by the user. The XSL file included alters the format suitably and the resulting output is displayed on screen.

Figure6.jpg
Figure7.jpg

Results

Examples of the maps and their associated SVG outputted by the completed application are shown below.

Choropleth Map:

Figure8.jpg

Choropleth Map SVG:

	<?xml version='1.0'?>
<svg onload='initSvg(evt)' xmlns='http://www.w3.org/2000/svg' xmlns:xlink = 'http://www.w3.org/1999/xlink'  viewBox='134148 1182444 111894 106482' preserveAspectRatio='yes'>
<script>
	var SvgDoc=null;
	function initSvg(evt) {
		SvgDoc=evt.target.parentNode;
	};
	function setYear(vYear) {
		var vValues;
		var vArray;
		var vRow;
		var vObject = SvgDoc.getElementById(vYear);
		vValues = vObject.firstChild.data;
		vArray = vValues.split('_');
		for (i=0; i < vArray.length - 1; i++) {
			vRow = vArray[i].split('~');
			var vTempObject = SvgDoc.getElementById(vRow[0]);
			vTempObject.setAttribute('name',vRow[1]);
			vTempObject.setAttribute('fill',vRow[2]);
		}
	};
	function data_default(id) {
		alert(id)
	}
	if(window.data==null) {
		window.data=data_default;
	};
</script>
<a xlink:href=''>
<path d='M177271,1266401
…
L177271,1266401z' fill='#ffa500' name="BUDOCK | 2" stroke='black' stroke-width='1' id='7506' onmouseover='data(evt.target.getAttribute("name"));' onmouseout='data(" ");'>
<set attributeName='fill' attribueType='CSS' to='#87cefa' begin='mouseover' end='mouseout'/>
</path>
</a>
				

Proportional Symbol (Pie Charts) Map:

Figure9.jpg

Proportional Symbol (Pie Charts) Map SVG:


<g id='Data_1987' visibility='visible'>
	<circle cx='465789' cy='1126861' r='3370.99931231621' fill='#ffa500' stroke='black' stroke-width='1' id='11874_1987' name='Unconfirmed | 2/2' onmouseover='data(evt.target.getAttribute("name"))' onmouseout='data(" ")'><set attributeName='fill' attribueType='CSS' to='LightSkyBlue' begin='mouseover' end='mouseout'/></circle>
	<path d='M190093,1235686 L170437,1235686 A19656.1348276724,19656.1348276724 0 0,1 203335,1221160 Z' fill='#ff0000' stroke='black' stroke-width='1' id='11879_1987' name='Confirmed | 25/68' onmouseover='data(evt.target.getAttribute("name"))' onmouseout='data(" ")'><set attributeName='fill' attribueType='CSS' to='LightSkyBlue' begin='mouseover' end='mouseout'/></path>

…

</g>

				

Acknowledgements

The author is grateful to DEFRA for funding this project (SB4013) and allowing for this application to be built.

Dr Peter Durr for writing the project and leading it as well as his guidance in writing this paper.

Dr Nigel Tait and Mr Ross McDonald for their assistance in obtaining spatial data from ArcGIS and other sources.

Mr Julian Greene for his work on the attribute database and the production of suitable views into the data.

Mr Stuart Eastland for his support in the initial stages of the development and guidance.

Bibliography

Adcock et al (2004)

GisVet Proceedings

www.gisvet.org

XHTML rendition created by gcapaper Web Publisher v2.0, © 2001-3 Schema Software Inc.