Spatial ETL, Mapping, and SVG

Don Murray
President
Safe Software
Vancouver
British Columbia
Canada
http://www.safe.com

Biography

Don Murray graduated with honours from Simon Fraser University with a B.Sc. in Computing Science in 1988, and completed his M.Sc. in Computing Science in 1990. Don is a key architect and co-inventor of the industry standard spatial data translator Feature Manipulation Engine (FME) in 1995. He has over 12 years experience as a Software Engineer, starting his career as a Software Engineer for MacDonald Dettwiler and Associates in Richmond, B.C.. He also was an Instructor with the British Columbia Institute of Technology for several years teaching a variety of courses including CASE, Advanced Operating Systems and Object Oriented Design and Programming.

Mary Jo Wagner
Freelance Writer
Unaffiliated
Vancouver
British Columbia
Canada

Biography

Mary Jo Wagner is a Vancouver-based freelance writer with 10 years experience in covering the geospatial technology.


Table of Contents


What is ETL?
What is Spatial ETL?
The Spatial ETL Trend
Spatial ETL and Mapping
Spatial ETL, Mapping, and SVG
Spatial ETL, Web-based Mapping and SVG
Spatial ETL, Wireless Mapping and SVG
Spatial ETL and SVG Working Together
Footnotes

What is ETL?

Business intelligence. It is what data extract, transform and load (ETL) tools have been bringing to both private and public industry for a decade. Spatial ETL tools aim to bring Geospatial Intelligence to organizations exposing the potency of the geographic position to the corporate desktop.

Simply put, the objective of ETL tools is to transfer data from one datastore to another. To reach that objective they perform three separate functions. First the extract function reads data from a specified source datastore, extracting the desired data. Next, the transform function processes the acquired data – transforming it and even perhaps combining it with other data – to package it into the correct structure for the destination datastore. Finally, the load function writes the resulting data to a target datastore.

ETL tools are used either to acquire a temporary subset of data for a customized report or a specific business application, or to obtain a more permanent dataset to populate a data warehouse, to convert from one datastore to another or to migrate from one datastore or platform to another.

The power of ETL tools is that they provide an economical mechanism to quickly combine information from very large data repositories, displacing disjointed databases with open windows to all business-related information and operations. The seamless interconnectivity gives organizations the opportunity to improve productivity and better leverage the information they already have to ultimately make more informed business decisions. ETL tools enable users [1] to work with the data they want, in the view they want, when they want. Bottom line? ETL tools help companies transform data into dollars.

What is Spatial ETL?

A term coined by Safe Software, Spatial ETL takes ETL to the next level by adding the spatial component. Spatial ETL products are designed to provide the same bottom-line benefit as traditional ETL systems, but with one distinct difference – they also include the spatial information within an organization, either in helping it create/mine spatial data or in capitalizing on the spatial data it already has. Here too the goal is to leverage existing spatial data – often the most untapped key corporate asset – to enhance business analysis and ultimately business decisions.

A Spatial ETL tool such as Safe’s Feature Manipulation Engine (FME) extracts data from a desired datastore, and then transforms it to the requested projection, format, and view. The data can then be presented to a requesting user application or loaded into another datastore. Again, the intended data can be obtained for temporary use by a user or application or as part of a permanent data migration/translation project. In short, the goal of Spatial ETL is to give users the right view, of the right data, right now!

Right View: The data is presented to the user in the way that the user finds most useful, rather than forcing the user to manually figure out what they need or do further analysis.

Right Data: The data is available to the user no matter what its source. The data is also the most recent data available for the task so that decisions are not being made with anything less than the highest quality most current data.

Right Now: The data is available to the user immediately when wanted.

Often companies employ data translators to migrate or translate one particular datastore to another. Spatial ETL tools can certainly also do the data translation task but their value is really met when they are used to restructure specific datasets or an entire datastore into another datastore during the translation process. However, unlike traditional data translators that are typically used when the source system is being abandoned, Spatial ETL systems can make the data available on both systems. This allows organizations to use both systems during the migration process – a feature that is particularly relevant for organizations with legacy systems.

As a practical business tool, Spatial ETL is able to transform databases of tables, lines, points and polygons into a seamless spatial composite of business operations on a neighborhood scale, regional scale, national scale or global scale. For the nationwide retail chain, for example, a Spatial ETL system can integrate locations of depots and stores with street network data and real-time traffic reports, and present that unified view to GIS software. Dispatchers can then use this data and the decision making capabilities within the GIS system to more effectively assign routes. They can then also monitor those routes, warning drivers in advance of traffic accidents or road construction and re-routing deliveries on the fly. It is important to note that Spatial ETL systems complement Database and GIS systems and are responsible only for presenting the data to the GIS or other applications for further analysis and decision making.

In short, spatial ETL will do for the spatial world what the traditional database ETL tools have done for the corporate world: provide the ability to integrate disjointed datastores and leverage that enhanced business knowledge to compete with corporate cunning. Spatial ETL provides a mechanism for users not only getting access to data but also getting access to data in the “view” that they need to make decisions. Indeed, as Figure 1 shows, different users need the data presented in different ways to perform different tasks.

image1.gif

Figure 1: Spatial ETL Providing Different Users with Different Views

The Spatial ETL Trend

If the spatial data pundits are correct in that 80 percent of all corporate data is location- based, then the case for using a Spatial ETL system to help access, integrate and apply that geospatial data is indeed a strong one.

Just as companies over a decade ago began to view and employ the traditional ETL tool as a facilitator to better corporate information, so too have private industry and public organizations begun to adopt the Spatial ETL tool as the de facto facilitator to harnessing the intelligence held within spatial information.

After all, business is so tied to the question of “Where is?” that the application of geospatial data and GIS are becoming unmistakable elements of daily business life. At the same time, however, effective spatial data management has also been dogged with data interoperability issues – proprietary formats rather than open formats have dominated, making it difficult for companies to easily and efficiently integrate and apply spatial information. Spatial ETL systems aim to resolve this issue by empowering a GIS package or similar spatial data management tool with the inherent interoperability to provide seamless interaction with a multitude of datastores and to share that information across the enterprise in real-time. It is this invaluable data pipe to spatial information that is positioning Spatial ETL on the strategic Geospatial Intelligence [2] map of an impressive number of business and government sectors.

Spatial ETL and Mapping

A large number of organizations in both industry and government have a large number of mapping or GIS systems in use. It is a heterogeneous world of applications with different applications performing different tasks and hence requiring the information in different structures and different formats. It is the task of Spatial ETL tools to bridge these disparate systems together, enabling data to flow between applications.

Indeed many organizations are in the process of consolidating all of their spatial data to databases in the same way that they moved all of their non-spatial data to databases over the last few decades. It is not practical for organizations to abandon their existing mapping applications as part of this process. Spatial ETL is the “bridge” between the new corporate spatial database and the existing mapping solutions, enabling users to work as before.

Spatial ETL, Mapping, and SVG

How does SVG (Scalable Vector Graphics) fit into the puzzle of bringing data to every desktop? SVG is a standards-based vector format with freely available viewers and Internet browser plug-ins. This enables users to view data using the one of the most familiar computer interfaces in existence – the Internet browser. Spatial ETL and SVG together make a cost-effective one-two punch for the majority of users that require “view only” access to data. Spatial ETL transforms data from the source datastore into SVG in the view that is required for the user’s task. The user then simply uses the SVG component within their browser (or other free SVG viewer) to view the data.

Why not just distribute raster data to each user? Unlike raster, SVG enables common operations like pan and zoom to be performed without the pixelization effects that are inherent in raster formats. SVG also enables attributes to be sent with the data so that the users can perform simple operations. Indeed, advanced Spatial ETL systems enable JavaScript to be embedded within SVG, allowing users to perform common operations or queries locally. The fact that SVG is backed by major corporations such as Microsoft, Netscape, Sun Microsystems, Corel, Adobe, Quark, IBM, and Apple only strengthens the case for SVG. These companies are committed to developing tools for SVG and indeed there are even no-cost SVG Viewers available (www.adobe.com/svg).

Figure 2 shows how Spatial ETL and SVG can work together to share data across an organization. It illustrates how Spatial ETL and SVG can be combined to provide many views of corporate data coming from multiple sources.

image2.gif

Figure 2: Spatial ETL and SVG with Multiple Datastores

Spatial ETL tools and SVG coupled together provide a cost-effective method of giving access to the data from all datastores supported by the Spatial ETL system. There are Spatial ETL tools today that are capable of converting data from over 100 different spatial formats to SVG.

For organizations that have made the transition to a single corporate spatial database, the Spatial ETL/SVG system is simplified by having only one data source for all the data. This illustrates yet another reason for organizations to use Spatial ETL to move to a single corporate database. The simplified system for organizations with a corporate database is shown in Figure 3 .

image3.gif

Figure 3: Spatial ETL and SVG with a Corporate Database

Spatial ETL, Web-based Mapping and SVG

The recent development and deployment of web-based mapping tools has further increased the availability of spatial data throughout organizations, taking them to another level of sophistication. The Spatial ETL/SVG data delivery discussed above presented a static view of the data. Each evening or at some set interval, the Spatial ETL system would be triggered to provide snapshots of the data to its users.

Granted, this is better than what many organizations currently have, but for organizations with their data holdings in a corporate database, web-based mapping tools enable users to view current data. How does SVG fit with this scenario? Currently the majority of web-mapping tools are raster-based or based on proprietary vector streams with downloadable plug-ins that enable users to perform some operations on the client machine.

Why not use SVG? A number of organizations such as Universal Map (check out Universal Map’s SmartMaps technology) have done this. In a traditional raster-based web-mapping solution, when a user performs a zoom operation a new request is made to the server to generate a new raster image. As mentioned previously, there is no need for this with SVG since the client can perform the zoom operation with the data that it has, and a new data request is made to the server only when a zoom threshold is crossed. In many cases, the zoom operation can be done completely on the client thereby reducing the number of server requests per client, which in turn improves the performance and scalability of the entire system. The user experience is also improved because the user is spending less time waiting for the server. From a development perspective, SVG also makes sense as it is a standards-based solution that is embraced by major players in the industry. Why not leverage the efforts of these companies?

Typically, for web-based mapping systems all users are presented with one view of the data rather than a different view for different purposes. Coupling (or embedding) Spatial ETL into a web-based mapping system enables users to have the atypical (for a web-mapping) ability to get the data in the view (and/or format) they want.

Coupling these two technologies together also makes it possible for the web-based mapping system to serve as a gateway to other applications. The web-based mapping system is used as the interface for users to search, identify, and select the data they want. The Spatial ETL component is used to present the data in the format and view needed by the consuming application. These two technologies complement each other very well.

Figure 4 shows a web-based system that uses SVG for the front end coupled with a Spatial ETL system that is used to generate the SVG for the browser, and also provides the ability for users to request desired data in any format supported by the Spatial ETL System.

image4.gif

Figure 4: Components of Web-based Mapping System Using SVG and Spatial ETL

Spatial ETL, Wireless Mapping and SVG

Having found that mapping data is very valuable for decision support, some organizations are now providing spatial data access to their mobile workers.

Again both Spatial ETL and SVG are well-suited to support this type of system. SVG has been designed to be a compact XML-based format that uses communication bandwidth efficiently. Further strengthening SVG’s position is the fact that it can be used to represent anything graphical, not just spatial data. This enables SVG to provide a unified approach to sending all graphical information to a mobile device.

There have been discussions of lightweight spatial data formats for wireless devices. The value of these over SVG would be dubious at best. Why would an organization building a wireless data infrastructure want to send mapping data in one format and other graphical information in a different format such as SVG?

Again SVG is well suited as the method for viewing mapping data for wireless/mobile devices.

From the Spatial ETL perspective, the wireless clients are no different from any other clients. The Spatial ETL system merely returns and transforms data as it does for the other systems described. Sure, when being configured the Spatial ETL system may perform more aggressive generalization operations on the spatial data and be more selective about the amount of attribute information that is returned because of bandwidth concerns, but those are decisions left to the personnel configuring the system. Notice the similarity between Figure 4 and Figure 5 . Figure 5 illustrates the Spatial ETL/SVG system supporting wireless devices.

image5.gif

Figure 5: SVG and Spatial ETL in a Mobile Solution

Spatial ETL and SVG Working Together

With the notion of enhancing business intelligence firmly established, the operative words for the future development of Spatial ETL tools are “application support” and Geospatial Intelligence. Mirroring the successful growth and subsequent adoption of traditional ETL tools in the corporate world, Spatial ETL vendors such as Safe Software are now expanding Spatial ETL functionality to complement the leading GIS vendors’ products and applications.

Giving users the right view of the right data, right now is what Spatial ETL is all about. Spatial ETL can be thought of as a data-pipe and SVG as the terminus that brings spatial data viewing to the masses in a cost-effective manner.

No matter the sophistication of an organization’s data holdings and infrastructure, Spatial ETL coupled with SVG provides a standards-based solution enabling spatial data to be disseminated throughout organizations – today!

Combining the capabilities of the spatial database, Spatial ETL, SVG and web-based mapping tools, the number of people with access to spatial data increases dramatically.

Never before have so many people had access to so much data. This can all be done today using 100% commercial/free software.

Providing Spatial Data Access to the masses. That is the future of Spatial ETL and SVG.

Footnotes

  1. Throughout this document, user is used to mean the consumer of the ETL data. This can be either an application or an end user.

  2. Geospatial Intelligence is a term coined by NIMA. It refers to the ability to gather the most relevant information from geospatial data to facilitate intelligent decisions.

XHTML rendition created by gcapaper Web Publisher v2.0, © 2001-3 Schema Software Inc.