A Public Transport Mapping Example Using perl and mySQL

There exists an ongoing challenge for presenting public transport internet information in a structure that the public can comprehend in the most optimal way. This challenge was greatly eased with the introduction of SVG. Benefits are experienced for intending passengers, as SVG opens up a whole new way of organising information about public transport services, times and routes.

This paper deals with a public transport mapping and related information site currently nearing completion for the internet public. Particular emphasis is placed on what I discovered as I constructed the mapping engine, as well as the results gained.

The internet site primarily addresses passenger rail information for the State of Queensland, Australia. The site's foray into SVG mapping represents an expansion of the site's focus into all modes of public transport. Presently the SVG mapping covers the the activities of the major rail operator, covering Queensland, and the major bus operator in the capital city of Brisbane. To illustrate the scale of the project, the Brisbane region is the third-most populous in Australia. The Brisbane City Council area in particular is home to 900,000 people. The QROTI database (describing the area) contains about 8000 bus stops/railway stations (combined) and 1900 kilometres of routes. We are using a dataset closely approximating actual transport operations, therefore the site should give a good idea of 'real-world' behaviour. The mySQL database comes in under 100 megabytes on disk.

The live internet address as a home page of http://qroti.com/placeinfo/map/. The purpose for providing and serving the live example is to advertise that such an example exists, and can be adapted to the scale of many other public transport regions throughout the world. To our knowledge it's the first SVG-powered example of its type in the world, debuting in September 2002.

QROTI was established in 1996 as a website operated by volunteers, providing an independent source of rail transport information for the Queensland region.

Readers are assumed to know how to read Perl code, how to retrieve packages stored on CPAN (the Comprehensive Perl Archive Network) , the process involved in transmitting SVG from a server to rasterisation on a user-agent, basic RDBMS (Relational DataBase Management System) theory, and a general understanding of the SVG 1.0 specification. An appreciation of GIS (Geographical Information Systems) and public transport usage is not necessary, but will be an advantage in understanding the subject matter.

1. Introduction

This paper describes my experiences with implementing SVG for the purpose of dynamically rendering maps on our website. The area of interest is the public transport infrastructure in Brisbane, Australia. We started work in 2000 and since we entered so early in the SVG development cycle, I largely resorted to creating our own SVG-generation code. I also had to deal with the reality of implementation-specific issues; therefore the solution is biased towards a combination of Windows 2000, Internet Explorer 5.5 and Adobe SVG Viewer 2.0 (as found in the now-obsolete Acrobat Reader 4.x package). The reader should assume that this combination of software is the default used in this paper unless otherwise advised.

2. Background

As the internet environment matured and QR started developing its official website, we continued to look for ways to provide freshly-unique internet content for the public community. In 2000 I noticed the SVG recommendation being developed along with the Adobe SVG Viewer. At that point, many possibilities arose for providing public transport information in a clearer structure. The results of many of these possibilities comprise the topic covered by the rest of this paper.

3. The Results So Far

This section demonstrates some examples of what we have acheived so far. Discussion of how the examples were created will appear in the following sections.

3.1 Extract of QROTI Site Map

3.2 Examples of Actual SVG Maps

Note that some of the examples that follow have a line of diagnostic information at the foot of the copyright block. This will be removed once that type of map is fully ready for production.

4. Choosing our Toolset

4.1 Why SVG?

We always wanted to provide mapping data on our website. However up until the advent of SVG, our only reasonable low-cost option was to manage a portfolio of static GIF (Graphics Interchange Format) images. The obvious problem here is change management – one change in the source data could require a change to several images overlapping the changed area. Additionally there was no easy way to determine exactly which images needed updating for a particular change. This was unsustainable given the volunteer (therefore time-limited) nature of website maintenance. Dynamic generation of GIFs via a server script also appeared to be an option with a lot of effort (or money) and a clearly limited lifespan. It also kept the considerable task of rasterising the maps on the server side, creating a prime bottleneck.

The SVG technology gave us a low-cost toolset - being the open specification and freely-available user agents - with which to provide mapping data on the QROTI website. The change management problem was therefore optimised to the point of only requiring the change on the source dataset. A web server script could then generate SVG documents dynamically from the dataset. We also liked the ability to print high-quality versions of the screen copy.

Initially the idea was to first write a server-side Perl script that used SVG for rendering mapping around railway stations. This would allow advanced users (otherwise known as trainspotters or gunzels, depending on your local dialect) to experience a virtual tour of the railway and station layouts. However, once that milestone was within reach, we thought about building upon that Perl script in two ways:

At the time of writing this paper, QROTI is well on the way with compiling bus information, whereupon completion we will integrate ferry information.

4.2 Why Perl, mySQL and Apache?

5. Design Considerations

Ultimately the genesis of this whole idea - of a public transport website, of creating a mapping engine - comes down to personal selfishness. I wanted to create this for myself, since a comparable example for my city did not previously exist in any form. The incremental cost to 'polish' my work for public consumption is balanced by my sense of satisfaction in doing so. Therefore I created the SVG mapping framework for my website.

Note that I wanted to address questions that arise when travelling in my city's network, as well as those that arise when I travel in foreign cities such as Sydney or Melbourne. A list of issues I specifically wanted to address in this project included:

What follows is a selection of design areas I considered when bringing this list to reality.

5.1 Transmission Latency

I started writing my code to render SVG graphics in 2000. At that time there was a deficiency of SVG-oriented APIs or other abstractions to merge into my Perl script, so the majority of code was rendered in-house. Therefore we traded the 'political correctness' of APIs for coding time.

Also, the server script normally (assuming a relatively light load on the server) outputs the SVG rendering stream quicker than a dial-up user can receive it. This means that the time it takes from the start of the user agent rasterising the data, until completion of the transmission, is about the same for both options. Therefore a map renders quicker using the in-house solution, allowing server scripts to have a shorter process lifetime. In theory this should permit greater workloads at the server end, as scripts are recycled more quickly.

Given the earlier start-of-transmission, our method also allows the user to terminate the download earlier if the map is not what they expected. In certain web server environments, there may be the possibility of terminating a script early if the user has cancelled their request. A future area of research could be to see if this behaviour exists in today's webservers.

5.2 Map Views

In cases 1 and 2 above, all map layers are shown being stops/stations, routes/railways, streets, landmark labels, and water features. The place of interest appears as a slow flash. A 'detailed view' can also be rendered which thins the linework, allowing finer detail to be viewed. Case 3 omits the detailed street layer to avoid a bloated document size. Also, only those stops on the route are shown instead of all stops in the view.

Other map layers were available to further annotate land use in the rendered maps, for example 'open space', 'retail trading areas', 'public utilities', and 'schools'. These were eventually rejected as they were judged to not carry enough extra information for the bloated SVG document size. 'Water features' was included since Brisbane is a 'river city' adjoining a bay, and many residents are familiar with the various points and reaches of the Brisbane River and Moreton Bay.

5.3 Hyperlink Density

All stops shown on each view are selectable, which then brings up a page relevant to that stop with a map centred on that stop. In future as we expand our coverage, we will need to allow road segments to be selected too. This is because routes outside the Brisbane region tend to be run on a 'hail and ride', not a ‘hail at marked stops only’ basis.

5.4 Anti-Theft Measures

5.4.1 Inbound Links by Third Parties

Given that our maps of Brisbane are some of the most comprehensive available on the internet, the possibility existed for third-party websites to link directly to our mapping engine URL. (We seem to be particularly popular with search engine users who found QROTI through a "Suburb map" query.) Since it is important to manage the bandwidth allowance of a website effectively, these inbound links needed to be discouraged.

Also, since the mapping engine relies on untrusted parameters (arriving over the internet) to determine what area to draw, there would be nothing stopping a malicious user requesting a full State of Queensland.

To solve this problem I created a time-limited ticket system. On our website, the web page containing the map of interest is created dynamically. As part of the dynamic construction of the outer page a ticket is created - comprising a random number, expiry date and trusted set of mapping engine 'area to display' parameters - and stored directly in the mySQL database. The outer page also is constructed so that it calls the inner SVG map with the same random number and 'area to display'. When the mapping engine is called, it validates the incoming parameters against the stored ticket in mySQL, ages-out old tickets from mySQL, and then renders the map if successful. If the ticket is invalid, a message is rendered recommending the user either:

5.4.2 Piracy of Base Mapping Data

Given that our base mapping dataset was provided under a favourable licence by MapInfo Australia, I wanted a measure of protection against users reconstituting proprietary layers (such as the general street network).

The nature of SVG means it will never be fully protected against misuse of rendered data in a single map. However I implemented the following countermeasures:

5.5 Bandwidth Usage

5.5.1 Data Compression

We use gzip encoding where possible. (After coding this option, our tests showed that gzip compresses our data transmissions over the wire to 25% or less of the uncompressed size.)

5.5.2 SVG Path Data

To get QROTI's SVG mapping engine working in a straightforward fashion, I used absolute coordinates in SVG paths (as detailed in [SVG1.0-REC] section 8.2). In future QROTI could migrate to relative paths for a small space saving. Similarly, QROTI could also omit successive L (lineto) commands for a further space saving.

5.5.3 Generalisation of Paths

Often the spatial information, as stored into the mySQL database, can comprise points that don't influence the general shape of an object. For example:

When zoomed-out sufficiently, these extra points would not be noticeable. Therefore a function was created to 'generalise’ the drawing path.

The Generalise function is set up so that you can supply the original path data and a 'maximum error'. The function's output guarantees that the generalised path never varies more than the 'maximum error' distance away from the original path, while using the minimum number of points possible to render that generalised path

5.6 Geographic Coordinate Datum and Precision

The geographic coordinates of spatial features gets stored in the database in terms of latitude and longitude. These coordinates get converted on-the-fly to an arbitrary system we use to render the SVG (see Section 5.4.2 as to why and how).

Remember that both the datum and the absolute precision of coordinates are not important, as all maps are generated with relative coordinates. Precision need only be supplied to the nearest metre, as that is all the application requires.

6. Coding Challenges

6.1 Incorporating Gzip Encoding

One thing we noticed was the large size of the raw SVG documents being produced. Of course, there is the .svgz (Gzip – as detailed in [SVG1.0-REC] section 1.2) option. How do we do this using Perl?

  print "Cache-Control: max-age=3600\n";
  ($USE_GZIP) && (print "Content-Encoding: gzip\n");
  print "Content-Type: image/svg+xml\n\n";

Figure 7: Example perl code to render HTTP Headers for compressed transmission

($USE_GZIP is set according to the desire to use gzip-encoding for your output. )

The Cache-Control header is an attempt to indicate that all rendered maps are valid for at least one hour. Sadly, the Adobe SVG Viewer doesn’t appear to honour this – instead it re-fetches from the server every time, even when going “back” to a page with SVG embedded in it. We do not know of a way to work around this at present; perhaps the parent “EMBED” element needs to be changed instead. Perhaps the use of the "OBJECT" could be investigated instead of "EMBED" - this is a subject for further study.

  use Compress::Zlib; # from CPAN
  use Carp;

  sub Cache_Print
  # A filter to printing SVG output
  # so that we may add GZipping or
  # (in future) even accumulate the
  # Content-Length.
  {
    my($text) = @_;
    if ($USE_GZIP)
    {
      if (!$gzip)
      {
        # first time Cache_Print's been called
        binmode STDOUT;

        # note $gzip becomes global here
        $gzip = gzopen(\*STDOUT, "wb")
          or croak
          "Cannot open stdout: $gzerrno\n" ;
      }

      $gzip->gzwrite($text)
        or croak
        "error writing: $gzerrno\n" ;
    }
    else
    {
      print STDOUT $text;
    }
  }

  sub Cache_Print_Cleanup
  {
    if ($USE_GZIP)
    {
      $gzip->gzflush(Z_FINISH);
    }
  }


  #
  # Declare the following whereever you use Cache_Print
  #
  sub Cache_Print;

Figure 8: Example perl subroutines to use instead of print to render compressed output

Elsewhere in your code you substitute Cache_Print for print, and add in a call to Cache_Print_Cleanup when you are certain you have no more output to write.

Once you use gzip there seems to be no reason to go back, except for troubleshooting purposes. For example, during our development some perl DBI calls were failing – causing errors to be written on stderr instead of stdout. Since apache seems to combine the two streams before transmission to the user agent, it results in the compressed SVG document corrupting part way through. You can only use the user agent's 'view source' command to determine the error message once you revert to non-gzip output.

7. Implementation-Specific Adjustments

7.1 Printing

7.1.1 Standalone Versus <EMBED>ded SVG Documents

Under old-fashioned browsers , standalone SVG documents seem to scale out to the dimensions of the paper they are being printed on, usually destroying the aspect ratio of the drawing in the process.

To work around this, QROTI currently <EMBED>s all of its SVG maps. Other solutions may also be available, such as using <OBJECT>, since <EMBED> is also a symptom of using the abovementioned old-fashioned browser .

7.1.2 Default Rasterisation Resolution

7.1.3 Internal Versus External Style Sheets

Also, the ASV 3 and IE5.5 combination didn’t seem recognise the external style sheet declaration upon printing the document. Instead, styles for SVG drawing primitives reverted to their defaults. This is unfortunate as our CSS is identical across all our maps.

The $STYLESHEET variable is either set to "EXTERNAL" (QROTI's preferred value) or "INTERNAL" depending on how you want to reference your style sheet. I kept this conditional processing to give us the future option of detecting user agents, and changing the script's output depending on the known conformance level of that user agent.

Of course there may be other more efficient workarounds available, but this solution will be enough for the moment.

8. Impediments to User Acceptance

Much as I want like to deploy SVG once and have all graphical user agents be above to receive my message, several impediments still remain to SVG becoming a well-regarded standard on the internet. These are my opinions only, not researched facts:

8.1 Market Penetration

Since SVG is still a new technology, the penetration of user agents still leaves something to be desired. In my experience, it is still often not included in the standard builds for business workstations, as opposed to the Macromedia Flash Player which is more well-known. However, if we keep writing compelling content in SVG (such as my contribution), then awareness and acceptance of the technology should follow.

A workaround for those users unable to display SVG could be to use the Apache Batik (URL: http://xml.apache.org/batik/) transcoder locally on the web server machine. This could take piped input from the server script and convert it to PNG (Portable Network Graphics) . This could either be done on the fly, but you revert to the problem of rasterising the map at the server-end. A second option could be to run an offline batch job of all the common SVG views through the transcoder, and put the results online as static PNG files. QROTI hasn’t used these workarounds as we haven’t had any user demand for it.

8.2 User Agent Speed

Rasterisation and display on the user agent can take a relatively long time compared to simply rendering a PNG image of equal pixel dimensions on the user’s display. Empirically speaking, this can be a problem for lower-end computers, for example at the vintage of machines with 300 MHz (megahertz) i386 (Intel 386 Family) machines.

At the time of preparing this paper, research still needs to be done to establish why this is happening. However, we suspect it is because QROTI’s maps liberally use text-on-a-path features (for labelling street segments) and anti-aliasing (for superior visual quality. These are two qualities I would need some convincing to trade for anything, especially since user-end computers continue to get faster.

However, a positive trade-off is that while the initial load may take longer than the equivalent raster image load, subsequent panning or zooming will be, of course, a lot quicker.

Perhaps a day will come where computers come with graphics chipsets capable of SVG hardware acceleration.

9. User Feedback

The QROTI website includes a link to its feedback form at the footer of every page. However, remarkably few people have used it to make comments on their experience with SVG. This is generally good news as it is currently the second most popular script on the site (in terms of raw hits) but generates a smaller proportion of feedback responses.

10. Future Directions

There is still a way to go before all of my ideas on SVG are exhausted. Here is a selection of possibilities:

10.1 Conversion of internal coordinates to floating point

Unfortunately for path data, the conversion from the mySQL stored format to text (for use inside SVG path data) occurs early in the current algorithm. For features like dynamic labelling to work (see Section 10.2 ), path data needs to stay in floating point format through the stage when label positions are calculated.

Early tests show the difference in speed between the two formats is possibly twenty-fold in favour of floating-point. Presumably this is because Perl requires numbers (stored as text) to be converted back to floating point - just in time - for each numerical calculation.

10.2 Dynamic Labelling

Early prototypes (not ready for publication) exist to add dynamic labels to QROTI SVG maps. The main purpose for these labels would be to indicate origin and terminus points of a route, as well as named bus stops along the way. The algorithm works well in terms of positioning quality, but is very slow.

One challenge that arises in the dynamic labelling problem is that there's no simple way on the server-side to determine if two SVG elements overlap each other. One could imagine having to recreate most of the code required to operate an SVG user agent in order to acheive a rigorous overlap-detection test.

One CPAN package that I've already identified as being useful is Font::AFM. You can use this to determine the actual widths of blocks of text in SVG. Therefore the server-side can 'think-ahead', avoiding the overlap of two separate text blocks.

10.3 Treatment of the Highlighted Area of Interest

Currently the highlighted bus stop or railway station appears as a slowly flashing point. This is inadequate in crowded or zoomed-out maps as the point is very small, making the flashing imperceptible. Another disadvantage is that printed versions, of course, lose any animative nuances.

10.4 Orientation of Maps

This option could be combined with a new map view to 'show where all routes go from this stop', which could be useful in adapting the QROTI mapping engine to preparing bus route maps for displays on-site at bus stops.

As part of preparing this new map view, the map of bus routes should be rotated so that the initial part of each route projects 'upwards' from the bottom of the map. This rotation of the map to the 'next stop faces up' direction would help intending passengers get a better idea of where each route travels as it departs the bus stop of interest.

10.5 Side-by-Side Route Drawing

Currently if several routes travel on the same segment of road, the displayed width of the road is not changed to indicate this. Most manually-produced schematic maps show concurrent routes as differently-coloured parallel lines. This is possible to do automatically, but requires a robust displacement algorithm.

10.6 "Routes on a Road" Map View

This enhacement will be required to properly respresent hail'n'ride services, that is, services in low-density areas that typically operate without formal bus stops. Once implemented, a user will be able to click on any road with a bus route (as well as the present-day ability to click on formal bus stops).

10.7 Indicating Volume of Service

Once the full corpus of bus timetable information is loaded into the system (still in progress as I write), a new dimension of maps can be produced. For example, the width of marked routes and the radius of bus stops could be varied to indicate the frequency of service through that point.

This has the potential to address problems such as that relating to real estate purchasing decisions; the user of the map could visually ascertain if a public transport route really is well serviced without resorting to the full timetable.

11. Wish Lists

Here we describe some limitations of the implementation and specification that would be useful addressing.

11.1 User Agents

11.1.1 Tool Tips

As far as we are aware, older versions of the Adobe SVG Viewer (version 2 as distributed with the Acrobat Reader version 4) didn’t honour the title element with a tool tip. Tool tips can be rendered purely in SVG, but are cumbersome in our situation where perhaps 1000 named features can appear on a map.

As a workaround, we used the status bar of the enclosing window of the user agent as a de-facto 'extra information’ area, such as:

  ...
  <defs>
   <title> <path
            id="pl1456" class="pRd"
            d="M12.901,42.778L12.865,42.791" />
  </defs>

  <g clip-path="url(#clipbox)">
   <use xlink:href="#pl1456"
        onmousemove="window.status='BLUNDER ROAD'"/>
  </g>

  <text>
   <textPath xlink:href="#pl1456"
             method="stretch"
             startOffset="50%"
             class="tSt">Blunder
             Rd</textPath>
  </text>
  ...

Figure 9: SVG code fragment, demonstrating using the status bar to display extra information upon mouse rollovers of map elements.

11.2 SVG Specification

11.2.1 Z-Order

When deciding upon an SVG rendering order, we have to remember what motivates the user to enter the web page that contains our maps. Generally they are not quite sure what to expect; if the user already knew, why would he/she still be visiting your web page? Take the example of bus route maps: In my estimation, an advanced user can get the context of a map purely from viewing the route shape itself without having to wait for suburb labels, surrounding streets, and so on. However, new users or tourists probably also need the whole complement of surrounding major roads, feature labels, water features and minor streets before they become confident about what the map represents.

The most value comes out of rendering the bus route diagram first, since at that point in time the advanced user can truncate the SVG rendering stream and move onto something else. New users still have the option to continue waiting while more of the surrounding context is built up around the route diagram.

However the present 1.0 specification offers only the “subsequent elements are painted on top” model ( [SVG1.0-REC] section 3.3). In our ideal case, however, the route diagram should remain 'floating at the top’ even as additional context gets rendered around it.

Therefore we'd love to see a concept of Z-order introduced into drawing elements. We understand this option is already being considered for future SVG Recommendations ( [SVG1.2-WD] section 8.2):

12. Conclusion

This paper describes a useful tool for visualising public transport information. It retrieves and displays the appropriate information in a competent fashion, if a little slowly. Hopefully you are inspired to consider how SVG can transform your SVG, I consider it a showcase of the kind of richly interactive experience that the SVG specification promises. Hopefully you are inspired to consider how SVG can transform your SVG,

Hopefully you are inspired to consider how SVG can transform your website, or even found it useful in cracking a stubborn problem. If you found this paper to be useful to you, want to read more or are considering using this kind of application in your own region then feel free to let us know with a QROTI Feedback Form (URL: http://qroti.com/about/feedback.shtml).

A Public Transport Mapping Example Using perl and mySQL

Prepared for the SVG Open 2003 Conference, 13-18 July, 2003

Abstract

Table of Contents