For this experiment, I am measuring the maximum update frequency of an SVG document. I am determining this frequency by counting the number of times a function is executed using the setInterval() ECMAscript function. I am not measuring the actual rendering frequency, because I have no way to directly determine that from script.

I am also explicitly not measuring either CPU or memory usage in these experiments. Although these performance measures are important and worthy of study, I felt that focusing on one measure at a time would yield more concrete results.

Another approach to script-based animation is to use setTimeout() to fire an update method. To make the animation continuous, the update method must reschedule the update with setTimeout() again. In theory, this would make the wait time be less dependent on the amount of work being done during the update. Unfortunately, this approach is subject to a bug reported in Firefox. [1] In my tests, there was no sign of symptoms of this bug. However, this is a valid animation technique that should be explored in a later test.

Absolute performance measures are incredibly hard to reproduce (or understand). So don't take the absolute numbers too seriously. The more important measures are relative. How do different interval sizes effect the update rate? How does one viewer perform relative to another (at least on my platform)?

I have run my performance tests on three different viewers (Batik 1.8 pre-release, Firefox 3.03, and Opera 9.6) with three different SVG applications. Each viewer is run with all three applications and the results for each application are compared. The first application is a reference application that does minimal work in order to establish a benchmark of the maximum update rates for each viewer. The other two applications are demos that I have had on my website for years. They have both been modified to contain code that does the profiling.

The reference application establishes a baseline update rate that is effectively the fastest the viewer can achieve. This gives some measure of the overhead of the profiling code on the viewer under test. The difference between this baseline and the measurements for another application shows the cost for the script actually executed in that application.

The control application (labeled Minimal) has a black rectangle as a background and one text area that is updated by the profiling code (just like the other two). On each update, the application calls the profiler to update the current count. No modifications of the DOM are made and no real work is performed by the script.

The next application is a profiled version of my Instruments demo. This application manipulates a set of SVG objects designed to mimic typical reporting instruments: bar graphs, radial dials, strip charts, and numeric readouts. This test does a small amount of calculation in the script and mostly performs scaling, rotation, and text updates to the various SVG objects.

The final application is a profiled version of my Wuse program. This application is basically nothing more than eye candy, but it does exercise the SVG scripting in a much different way than the Instruments demo. On each update, this application performs some calculations, updates the position of one end of a line and modifies the color of several lines.

Obviously, the measurements only make sense in the context the platform I am using for the tests. My tests were run on Ubuntu running on a 1.66 GHz Core 2 Duo machine. Batik was run with Sun's Java version 1.6. The numbers on a different machine could be very different.

One might ask why we should benchmark script-based animation. After all, SVG supports a very powerful declarative animation system using SMIL. There are several answers to this question.

• SMIL is not supported in all viewers.

• Many animation techniques (such as data-driven animations), cannot be accomplished through SMIL alone.

• There are many other applications for using script to manipulate the SVG document besides animation.

For these reasons, scripting will probably remain a useful tool in the SVG developer's tool chest. Testing the performance of script-based animation gives some information on how useful the technique is for many applications.

I had a few applications that were already doing script-based animations. One of them, Wuse, had performed miserably on early versions of the Firefox browser. This was not unexpected since SVG support was very new at that point. I recently opened Wuse again in a more modern Firefox and was surprised at the improvement in the performance. This lead me to wonder how it compared to other SVG viewers that I have access to.

Rather than spend a lot of time trying to develop a powerful, comprehensive method for profiling the code, I decided to stick with a simple counter-based approach (testing methodology). It has the following advantages:

Using this profiler we can easily test how changes to the SVG modifies the performance characteristics.

The goal is not to provide a definitive answer about the speeds of the various viewers or to minutely measure the performance of various techniques. The idea is to determine if script-based animation in SVG is good enough for different purposes. It also gives a more analytical measure than it seems faster.

The use of this profiling approach does rely on some assumptions. The code assumes that the viewer calls the scheduled update function at an interval that has some relation to the requested interval. The approach also assumes that the Date.getTime() method returns a value related to the actual time. We would also like to assume that the viewer renders the changes some time shortly after the changes are made to the image.

David Dailey has reported that the update function driven by setTimeout() will only be executed after the rendering for the previous update is complete. [2] Further testing would be needed to prove the same effect applies for setInterval(). Dailey's paper performs a similar form of profiling. In his paper, the experiments measure differences between the expect time for updates to occur and the actual time.

The Dailey paper also focused more on testing individual techniques than on larger applications. My experiments used an automated approach to gather more data at a time, based on the (possibly invalid) assumption that I could not reasonably measure rendering. The Dailey experiments used a more flexible means of modifying parameters to test a larger number of variations in a single test. The results described here can be seen as somewhat complementary to the results from that paper.

In order to profile an application, you need to identify what is being measured. See Profiling Assumptions.

The SVG files being profiled use a relatively standard approach to script-based animation. The setInterval() function is used to schedule an update function to be called every x milliseconds. Each update will take some period of time to execute. If the update takes more time to execute than the interval size, the next update is delayed.

This gives an obvious approach to profiling the code. Count the number of times the update method is called. This count is then sampled in another function scheduled by a second call to setInterval() with a much longer interval. The change in the count from the last value divided by the amount of time since the last sample gives a good average update rate.

By varying the time interval used for the update method, we can study the relation between requested update rate and actual measured update rate. The update time interval was varied in 5 ms increments in the range of 5 ms to 50 ms.

In this thread from mozilla.dev.tech.svg, Boris Zbarsky explains how the setTimeout() and setInterval() methods work on Gecko-based browsers. [5] One interesting datapoint that has a major bearing on this experiment has to do with scheduled code that takes longer than the defined interval. Instead of shifting the interval by a small amount to compensate for the overrun, the Gecko engine (at least) skips the update at the interval that was overrun and lets the update happen at the next interval. In other words, if we have an interval of 20ms and a function that runs reliably in 21ms, we can expect the update function to be called at 20ms, 60ms, 100ms, etc. skipping every other interval.

In addition to the update method scheduled using setInterval(), another function was scheduled at a longer interval of 10 seconds. This 10 second sampling interval seemed long enough to reduce the sampling error. The time difference between calls to this sampling function was measured using the Date.getTime() method. The difference in the count generated with the update method was also measured. The count divided by this time difference gives an average update rate. A number of samples like this were taken at each update interval. These samples are averaged to give a value for each interval size.

The data at each sample were written to a text element to allow the user to gauge progress. In addition, the data was sent as parameters in a request to an HTML page. The results of this page request were ignored. The only purpose in this request was to generate a webserver log entry for the request. The data would eventually be harvested from these log files.

Three different applications were tested to provide measures of different amounts of work. The same data for each application was collected for three different viewers. All of the data was graphed to allow visual comparison of the data.

Another script makes a new page request after a given (long) period of time. This allows for automated runs of the test on a given viewer. The script cycles through the interval sizes for a given application and then moves to the next application. This gives repeatable results without me needing to sit in front of the computer and change the intervals and applications. The only difficulty with this approach was that the functionality to request a new page was missing from Batik. [6] A patch was submitted for the 1.8 pre-release version to support this functionality.

When measuring complicated systems, there are a number of different sources for error. By using statistical methods, it is still possible to get reasonable information. See Sources of Errors for more information.

The data for a given interval contains a fair amount of variation. There is also a bit of error in the sampling interval. The data for a given interval is averaged to give an update rate for the interval. Three different sampling approaches were explored to reduce the variation in the data.

Each of these sampling strategies were tried and three statistics were calculated: mean, median, and standard deviation. Comparing the mean and median showed that the two were relatively close. In addition, sometimes the mean was larger and other times it was the median. There did not seem to be any systematic bias between the two.

Since the measured update rates cannot be negative and the fact that the mechanisms of bias should be mostly random, assuming a gamma distribution seems fairly reasonable.

If the mean of a gamma distribution is large enough, it tends to look something like a normal distribution, the standard deviation of the data should give an idea of which sampling approach would be the most stable. Of the three sampling approaches, the third gave the lowest variation.

In the end, the measurements were done using the third sampling method with N equal to 50. So, at least 52 samples must be taken.

Although there is a lot of variation between the three viewers tested at very high update rates, they all performed fairly consistently in the 20 ms to 50 ms interval size. As expected, different work in the scripts has a very strong effect on the update rate. As long as the interval size does not become too small, the viewers could sustain update rates around 20 to 50 updates per second or better. This update rate is high enough to generate fairly smooth animation.

Someone might question the need for update rates faster than the frame rate of the monitor displaying the SVG. Most movies use a frame rate of 24 fps. Monitors are actually capable of refresh rates of between 50Hz and 120Hz. This gives a fairly wide range of potentially useful update rates.

Choosing an optimal update rate is would depend on how quickly a human observer can detect the change. Updates that occur faster than a person can detect could be considered to be a waste of resources. However, a person's ability to detect changes is actually a very complicated problem. [7] There does not seem to be any firm value that we can call the fastest refresh rate that matters.

Instead of making an arbitrary cutoff, I decided to make a series of measurements to determine what was possible. While a faster update rate may not be directly useful, we might expect that a viewer capable of faster update rates may be able to generate smoother animation. Moreover, if a particular viewer can do a given amount of work with a very high update rate, it may be able to do more work while still sustaining a reasonable update rate. Proving this would make a good experiment for a different time.

The only results worth reporting are ones that others can reproduce, so I'm including the materials I used to run the tests and my initial sets of result data in this tarball. To duplicate the tests, you will need access to a webserver where you can put the appropriate files. You can then follow the instructions in the enclosed README file.

Now that the infrastructure and profiling tools have been written, there are many other directions that can be explored. The most obvious is to profile more SVG files, with different mixes of manipulations.

Tests comparing different image manipulations with similar effects might be interesting. For example, a comparison of changing the center of a circle vs. using the transform attribute to translate it.

Measurements of different filter effects could also be interesting.

There have been a number of reports that some viewers (notably Batik) are slow with some text operations. These tools could be used to give some solid information about this issue. We could also add performance information to the ongoing debate of the difference between presentation attributes and CSS styling.

Obviously, tests of more viewers and platforms would be useful. Although we can only usefully compare viewers on the same platform, this kind of profiling is still useful to gauge the feasibility of different applications.

Although the current tests are somewhat automatic, more automation of the testing would definitely be a benefit to anyone duplicating this test. At the moment, the post-processing stage is somewhat primitive and could benefit from more work. Better analysis of the data might yield more useful information from the data logged in the test.

Finally, a few longer test runs are needed to help validate the assumption of a gamma distribution of the samples.

More testing of intervals close to the 10ms estimated cutoff for Firefox and Opera would be useful to establish the actual value of the cutoff.

It would probably be worth testing the setTimeout()-based scripting approach. That would allow comparison with the current approach. It might also be possible to test the Firefox bug described earlier. [1]

A test that verifies whether other viewers treat interval overrun the same as Firefox would be useful.