efficient xml

There are also four key performance “metrics” involved on the Win32 platform:

Working set:

The peak amount of memory used by MSXML to process requests. Once the working set exceeds available RAM, performance usually declines sharply as the operating system starts paging memory out to disk.

Megabytes per second:

The raw speed for a given operation, such as the document load method.

Requests per second:

How many requests the XML parser can handle per second. An XML parser might have a high megabytes-per-second rate, but if it is expensive to set up and tear down that parser, it will still have a low throughput in requests per second. For example, if the clients hit the server at a peak rate of one request per second, and if the server can do 150 requests per second, the server can probably handle up to 150 clients.

Scaling:

How well your server can process requests in parallel. If your server is processing 150 client requests in parallel, then it is doing a lot of multi-threading. Processing 150 threads in parallel is a lot for one processor, it will spend a lot of time switching between threads. You could add more processors to the computer to share the load.

The fastest way to load an XML Document
The fastest way to load an XML document is to use the default “rental” threading model (which means the DOM document can be used by only one thread at a time) with validateOnParse, resolveExternals, and preserveWhiteSpace all disabled, like this in Javascript:

var doc = new ActiveXObject
(“MSXML2.DOMDocument”);
doc.validateOnParse = false;
doc.resolveExternals = false;
doc.preserveWhiteSpace = false;
doc.load(“mystuff.xml”);

If you have an element-heavy XML document that contains a lot of white space between elements and stored in Unicode, it can actually be smaller in memory than on disk. Files that have a more balanced ratio of elements to text content end up at about 1.25 to 1.5 the UCS-2 disk file size when in memory. Files that are very data-dense, such as an attribute – heavy XML – persisted ADO recordset, can end up more than twice the disk-file size when loaded into memory.

Attributes vs. Elements

You could conclude that attribute-heavy formats (such as an XML – persisted ADO recordset) deliver more data per second than element- heavy formats. But this should not be the only reason for you to switch everything to attributes. There are many other factors to consider in the decision to use attributes versus elements.

Unique elements

My friend, in his honest but less than informed effort to create a useful XML document, made the mistake of attempting to use the XML elements as if they were unique “database fields”. For example if you have an XML Document that consists of survey questions, you could conceive that in order to make each element “unique” you would give the tag a unique name. So your survey questions document might end up looking kind of like this:

<POLLQUESTIONS>
<Q10000002>Who is central Florida’s best Internet Service Provider?</Q10000002>
<A10000009>MPINet</A10000009>
<A10000010>EarthLink</A10000010>
<A10000011>MindSpring</A10000011>
<A10000012>Access Orlando</A10000012>
<Q10000003>What is your favorite search engine?</Q10000003>
<A10000018>Yahoo</A10000018>
<A10000019>Altavista</A10000019>
<A10000020>Lycos</A10000020>
</POLLQUESTIONS>

Now what is wrong with the above document fragment? Two things, actually. First, it does not lend itself easily to XPATH statements that allow you to walk the DOM and find isolated nodes and / or subnodes of elements. True, each element has a “unique” tag name, but that’s not the point. XML is a tree-like hierarchical structure. If you need to be able to find an element or a node by number, or to sort, search or group, it’s better to use either an attribute () to identify the unique “ID” of elements or nodes, or to include a sibling element (1) inside each Question tag. You can also use the position() operator. The second thing that’s “wrong” is that the answer tags simply follow closed Question tags here — there is no closing “Question” element that encompasses both the question and it’s answers. A more productive version of the above might look like this:

<POLLQUESTIONS>
<Question QNum=”1″>
<QText> Who is central Florida’s best Internet Service Provider?</Qtext>
<Answer>MPINet</Answer>
<Answer>EarthLink</Answer>
<Answer>MindSpring</Answer>
</Question>
<Question QNum=”2″>
<QText> What is your favorite search engine?</QText>
<Answer>Yahoo</Answer>
<Answer>Altavista</Answer>
<Answer>Lycos</Answer>
</Question>
</POLLQUESTIONS>

Separate Memory Structure for unique elements

With the second example, we can find any question by its number using XPATH like: //Question[@Qnum=”2″]. We can sort, search, grab a question node along with its answers, and so on. And there is another very important but often overlooked reason to try and arrange your XML documents so that all the major tags have the same names: when the XML parser loads and processes your document, it creates a separate memory structure for each unique element name. So conceivably the first example above, if it had 1000 questions, could occupy orders of magnitude more memory than the second example with 1000 questions, taking a lot longer to parse, and possibly a lot longer to search or sort as well.

Walking the DOM tree for the first time also has an impact on the working set metric because some nodes in the tree are created “on demand”, they are not automatically “there” after loading the document. Creating a DOM tree from scratch results in a higher peak working set than loading the same document from disk. Loading a document is roughly five times faster than creating the same document from scratch in memory. The reason is that the process of creating a document requires a lot of DOM calls, which slows things down.

Walk Fast

The fastest way to walk the tree is to avoid the children collection and any kind of array access. Instead, use firstChild and nextSibling:

function WalkNodes(node) {
var child = node.firstChild;
while (child != null)
{ WalkNodes(child);
child = child.nextSibling;
}
}

However, if you are looking for something in the tree, the fastest way to find it is to use XPath via the selectSingleNode or selectNodes methods.

Free-Threaded Documents

The “free-threaded” DOM document exposes the same interface as the “rental” threaded document. This object can be safely shared across any thread in the same process. It can be safely stored in ASP Application state on IIS.

Free-threaded documents are generally slower than rental documents because of the extra thread safety work they do. You use them when you want to share a document among multiple threads at the same time, avoiding the need for each of those threads to load it’s own copy. In some cases, this can result in a big performance gain.

For example, suppose you have a 12K XML file on your Web server, and you have a simple ASP page that loads that file, increments an attribute inside the file, and saves the file again. Such ASP code is likely to be completely tied up with disk I/O. However, you could put the file into shared-application state using a free-threaded DOM document:

<%@ LANGUAGE=JSCRIPT %>
<%
Response.Expires = -1;
var doc = Application(“Stuff”);
if (doc == null)
{
doc = Server.CreateObject
(“Msxml2.FreeThreadedDOMDocument”);
doc.async = false;
doc.load(Server.MapPath(“stuff.xml”));
Application(“Stuff”) = doc;
}
Application.Lock();
var c = parseInt(doc.documentElement.getAttribute(“count”))+1;
doc.documentElement.setAttribute(“count”,c);
Application.UnLock();

%>
<%=c%>

This second approach using the free-threaded DOM document can easily be seven times faster than the other.

IDispatch

Late-bound scripting languages such as JScript and VBScript add a lot of overhead to each method call and property access in the DOM interface. The script engines invoke the methods and properties indirectly through the IDispatch interface and first call GetIDsOfNames or GetDispID, which will pass in a string name for the method or property and return a DISPID. Finally the engines package all the arguments into an array and call Invoke with the DISPID.

This is slower than calling a virtual function in C++ or compiled Visual Basic. For this reason, you may want to consider calling all your DOM functions from a “wrapper” compiled component that has the generic methods you need to do what you want, when using a script based application environment such as ASP. With VB, you want to also avoid late- bound DOM object invocation calls like the following:

Dim doc as Object
set doc = CreateObject(“Microsoft.XMLDOM”)

This will be as slow as VBScript or JScript. To speed this up, from the Project menu, select References and add a reference to the latest version of the “Microsoft XML” library. Then you can write the following early-bound code:

Dim doc As New MSXML.DOMDocument
Use XSL to get the data you need for speed

XSL can be a big performance win over using DOM code for generating “transformed” reports from an XML document. For example, suppose you wanted to show all the questions and answers matching a certain key word category element. You might use selectNodes to find all the questions matching the category, then use another selectNodes call to iterate through the answer elements of each of those questions.

But you could also write an XSL stylesheet:

<xsl:template xmlns:xsl=”http://www.w3.org/TR/WD-xsl”>
<xsl:for-each select=”/Question[CATEGORY=’ASP’]”>
<xsl:for-each select=”Answer”>
<xsl:value-of/>
</xsl:for-each><hr/></xsl:for-each>
</xsl:template>

You could then create your output with a function like:
function report(doc) {
var xsl = new ActiveXObject(“Microsoft.XMLDOM”);
xsl.async = false;
xsl.load(“pollquestions.xsl”);
return doc.transformNode(xsl)
}

This XSL transformation could be from 5 to 10 times faster than iterating through the DOC looking for your data!

The “//” Operator
The “//” operator walks the entire subtree looking for matches. if you are lazy like me, you use it more than you should because you are too lazy to look up and type in the full path. If you can, use the full path to get your data; it will typically give you up to a 15% performance boost. In editors such as XML Spy, there is a clipboard copy function that will return the XPATH statement to return any element.

Many of the items I”ve chosen to cover in this article, and others, along with real – world test results and detail, can be found in some excellent work by Chris Lovett of Microsoft. Also, author Kurt Cagle has some great work on using the IXSLTemplate interface to use cached template processors that will speed up repetitive transformations big time. You can find articles by both authors at MSDN online, by simply searching on their names.

Dr. Peter Bromberg is a Senior Programmer/Analyst at Fiserv, Inc. in Orlando and a co-developer of the http://EggheadCafe.com developer
website.