DB2 9 and pureXML : Part 1

I'm not going to start this post by speaking about how important XML is, or how it is the backbone of the service oriented architectures of so many enterprises. I'm not going to get into this, not because I disagree with these statements, but because I think they are generally accepted notions in today's IT industry. I think what is important to talk about is how we can more efficiently work with this technology, and how we can further integrate it into the enterprise information archictecture.

IBM is not about to be left behind and they have recently released DB2 9 (code-named Viper) which includes something they have coined 'pureXML'.

XML data requires the same coveted qualities of service that relational databases provide: high availability, reliability, protection, and performance. The pureXML™ technology in DB2 9 unlocks the latent potential of XML by providing simple efficient access to XML with the same levels of security, integrity, and resiliency taken for granted with relational data. With pureXML innovation in DB2 9, your organization can expect breakthrough increases in availability, speed, and versatility of your XML information, along with dramatically reduced development and administrative costs.

With the addition of a new 'xml' datatype you can now store XML content in a table. Having this option allows you to overcome many issues that arise when working with XML. Having the ability to store XML in a table, rather than files, we can now write more scaleable applications, and deal with concurrency and security issues more efficiently.

Besides these advantages, the addition of pureXML let's us actually query the XML data that is stored in the table with regular SQL (SQL/XML a.k.a. SQL with XML Extensions) or XQuery. In this first post of a two part series, I will explain how to query data stored in an XML column of a DB2 9 table using SQL and SQL/XML.

Let's say that we have a table defined as follows:

CREATE TABLE Catalog (
   id INT PRIMARY KEY NOT NULL,
   catalogname VARCHAR(40),
   company VARCHAR(40),
   contents XML
)

Now let's say that the'contents' XML column contains the following data:

<?xml version="1.0" encoding="UTF-8"?>
<Catalog>
   <Product>
      <ProductID>100</ProductID>
      <ProductName>Columbian Coffee</ProductName>
      <ProductDescription>A bold blend of Columbia's finest beans.</ProductDescription>
      <Price>7.99</Price>
   </Product>
   <Product>
      <ProductID>200</ProductID>
      <ProductName>Kona Coffee</ProductName>
      <ProductDescription>A mild blend of Hawaii's tropical java.</ProductDescription>
      <Price>10.99</Price>
   </Product>
</Catalog>

Despite having the new XML column as part of the 'Catalog' table, you can still make regular SQL queries such as:

   SELECT * FROM Catalog
   WHERE company = 'World Coffes Inc.'

Obviously this query will return all of the rows where 'company' is equal to 'World Coffee Inc.'. Keep in mind that the data as stored in the XML column, 'contents' will also be returned by the query, letting the programmer handle parsing the markup in some traditional manner - which frankly, isn't very exciting.

But would you say if I could return the names of all of the products contained in the 'contents' XML offered by 'World Coffee Inc.'. Now that is pretty exciting!

Before we write a query to do this, let's define SQL/XML.

As the name implies, SQL/XML is designed to bridge between the SQL and XML worlds. It evolved as part of the SQL standard effort and now includes specifications for embedding XQuery or XPath expressions within SQL statements. XPath is a language for navigating XML documents to find elements or attributes. XQuery includes support for XPath.

Now, back to the query:

   SELECT xmlquery('$c/Catalog/Product/ProductName
   PASSING contents as "c")
   FROM Catalog
   WHERE company = 'World Coffee Inc.'

Now to explain what this query actually means. The first line uses the built in pureXML function, xmlquery to state what part of the XML we are interested in. We pass in the XPath query, Catalog/Product/ProductName, to specify what node we are interested in - in this case, the ProductName.

The PASSING contents as "c" simply lets us specify the table's 'contents' column as the source for the XML. The $c which is plugged in front of our XPath statement is simply a variable reference to the PASSING clause.

The rest of the query is nothing more than regular SQL syntax.

Let's look at another example where we want to return all 'catalognames' that contain products with prices over $10.

   SELECT catalogname
   FROM Catalog
   WHERE xmlexists(
      '$c/Catalog/Product[Price > 10]'
      PASSING Catalog.contents as "c"
   )

You can probably figure out the nuances of this query based on what you learned from the previous example. Something that you may not recognize is the xmlexists function which evaluates to 'true' if the XPath expression, Catalog/Product[Price > 10] is valid. So we are essentially ensuring that we are retrieving only the 'catalogname' where the product's price is greater than $10.

These two examples only scratch the surface of what can be accomplished in DB2 9 and pureXML, but I hope it has given you a taste of just how effective they can be. By downloading some of this XML processing to the database level, developers can take advantage of the database engine's power.

In Part 2 of DB2 9 and pureXML I will introduce some examples of querying XML data with XQuery and see why DB2 9 is considered a bilingual product. If you're interested in further reading, check out the IBM Redbook - DB2 9 : pureXML Overview and Fast Start.

Update Your Subscription

This is actually a housekeeping post, rather than a technical article. It is just to ask anyone who subscribes to this feed to use this new link in their aggregator :

http://feeds.feedburner.com/LetsGetTechnical.

I'm using the FeedBurner service so I can track statistics more accurately. If you haven't heard of this service, I encourage any blogger to check it out. Here's a little blurb from their site describing their service.

FeedBurner is the world's largest feed management provider. Our Web-based services help bloggers, podcasters and commercial publishers promote, deliver and profit from their content on the Web.

FeedBurner also offers the largest advertising network for feeds that brings together an unprecedented caliber of content aggregated from the world's leading media companies, A-list bloggers and blog networks and individual publishers.

Moving to XForms

In the early days of the world wide web, the user experience was more-or-less static - the internet was a way of sharing data among the academic community. As it matured, the web started to become more interactive and dynamic. Much of this was made possible by the user submitting information through HTML forms. Anyone familiar with HTML has no doubt come across forms.

In the early days of forms, data was usually passed to a CGI (Common Gateway Interface) program on a server, allowing programmers and webmasters to provdide the user with dynamic content. Today forms are still used, but programmers have stretched the technology, producing very complex interactions, especially in many e-commerce implementations.

Forms have come very far and have been successful in creating cutting-edge websites, but they do have limitations. The main problem with forms is that they mix presentation and content. Generally, good programmers try to separate presentation from data and behaviour (as dictated in the Model View Controller design pattern). With this lack of separation the forms will generally only render well in a web browser - forget about getting that form to look proper in a cell phone, PDA, or other devices.

In October of 2003, a W3C recommendation was made for XForms 1.0. The XForms standard is supported in many applications including Firefox, Mozilla, Seamonkey, and Internet Explorer if you install the XForms extension or related plugins. IBM defines XForms as :

...the next generation of Web-based data processing. It replaces traditional HTML forms with an XML data model and presentation elements.

W3 Schools notes these key factors of XForms.

• XForms is the next generation of HTML forms


• XForms is richer and more flexible than HTML forms


• XForms will be the forms standard in XHTML 2.0


• XForms is platform and device independent


• XForms separates data and logic from presentation


• XForms uses XML to define form data


• XForms stores and transports data in XML documents


• XForms contains features like calculations and validations of forms


• XForms reduces or eliminates the need for scripting


• XForms is a W3C Recommendation

Now that we've defined XForms, let's take a look at how it actually works through a simple example. Imagine that we had a traditional HTML form used for entering credit card information for an online order.

<html xmlns="http://www.w3.org/1999/xhtml">
   <head>
      <title>Enter Credit Card Information</title>
   </head>
   <body>
      <h1>Enter Credit Card Information</h1>
      <p>
         Enter your credit card information to complete the order.
      </p>
      <form action="http://localhost/imaginary-processing-engine" method="get">
         <p>
            Name : <input type="text" name="name" />
            <br>
            Credit Card Type :
            <select>
               <optgroup label="cardType">
                  <option value="1">Visa</option>
                  <option value="2">Mastercard</option>
                  <option value="3">American Express</option>
            </optgroup>
         </select>
         <br>
         Credit Card Number : <input type="text" name="cardNumber" />
         <br>
         Expiry Date : <input type="text" name="expiry" />
         <br>
         <input type="submit" value="Submit" />
         </p>
      </form>
   </body>
</html>

The form is pretty straight forward. It takes four data entries (Name, Credit Card Type, Credit Card Number, and Expiry Date) and submits them to an imaginary credit card processing engine running on the local server when the 'submit' button is clicked. This probably looks quite familiar to anyone who has used a traditional HTML form (I know it does to me).

When the 'submit' button is clicked the data is 'sent' to the server in a name / value pair. For example :

• name, Magic Johnson


• cardType, 2


• cardNumber, 4111111111111111


• expiry, 03-2008

This works fine - the programmer would just retrieve those values from the request object being passed to their program (such as a Java Servlet) and pass them along to the credit card processing engine. But what happens if the engine is fairly new and only understands XML messages? The programmer would have the daunting task of transforming those name / value pairs into XML. This isn't usually very fun.

That's where XForms comes to the rescue. Remember how they store and transport data in XML documents? Now your forms on your front end can send XML directly to your programs. Let's re-write our credit card form using XForms.

<?xml version="1.0" encoding="UTF-8"?>
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:xf="http://www.w3.org/2002/xforms">
   <head>
      <title>Enter Credit Card Information</title>
      <!-- form model -->
      <xf:model id="creditcardprocessing">
         <xf:instance src="creditcardorder.xml" />
         <xf:submission action="http://localhost/imaginary-processing-engine" method="get" id="submit" />
      </xf:model>
   </head>
   <body>
      <h1>Enter Credit Card Information</h1>
      <p>
         Enter your credit card information to complete the order.
      </p>
      <p>
         <!-- define inteface controls -->
         <xforms:input id="txtname" model="creditcardprocessing" ref="/CreditCardOrder/Name">
            <xforms:label>Name</xforms:label>
            <xforms:hint>Enter the name as it appears on the credit card.</xforms:hint>
         </xforms:input>
         <xforms:select1 model="creditcardprocessing" ref="/CreditCardOrder/CardType" appearance="full">
            <xforms:label>Credit Card Type</xforms:label>
            <xforms:hint>Select the type of credit card. </xforms:hint>
            <xforms:item>
               <xforms:label>Visa</xforms:label>
               <xforms:value>1</xforms:value>
            </xforms:item>
            <xforms:item>
               <xforms:label>Mastercard</xforms:label>
               <xforms:value>2</xforms:value>
            </xforms:item>
            <xforms:item>
               <xforms:label>American Express</xforms:label>
               <xforms:value>3</xforms:value>
            </xforms:item>
         </xforms:select1>
         <xforms:input id="txtcardnumber" model="creditcardprocessing" ref="/CreditCardOrder/CardNumber">
            <xforms:label>Credit Card Number</xforms:label>
            <xforms:hint>Enter the credit card number.</xforms:hint>
         </xforms:input>
         <xforms:input id="txtexpirydate" model="creditcardprocessing" ref="/CreditCardOrder/ExpiryDate">
            <xforms:label>Expiry Date</xforms:label>
            <xforms:hint>Enter the credit card expiry date (MM-YYYY).</xforms:hint>
         </xforms:input>
         <xforms:submit submission="submit">
            <xforms:label>Submit</xforms:label>
            <xforms:hint>Submit the order.</xforms:hint>
         </xforms:submit>
      </p>
   </body>
</html>

In the <head> element we define the form model and give it an 'id' to which we can refer to later. The <xf:instance src="creditcardorder.xml" /> node defines the data we are collecting. As you can see, I am referring to an external document called 'creditcardorder.xml'. This file looks like:

   <?xml version="1.0" encoding="UTF-8"?>
   <CreditCardOrder>
      <Name />
      <CardType />
      <CardNumber />
      <ExpiryDate />
   </CreditCardOrder>

This is what the XML will look like when we submit the form (but will not include empty tags - actual data will be in there). We could have also defined this XML structure inline, but I prefer having it in a separate file.

The <xf:submission action="http://localhost/imaginary-processing-engine" method="get" id="submit" /> node allows us to define how the form submission will be handled. As you've probably guessed, it includes the URL to where the form is to be submitted, as well as the submission method. Note that you can replace the URL with an actual file location, allowing you to save the form XML to a file. This could be handy if you want to let the user save a partially entered form that they can return to at a later time.

Let's take a look at how we actually build the form that will be rendered. The XForms standard provides a number of different form elements, such as text fields, radio buttons, checkboxes, etc. My example makes use of the <input>, <select1>, and <submit> controls. XML.com put together a nice table cross-referencing the traditional form controls to XForms controls. I suggest you take a look at it here.

Take notice of the ref attribute in each form control. It's value is an XPath - /CreditCardOrder/Name. The data that is entered for this form element by the user will be 'plugged' into the XML structure we defined in the XForm model. So if the user enters 'Magic Johnson', that value will be injected into the XML document at the referenced XPath, when the form is submitted.

One thing to note in the <xforms:submit submission="submit"> node is that we are referencing the xf:submission model we defined in the <head> with an id="submit". The significance of this is that we could have defined multiple models that submit to different servers. Or as I stated earlier, we could have included an additional button called 'save' that would save the form data to a file. Traditional forms only allow one submission scheme per form.

So how does this all come together? Well, if we were to enter the same data as I illustrated earlier in my name / value pairs, submitting the form would send an XML document to the server to be processed. That document would look like :

<?xml version="1.0" encoding="UTF-8"?>
   <CreditCardOrder>
   <Name>Magic Johnson</Name>
   <CardType>2</CardType>
   <CardNumber>4111111111111111</CardNumber>
   <ExpiryDate>03-2008</ExpiryDate>
</CreditCardOrder>

Now when our XML enabled program receives the request, no ugly transformations from name / value pairs have to be made. We can pass the XML straight through to our credit card processing engine.

As you can see through my simple example, XForms are a necessity in a software environment that has quickly, and decisively adopted XML. The XForms standard lets us create an abstraction between a form's data model and its presentation. It also adds the ability to integrate with XML schemas, providing simple, and accurate, data validation (which I didn't illustrate). I encourage you to read more about XForms, and start to think about integrating this technology into your current web solutions.

Here are some links I found quite useful while experimenting with XForms:

XForms Tutorial at W3Schools
XForms Basics at DevShed.com
What are XForms at XML.com
Introduction to XForms at IBM DeveloperWorks

Using Microsoft's Service Factory

It's obvious that the development of a new system is a lot of work. There are many things to consider when designing the system, and this can be very time consuming. Since many of the architectural hurdles you face have been met by others in the industry, it only makes sense to learn from their experiences. This saves you from making the same mistakes that others have already made. This saves you time. And that in turn saves you money.

Microsoft's Patterns & Practices provides recommendations on how to, "design, develop, deploy, and operate architecturally sound applications for the Microsoft application platform." Leveraging their 'software factories', architects and developers can spend their time on business opportunities and building an application that addresses the needs of their organization.

Microsoft patterns & practices contain deep technical guidance and tested source code based on real-world experience. The technical guidance is created, reviewed, and approved by Microsoft architects, product teams, consultants, product support engineers, and by Microsoft partners and customers. The result is a thoroughly engineered and tested set of recommendations that you can follow with confidence when building your applications.

I recently started a .NET design for a new system, and while researching 'best practices' I came across Microsoft's newest addition to their Patterns & Practices Software Factories - The Web Service Software Factory (a.k.a. Service Factory). Since I was designing an SOA (Service Oriented Application), this software factory fit the bill.

Not to get into the specifics of the system I'm designing, its main component is a web service that exposes various web methods for placing orders. Having a strong SOA understanding, I know the general architectural requirements for a web service driven application. What I wasn't familiar with was the slight nuances needed for designing one using the .NET framework (which I am new to). Discovering the Web Service Software Factory gave me the boost I needed.

Once you download and install the Service Factory and related Guidance Packages, you will have a new project type available to create in Visual Studio. Under 'Project Types' there will be a subcategory :

• Guidance Packages
• Web Service Software Factory (ASMX)
• ASMX Service (this is an installed template)

When you create a new Visual Studio project using the ASMX Service template, a default project structure is created for you, following the Service Factory architecture. Three specific layers are created, separating the components of your solution. Each of the three components should be cohesive, about the same level of abstraction, and loosely coupled. Below are description of each layer as defined by Microsoft.

Service Interface Layer: This layer defines the operations of a service, the messages required to interact with each operation, and the patterns by which these messages interact—these patterns are referred to as message exchange patterns. The service interface layer contains a service contract, which describes the behavior of a service and the messages required as the basis for interaction. The service interface layer may also contain a service adapter, which is used to implement the service contract and to expose this functionality on a certain endpoint.

Business Layer: This layer incorporates components that implement the business logic of the application. Simple services often require only a very simple business action, but services with more complex requirements may implement a Controller pattern or business rules for implementing service behavior. The business layer also includes business entities that are used to represent objects specific to a business domain. These entities can incorporate both state and behavior. This layer can also include business workflows. These define and coordinate long-running, multi-step business processes.

Resource Access Layer: This layer contains the logic necessary to access data. The layer also contains service agents, which isolate the nuances of calling diverse services from an application and can provide additional services, such as basic mapping between the format of the data exposed by the service and the format an application requires.

Under each layer, the Guidance Package creates separate projects representing the architecture of the Service Factory. For example, if your solution was named 'MyService', your solution would have a structure as follows:

Solution 'MyService'
   Source
      Service
         MyService.DataTypes
         MyService.ServiceContracts
         MyService.ServiceImplementation
         C:\MyProjects\MyService.Host\
      Business Component
         MyService.BusinessEntities
         MyService.BusinessLogic
      Data Access Layer
         MyService.DataAccess

By right-clicking on most of the projects, such as 'MyService.DataTypes', or 'MyService.BusinessEntities', you are able to invoke a wizard to create particular components of the application. By making use of these wizards, it makes it easy to stay within the Service Factory pattern architecture. Using the wizards can be a little confusing the first few times you use them, leaving you wondering what information you are supposed to enter. I made it through the first few times by referencing the help system that is included, as well as the examples provided.

The Service Factory is fairly flexible, allowing you to create your web service as you see fit, and to whatever level of complexity you need. For example you can define your messgae types following an implicit (parameter programming model) or explicit (message contract degin programming model) approach, depending on what your organization requires. And since services designed using the .NET platform default to SOAP Document-based message formatting, you're web service maintains its message-based architecture.

So far (in my early development using the Service Factory) I've been able to get away with using the unmodified version of the Software Factory. However it's good to know that I am able to customize it if the need arises. There is detailed instructions on how to :

• Modify Responsibilities for the Data Access Layer


• Modify Responsibilities for the Service Layer


• Create Custom Solution Structures


• Create New Guidance Packages that Include Your Configuration Settings


• Modify the Documentation

Adding this level of customization, many architects and developers may feel less apprehensive about using the Service Factory.

Like I said, I'm only early in the development stage of the web service I've been referring to. I haven't actually created any 'real' code yet, but have been using the Service Factory to aid in the design phase of development. As I try to create the data models and service contracts for the service, the Service Factory has been indispensible in helping me define the core components of my application. I'll be sure to add a follow-up post documenting my experiences once I start creating the meat of the service.

Using Google's Blogger Data API

Google's Blogger Data API allows you to write client applications to manipulate Blogger content using GData feeds. For those not familiar with GData (Google Data API), Google defines it as follows:

A simple standard protocol for reading and writing data on the web. GData combines common XML-based syndication formats (Atom and RSS) with a feed-publishing system based on the Atom publishing protocol, plus some extensions for handling queries.

Google provides client libraries in both Java and C#, allowing for easy retrievals, additions, and updates of Blogger feeds. You can also manipulate the feeds manually, providing XML representing an entry. Unless you already have a client that constructs this data, I can see no reason not to use the client library Google provides - it really makes working with the feeds easy.

One thing you'll have to keep in mind is that Blogger is currently going through a transition phase. As it is now, Blogger uses it's own credentials, but there is a new version which is still in beta, that uses Google accounts. This can pose problems during any authentication phases (such as an update to an entry of a feed). The API docs do a good job of explaining how to handle this issue, so be sure to take note of it.

Like I stated earlier, using the Google Blogger API is easy. There is a general flow your application will have to follow to communicate with Blogger.

1. Retrieve the feed URL for a blog.


2. Using that feed URL, make a request to retrieve the blog's feed.


3. Manipulate the feed, that is, add, update, or query a post.

Retrieving the feeds URL can be done by making an HTTP GET request to the blog's URL, that is, the one you would use in your web browser. Once you receive the response, you will have to parse the XML for the <HEAD> section. Here you will find the <link rel="alternate"> tags. Depending on which version of Blogger you're speaking to (current, or the new beta), there will be different feed URLs. Regardless of this, you should always use the Atom 1.0 version - may as well stay current.

Once you have the feed URL, you can use the Blogger API to get the actual feed. As Google explains in their documentation, you can write the following Java code to do this. Obviously you'll have to replace the URL string with the one you retrieved in the previous step. The credentials too will have match the Blogger account you are referring to.

URL feedUrl = new URL("http://www.blogger.com/feeds/blogID/posts/full");
GoogleService myService = new GoogleService("blogger", "exampleCo-exampleApp-1");

// Set up authentication (optional for beta Blogger, required for current Blogger):
myService.setUserCredentials("liz@gmail.com", "mypassword");

// Send the request and receive the response:
Feed myFeed = myService.getFeed(feedUrl, Feed.class);

// Print the title of the returned feed:
System.out.println(myFeed.getTitle().getPlainText());

To add a post to the blog via a web client, using the client library, you can use the following Java code as a guideline, as Google has provided (once again replacing the URL with the appropriate value).

googleService.setAuthSubToken(sessionToken, null);

URL postUrl = new URL("http://www.blogger.com/feeds/blogID/posts/full");
Entry myEntry = new Entry();

myEntry.setTitle(new PlainTextConstruct("Marriage!"));
myEntry.setContent(new PlainTextConstruct(
"<p>Mr. Darcy has <em>proposed marriage</em> to me!</p>
<p>He is the last man on earth I would ever desire to marry.</p>
<p>Whatever shall I do?</p>"));

Person author = new Person("Elizabeth Bennet", null, "liz@gmail.com");
myEntry.getAuthors().add(author);

GoogleService myService = new GoogleService("blogger", "exampleCo-exampleApp-1");

// Send the request and receive the response:
Entry insertedEntry = myService.insert(postUrl, myEntry);

The GData API also allows you to make queries against a feed based on a set of criteria you provide. This can range from searching for feeds with a specific instance of a word, or posts that fall between two dates. As Google has illustrated, the following code returns a feed containing the blog posts that fall between a specified date range.

URL feedUrl = new URL("http://www.blogger.com/feeds/blogID/posts/full");

Query myQuery = new Query(feedUrl);
myQuery.setUpdatedMin(DateTime.parseDateTime("2006-03-16T00:00:00"));
myQuery.setUpdatedMax(DateTime.parseDateTime("2006-03-24T23:59:59"));

GoogleService myService = new GoogleService("blogger", "exampleCo-exampleApp-1");

// Send the request and receive the response:
Feed resultFeed = myService.query(myQuery, Feed.class);

From the examples I have provided, I hope to have demonstrated how simple it can be to communicate with the Blogger service using the Google Blogger and GData APIs. For those who wish to integrate a blogging solution into their applications, these APIs provide a backbone to build upon. If you are serious about diving into these libraries, I highly suggest reading the full docs, as this post was simply to act as a primer, and get you excited. Keep in mind that Google also provides an API for Google Calendar. I've hacked around with it as well, and using it is very similar to the Blogger API. A GoogleBase API is also on the way, and I have the privelege of being a 'trusted tester', giving me early access. As the suite of Google APIs continue to grow, developers are getting the opportunity to push out some really kickin' apps.

Selective Copy Using Recursion in XSLT

As you continue to work with XML and XSLT, you may find yourself needing to copy an entire XML file, while suppressing certain elements. Take a credit card transaction represented in an XML file, for example. Let's say that once the transaction completes, a confirmation email is sent to the user. We'll assume that the confirmation email is generated by forwarding the transaction XML to some external web service as a message (perhaps JMS - Java Messaging Service). You feel that the email should contain all of the transaction information, but it would be more secure not to include the credit card number, for obvious reasons.

If you're using XPath 2.0, you have a fairly easy way to suppress nodes by using the 'except' operator. But if you're still using an older version (which the majority of the industry is at the moment), this task, that sounds as if it should be relatively simple, can become a quite difficult, especially if you're working with a complex XML document.

You may be able to get away with using :

<xsl:copy-of select="node1|node2|node4|node5"/>

The above would copy all nodes, except 'node3'. As you can imagine, this would become very tiresome, and difficult to write for complex documents. As I was fighting with this exact issue, I discovered a simple way to exclude nodes using recursion in my XSLT documents. Keeping with our credit card transaction example, let's consider the following XML.

<?xml version="1.0" encoding="UTF-8"?>
<transaction orderID="100">
   <LastName>Smith</LastName>
   <FirstName>John</FirstName>
   <CreditCardType>VISA</CreditCardType>
   <CreditCardExpMonth>10</CreditCardExpMonth>
   <CreditCardExpYear>2010</CreditCardExpYear>
   <CreditCardNumber>4111111111111111</CreditCardNumber>
</transaction>

Obiviously, this is a very simple document and it would be easy enough to state what nodes to copy, as I did previously. But to make things easier to follow, let's use it for illustrating the recursive solution.

By using the XSLT below, the complete XML file representing the transaction will be copied.

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
   <xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes"/>
   <xsl:template match="/">
      <xsl:apply-templates select="Transaction" mode="RecursiveDeepCopy" />
   </xsl:template>
   <xsl:template match="@*|node()" mode="RecursiveDeepCopy">
      <xsl:copy>
         <xsl:copy-of select="@*" />
         <xsl:apply-templates mode="RecursiveDeepCopy" />
      </xsl:copy>
   </xsl:template>
</xsl:stylesheet>

But we're not done yet. We still need to exlude the credit card number. This can be accomplished by adding another 'template' whose 'match' value is the name of the node we want to suppress. Since we are going to leave the template empty, the node will not be copied.

<xsl:template match="CreditCardNumber" mode="RecursiveDeepCopy" />

You can add these empty templates for any node you want to exclude. If credit card type was considered to be sensitive, we could simply add another template.

<xsl:template match="CreditCardType" mode="RecursiveDeepCopy" />

I personally found this technique for performing a deep copy extremely handy. In my case I was generating multiple versions of a feed using the same XSLT, but certain ones needed to have specific private information blocked (such as email addresses, etc.). I struggled with this problem, and settled for writing complex XSLT templates. However, once I discovered the recursive method, everything was smooth sailing.

XSLT Key is the Key

Anyone who has used XSLT (eXtensible Stylesheet Language Transformations) knows that it can be a very powerful template language. XSLT is generally used for the transformation of an XML document into another type of document. I've used it to transform XML into other formats of XML, HTML, SQL, and even NAnt build files. As you start to use XSLT, it can be a little difficult to grasp the 'template' way of thinking, but with a little practice (as I've discovered), you'll find that it becomes quite simple. And as you work more and more with the technology, you'll start to discover the different elements and functions that will help you write more efficiently.

During my early XSLT experience, I found the key() function to be indespensible. W3 Schools defines this function as follows:

The key() function returns a node-set from the document, using the index specified by an element.

This definition may not make much sense on its own, so I'll provide an example. Let's say we have an XML document as displayed below that describes countries, territories, and cities in the world. The document also contains telephone information for each of the territories.

<?xml version="1.0" encoding="ISO-8859-1"?>
<World>
   <Country name="Canada">
      <Territory name="British Columbia" />
         <City name="Vancouver" />
      <Territory name="Alberta" />
         <City name="Edmonton" />
      <Territory name="Saskatchewan" />
         <City name="Saskatoon" />
      <Territory name="Manitoba" />
         <City name="Winnipeg" />
      <Territory name="Ontario" />
         <City name="Toronto" />
      <Territory name="Quebec" />
         <City name="Montreal" />
      <Territory name="New Brunswick" />
         <City name="Moncton" />
      <Territory name="Newfoundland and Labrador" />
         <City name="St. John's" />
      <Territory name="Prince Edward Island" />
         <City name="Charlottetown
      <Territory name="Nova Scotia" />
         <City name="Halifax" />
   </Country>
   <Country name="United States of America">
      <Territory name="New York" />
         <City name="Albany" />
      <Territory name="California" />
         <City name="Los Angeles" />
      <Territory name="Arizona" />
         <City name="Phoenix" />
      <Territory name="Florida" />
         <City name="Miami" />
   </Country>
   <Telephone area-code="306" territory="Saskatchewan" />
   <Telephone area-code="519" territory="Ontario" />
   <Telephone area-code="905" territory="Ontario" />
   <Telephone area-code="780" territory="Alberta" />
</World>

Now let's say that we want to transform this document into the one below.

<?xml version="1.0" encoding="ISO-8859-1"?>
<Phonebook>
   <Country name="Canada">
      <AreaCode value="306" province="Saskatchewan" />
      <AreaCode value="519" province="Ontario" />
      <AreaCode value="905" province="Ontario" />
      <AreaCode value="780" province="Alberta" />
   </Country>
</Phonebook>

The XSLT used to do this would look something like the following:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
   <xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes"/>
   <xsl:template match="/">
      <Phonebook>
         <xsl:apply-templates select="World/Country" mode="CountryMatch" />
      </Phonebook>
   </xsl:template>
   <xsl:template match="Country" mode="CountryMatch">
      <Country>
         <xsl:attribute name="name">
            <xsl:value-of select="@name" />
      </xsl:attribute>
      <xsl:apply-templates select="Territory" mode="TerritoryMatch" />
      </Country>
   </xsl:template>
   <xsl:template match="Territory" mode="TerritoryMatch">
      <xsl:variable name="CurrentTerritory" select="@name" />
      <xsl:for-each select="../../Telephone[@territory=$CurrentTerritory]">
         <AreaCode>
            <xsl:attribute name="value">
               <xsl:value-of select="@area-code" />
            </xsl:attribute>
            <xsl:attribute name="territory">
               <xsl:value-of select="$CurrentTerritory" />
            </xsl:attribute>
         </AreaCode>
      </xsl:for-each>
   </xsl:template>
</xsl:stylesheet>

This works just fine, and you may not ever have a problem with it. However as the documents you are transforming increase in size, this technique may not work as efficiently as you would hope. The particular line I am referring to that may cause a problem is:

<xsl:for-each select="../../Telephone[@territory=$CurrentTerritory]">

Walking back node levels in the document to evaluate an expression can prove to be very inefficient when dealing with large documents. Consider that you had an XML document that contained all of the countries of the world, all of their territories (such as state, province, etc.), and all of their cities over 50,000 people. This would be a very large document. Now imagine the work the XSLT engine would have to perform to process the problem statement. For each territory in each country, the processor would have to back up two node levels and then search for each matching territory in each Telephone node - that's a lot of work.

I ran into this very problem recently when I was working with extremely large XML documents - around 350 MB. The transformations were part of a nightly NAnt build cycle and some were taking over an hour to finish. After looking into the problem, I discovered that the culprit was a similar statement to the one above. After a little research I realized that I should have been using the XSLT function, key(). As stated earlier, the key() function (in conjunction with the xsl:key element) finds the node with a given value for a named key. So, to make use of the key() function, the XSLT document from above would look slightly different.

Under the root element, World, you would define a key.

<xsl:key name="MatchedTelephone" match="World/Telephone" use="@territory" />

The 'name' attribute simply holds the name of your key. The 'match' attribute holds a match pattern identifying the collection of nodes where the lookups will take place. And finally, the 'use' attribute is what is used to find a matching node in your defined match pattern.

Now that we have a key defined, we can re-write our problem statement using the key() function as follows:

<xsl:for-each select="key('MatchedTelephone', $CurrentTerritory)">

What we are essentially doing here is saying, for each of the World/Telephone elements where the 'territory' attribute is equal to our current 'territory', create an AreaCode element.

Once I made these changes, the large document I was trying to transform came down from over an hour process to less than 5 minutes. The effects were incredible, to say the least (just keep in mind, that keys operate in the current context, meaning that they will not work with an external document, ie. using the document function). Like I stated earlier, if you're dealing with relatively small documents, you probably won't notice any type of performance gain, but once you get into large documents, the peformance benefits are huge.

The XPath to Cleaner Java Code

It was about three years ago that I got heavily involved in XML. The company I was working for at the time took the leap into SOA, and XML was a major component. I can remember the dilemmas I faced when trying to parse the XML messages being passed around from system to system. There were many options to dealing with the XML and after evaluating many of them, I decided to use the Document Object Model.

Granted, using the DOM gives you a lot of power, but writing the code to actually navigate an XML document can be a real hassle. And once you do finally get your logic in place, trying to come back to look at that code a few months later can bring on a serious migraine. No one said maintaining such code would be easy.

Now that I look back on those days, I often wonder why I didn't choose another API such as Xalan, or Saxon. Sure, I'd have introduced a dependency on an external engine, thereby locking me into their API, but I would have had the advantage of using XML Path Language, which is more commonly referred to as XPath. Since joing my new company, I have done extensive work using XPath (mostly while writing XSLT in a .NET 2.0 environment) and it was only at this time that I fully realized its power. It's safe to say that I didn't fully appreciate it during my initial evaluation.

For those who aren't familiar with XPath it is a powerful query language for extracting information from an XML document. Like SQL is a query language optimized for extracting data from a relational database, XPath is optimized for easily navigating an XML document to find the information you're interested in. IBM's technical XML library on their developerWorks network made a great analogy.

If you send someone out to purchase a gallon of milk, what would you rather tell that person? "Please go buy a gallon of milk." Or, "Exit the house through the front door. Turn left at the sidewalk. Walk three blocks. Turn right. Walk one half block. Turn right and enter the store. Go to aisle four. Walk five meters down the aisle. Turn left. Pick up a gallon jug of milk. Bring it to the checkout counter. Pay for it. Then retrace your steps home." That's ridiculous. Most adults are intelligent enough to procure the milk on their own with little more instruction than "Please go buy a gallon of milk."

This gives a great sense as to how much simpler it can be to write an XPath expression, as opposed to having to write complicated DOM code. Let's say I had an XML document that contained a list of cities, in a list of territories, and countries. If I wanted to find all of the cities listed in Ontario, my XPath statement would look something like this (assuming 'world' is the root element of the XML document:

//world/[country='canada']/[territory='ontario']/city

I'm not even going to get into writing the DOM code for doing something like this. I think you can imagine what it would look like. I sure can, because I got my practice writing a lot of it during the project I was referring to earlier. I guess if you were not going to have a lot of lookups, using the DOM would be fine and as a developer it is your decision to weigh your options. Do you want to write some code that will be difficult to maintain, or do you want to get locked into an external API and it's particular engine?

Well, with Sun's introduction of Java 5 this decision may have gotten a lot easier. This release of the Java platform now includes a package javax.xml.xpath that has everything you need to perform XPath queries right out of the box. No longer will you have to rely on an external engine to take advantage of XPath. I just wish it was available when I was trying to make that decision three years ago.

For further information on using javax.xml.xpath, check out IBM article, The The Java XPath API and Sun's official JavaDocs for Java 5.

AJAX Still Gaining Momentum

If you are a developer and haven't heard of AJAX (Asynchronous JavaScript and XML), perhaps it's time you crawled out from under that rock you live. AJAX technologies have been around for a few years now, but within the last 12 months or so it has really started to take off. In fact, all of the major players seem to have embraced the idea. If you head to Sun's Java site, the landing page is filled with links to AJAX articles, tips, and tools. In fact today, the site contains an announcement that Sun has joined the OpenAJAX Alliance and Dojo Foundation. I guess Sun thinks AJAX is relevant enought to "actively participate in these two communities to help drive open standards for AJAX programming and increase interoperability across AJAX technologies." And let's not forget the other major players; Microsoft's Atlas framework is offered for free and integrates seamlessly with ASP .NET 2.0. Google wasn't about to miss the boat either, recently releasing their Google Web Toolkit. GWT let's you "You write your front end in the Java programming language, and the GWT compiler converts your Java classes to browser-compliant JavaScript and HTML."

I really think that AJAX applications are going to continue to flood the web. All the coolest stuff out there seem to make use of it in some way or another, such as GoogleMaps, GMail and Google Calendar, and Zip.ca. As these applications find a place on the internet, gone are the days of differentiating between desktop applications and web applications - or at least we'll be a lot closer.

I have yet to dive into the world of AJAX from a hands-on perspective, but I have read a lot of material on the subject and have started to put together some preliminary work. Actually, I'm gonna take that back. I have done a significant amount of "AJAX" development, and this was about 3 years ago (before it was actually called AJAX). I developed a web application for three agriculturalists that allowed users to create quotes for crop hail insurance, compare prices, and finally purchase the insurance. The application made it possible to change different values, resulting in an update of all of the information on the page, without an annoying refresh.

In Professional Ajax, the Hidden Frame Technique is defined, and considered to be the first asynchronous request / response model for web applications.

As developers began to understand how to manipulate frames, a new technique emerged to facilitate client-server communication. The hidden frame technique involved setting up a frameset where one frame was set to a width or height of 0 pixels, and its sole purpose was to initiate communication with the server. The hidden frame would contain an HTML form with specific form fields that could be dynamically filled out by JavaScript and submitted back to the server. When the frame returned, it would call another JavaScript function to notify the original that data had been returned. The hidden frame technique represented the first asynchronous request/response model for web applications.

Even though this is far from the current AJAX frameworks which make use of an AJAX engine, I think it is safe to say that I was an early adopter of this revolutionary model. As I learn more about how to implement this framework, I can't help wonder how much cooler I could make the above mentioned hail insurance system. I've thrown the idea at the three guys who own the company, and a real interest was there after I explained the advantages it could offer. It also makes it a much easier sell when you show them some of those revolutionary web apps that are already out there.

We Have Lift-off

Information Technology has been a huge part of my life for sometime now. Since I was in grade school I was programming, and I guess it was only natural that this would blossom into a career. I'm thirty-one now and have been in the IT business professionally for about seven years. My career choice saw me move halfway across the country and back to where I grew up, letting me experience different parts of the country, as well as gain valuable technical skills.

To start my career off, I was part of the IT department of a large insurance / financial company, and I have recently joined a small, cutting edge software company. Needless to say, I have seen two extremes of the different IT work environments. I think testing out the waters of two different types of companies will help me make the final decision on what type of company I would like to spend the rest of my career with. At this point, it's a tough decision since both environments have so many different advantages.

Anway, I didn't want to get into my life story, but just wanted to start off with a brief background of myself. In terms of the type of technical work I've done, I've had the privilege of experiencing a wide spectrum of different technologies. I started off on a development team building a fat client using Java Swing. I then moved on to building distributed systems leveraging J2EE and web services. I now find myself wrapped up in the .NET world, heavily focussing on data syndication. In the mean-time, I always find time to do a little consulting side projects. These are usually J2EE web systems.

I'm lucky to say that I wake up everyday eager to get to the office to see what awaits me. I see no sign of slowing down anytime soon, and if that time every comes, I guess it's time to find a new career. Be sure to check back (or better yet, subscribe to this feed) for my upcoming technical articles.

Take care.