Typically when you poll data out of SQL it is a single table or the output of a stored procedure, this limited the type of data that could be returned, the example is that it was difficult to return the order header and all the detail lines with a single polling statement.  The XML polling feature of the WCF-SQL adapter allow you to create your own XML document in SQL and return it to BizTalk via a polling statement.

In this blog post I am going to show how to do this.

Background

I will be using the Sample AdventureWorks database and returning and Order XML document.

Install the AdventureWorks SQL sample database from http://sqlserversamples.codeplex.com/

Install the BizTalk 2010 LOB Adapter SDK and the LOB Adapter Pack (in not already installed)

In SQL

Create a SQL Stored Procedure to return the desired data, using the xmlnamespaces and for xml path.  You will need to decide when you create your stored procedure whether it will be returning a single Order or a batch of Orders, there will be not difference to the XML generated in the stored procedure just a difference in how you select the records to return.  The stored procedure that I have created returns up to 5 Orders per call.

(complete SQL Stored Procedure available in download below)

Grant the BizTalk Application User group access to the AdventureWorks database and execute privileges to the GetSalesOrdersToProcess stored procedure.

Execute the stored procedure to create a sample XML document and save it to order.xml

In Visual Studio

Create a BizTalk Project (I am using Demo.WCFSQL.Polling for the solution name and AdventureWorks.Schemas for the project name)

In the BizTalk project add a generated item and select generated schema, in the generated schema dialog select Well-Formed XML, then select the order.xml file saved above.  This will create an order.xsd in your BizTalk project.  You may also need to update any numeric fields to the correct data types, the generated schema wizard sets number to the smallest data type that fits the sample data.

If you decide when creating your stored procedure to return multiple orders per execution, you will need to create and orders.xsd envelope schema, like this:

And set the Body XPath to the root node Orders.

Strong name your BizTalk project and set the deployment application name (I am using Demo.WCFSQL.Polling)

This is the complete Solution

Build and deploy your solution.

In BizTalk Administration Console

Expand the application that you deployed your solution into (I am using Demo.WCFSQL.Polling)

Create Receive Port and Receive Location

You now need to create the one-way WCF-SQL receive port and receive location (I am using AdventureWorks.Orders.Polling as the receive port name and AdventureWorks.Orders.Polling.WCFSQL as the receive location name)

The receive location can either be created as a WCF-Custom with the sqlBinding or as a WCF-SQL adapter (I am using the WCF-Custom with the sqlBinding)

After selecting WCF-Custom, click Configure.

On the General Tab type in the Address (URI), this consist of the protocol, server name, database name and Inbound Id like below:

mssql://<servername>//<databasename>?InboundId=Orders (my example looks like: mssql://.//AdventureWorks?InboundId=Orders)

On the Binding Tab, select sqlBinding in the Binding Type drop down list

Set the following setting

Note: useAmbientTransaction defaults to True, if MSDTC is not configured between BizTalk and the source SQL Server you will have to set this to false

Click Ok, to close the WCF-Custom Transport Properties dialog

Select the appropriate Receive handler and set the Receive Pipeline to XMLReceive, then click OK to close the Receive Location Properties

Create Send Port

For this example we are just going to send the XML message directly out to a file location

You now need to create a one-way send port (I am using AdventureWorks.Orders.Send.FILE for the send port name)

Select a file location for the output XML messages (I am using C:\Working\Demo.WCFSQL.Polling\Out\%MessageID%.xml)

On the Filters tab, set the following filter: (this is for my example, yours may differ)

BTS.ReceivePortName == AdventureWorks.Orders.Polling

Click Ok to close the Send Port Properties dialog

Start the application

Right click on the Application and select start

Check the Output directory

You will notice with my example, 5 files are create at a each polling interval in the directory and that each file represents an order.

Note: there are over 31,000 orders in the AdventureWorks database.

Summary

I hope this example give you the information that you need to get started with XmlPolling with BizTalk and the WCF-SQL Adapter.

Code for this Sample

Bill Chesnut, Principal Consultant, Mexia

The underlying issue with the 2nd occurrence of this issue was a failed upgrade from 2006 R2 to 2009, although the upgrade seemed to work correctly, the user running the upgrade did not have sysadmin privileges on the BizTalk SQL Server, as required in the install instructions.  This lack of permission lead to the service account for the Tracking host not being given permission on the tables and stored procedures that are used doing the movement of tracking data from the message box database to the tracking database.  Once the permissions were update (cross checked permissions from a successfully upgraded 2009 BizTalk SQL Server), tracking data started to be moved from the message box database to the tracking database.

The 3rd occurrence is still under investigation, but the symptoms are the same, High CPU on the BizTalk server and large number of records in the tracking_1_x tables in the message box database.

From now on, one of my 1st checks for a poorly performing BizTalk system will be check the tracking_1_x tables and the TDDS_StreamStatus table.  I will be publishing a SCOM monitor to detect the tracking data not being copied and hopefully a rule for message box viewer that will clearly identify this issue.

xlang/s engine event log entry: Uncaught exception (see the ‘inner exception’ below) has suspended an instance of service ‘client.namespace.ProcessBatchRequest(8a00989b-e30d-c936-0b7e-0d20df729132)’.
The service instance will remain suspended until administratively resumed or terminated.
If resumed the instance will continue from its last persisted state and may re-throw the same unexpected exception.
InstanceId: 2dd299ec-0a2e-4505-8457-743a7e429cc9
Shape name: SerialiseResponse
ShapeId: cb127047-22bc-4df7-a1f9-7a8dd5d9879f
Exception thrown from: segment 1, progress 166
Inner exception: The underlying connection was closed: The connection was closed unexpectedly.
       
Exception type: WebException
Source: System
Target Site: System.Net.WebResponse GetResponse()
The following is a stack trace that identifies the location where the exception occured

   at System.Net.HttpWebRequest.GetResponse()
   at System.Xml.XmlDownloadManager.GetNonFileStream(Uri uri, ICredentials credentials, IWebProxy proxy, RequestCachePolicy cachePolicy)
   at System.Xml.XmlUrlResolver.GetEntity(Uri absoluteUri, String role, Type ofObjectToReturn)
   at System.Xml.XmlTextReaderImpl.OpenAndPush(Uri uri)
   at System.Xml.XmlTextReaderImpl.PushExternalEntityOrSubset(String publicId, String systemId, String baseUriStr, Uri& baseUri, String entityName)
   at System.Xml.XmlTextReaderImpl.DtdParserProxy_PushExternalSubset(String systemId, String publicId)
   at System.Xml.DtdParser.ParseExternalSubset()
   at System.Xml.DtdParser.Parse(Boolean saveInternalSubset)
   at System.Xml.DtdParser.System.Xml.IDtdParser.ParseInternalDtd(IDtdParserAdapter adapter, Boolean saveInternalSubset)
   at System.Xml.XmlTextReaderImpl.ParseDtd()
   at System.Xml.XmlTextReaderImpl.ParseDoctypeDecl()
   at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
   at System.Xml.XmlLoader.Load(XmlDocument doc, XmlReader reader, Boolean preserveWhitespace)
   at System.Xml.XmlDocument.Load(XmlReader reader)
   at System.Xml.XmlDocument.LoadXml(String xml)
   at client.Common.Types.TempPPSRBatch.CreatePayloadFromString(String p)
…
   at Microsoft.XLANGs.Core.SegmentScheduler.RunASegment(Segment s, StopConditions stopCond, Exception& exp)

The last line of custom code (highlighted above) was the call to the “LoadXml” method on an XmlDocument object (a method every BizTalk developer knows intimately!)  Why on earth, you ask, would this yield a WebException?? At most, we would normally expect an XmlException if the string parameter was invalid XML.

A bit of Googling yielded the answer in this post, as well as the pointer to the explanation supplied by http://stackoverflow.com/users/88558/lucero. Basically, it looks like the W3C has grown tired of its servers being hammered by XML parsers downloading the DTD documents referenced as standard namespace declarations (e.g. http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd).

Based on the W3C post, this seems like it would be frequent problem, but I believe the issue was caused in this case by the fact that the payload XML string we were loading contained some “text/html” content with a “<!DOCTYPE>” reference including the “http://www.w3.org/TR/html4/loose.dtd” namespace. In any case, it would be wise to consider this anytime you are loading an XmlDocument object, particularly if you are not sure whether the content may cause the XmlResolver to look up external resources. This solution provided by Edwin de Koning worked well for me (translated to C# from his sample VB code):

XmlDocument doc = new XmlDocument();
XmlReaderSettings settings = new XmlReaderSettings();
settings.XmlResolver = null;
settings.DtdProcessing = DtdProcessing.Ignore;

using (StringReader sr = new StringReader(xmlString))
{
    using (XmlReader reader = XmlReader.Create(sr, settings))
    {
        doc.Load(reader);
    }
}

This article by James Shropshire gives some helpful advice about using these two free tools from Microsoft to troubleshoot issues with MS DTC:

http://www.remotedbaexperts.com/Blog/2011/01/troubleshooting-biztalk-dtc-issues-with-dtcping-and-dtctest/

Yesterday, the DTCTest proved very helpful in identifying issues at a client where they had used an unsupported cloning tool to create the BizTalk and the SQL server images, which left both with the same CID (these must be unique). Fortunately, the problem was solved by running the following two commands:

msdtc –uninstall

msdtc –install

More information about this can be found here:

http://msdn.microsoft.com/en-us/library/aa561924.aspx

At first this seemed a weird question to me, to communicate with Azure Service Bus it is all outbound traffic, but more and more companies lately are restricting outbound traffic, for example the default Microsoft Threat Management Gateway (TMG) outbound rule only allow 80 and 443.

So for my own reference and for others that may get asked the same question, here are the steps for create an TMG outbound firewall rule for Azure Service Bus:

In the TMG Management Console under Web Access Policy, we need to define the network object that represents our BizTalk or Windows Server that needs to access SB

We also need to create a Protocol rule that defines that SB Traffic, TCP 9350 to 9354

We then need to create the Web Access Policy for Azure ServiceBus

Protocols HTTP, HTTPS and the ServiceBus Protocols we defined earlier

The Source of the Traffic we defined earlier

The destination of the traffic, in our case the external interface on our TMG Server

Once the Web Access Policy is created you will have to activate the change to the TMG Server

The new Web Access Policy is now active, you BizTalk or Windows Server will now have access to the Azure Service Bus

I was keen to modify the exported wsdl on a WCF service and checked out the many articles out there in the ether to fast-track my efforts.  I managed to build a simple HelloWorld service (with client console) and added in the plumbing to implement both IWsdlExportExtension and (in my case), IEndPointBehaviour.

After I felt I had everything in place and (seemingly) configured correctly I tested the service by browsing to it and visually checking the wsdl – no, my modified wsdl wasn’t showing.  No errors, just no change in the service description.

I put traces in each of the IWsdlExportExtension and IEndPointBehaviour methods and when I’d hit the service, nothing was coming out in the trace – again, no errors, no output and no change in the wsdl.  Frustration Plus.

I thought I’d change tack and get my worker class to implement IServiceExtension instead (just to see some kind of change in behaviour).  I modified the configuration appropriately and this time saw that each of my IServiceExtension methods were getting hit…something was happening!

After the umpteenth magnifying-glass-close scrutiny of my service web.config (and after combing over an excellent blog by Carlos Figueira) I noticed a discrepancy – the “name” attribute of my “service” node included the service namespace but the name was slightly off – “HelloWorld” instead of “HelloWorldService” in this case.  No match on name, no pointing to the behaviour I had configured!!!  I made the change and voila – everything worked – methods getting hit, wsdl getting modified and emitted as expected.

I’ve included my web.config snippet below.

  <system.serviceModel>

    <extensions>

      <behaviorExtensions>

        <add name=“simpleBehavior“ type=“Mexia.Framework.TestService.SimpleEndPointBehaviour, Mexia.Framework.TestService, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null“ />

      </behaviorExtensions>

    </extensions>

    <behaviors>

      <endpointBehaviors>

        <behavior name=“simpleServiceBehaviour“>

          <simpleBehavior />

        </behavior>

      </endpointBehaviors>

      <serviceBehaviors>

        <behavior>

          <serviceMetadata httpGetEnabled=“true“/>

          <serviceDebug includeExceptionDetailInFaults=“true“/>

        </behavior>

      </serviceBehaviors>

    </behaviors>

    <services>

      <service name=“Mexia.Framework.TestService.HelloWorldService“>

        <endpoint address=“” binding=“basicHttpBinding“ contract=“Mexia.Framework.TestService.IHelloWorldService“ behaviorConfiguration=“simpleServiceBehaviour“ />

        <endpoint address=“mex“ binding=“mexHttpBinding“ contract=“IMetadataExchange“ />

      </service>

    </services>

    <serviceHostingEnvironment multipleSiteBindingsEnabled=“true“ />

  </system.serviceModel>

Also for some excellent references on WCF extensibility check out some of Carlos’ blogs:

http://blogs.msdn.com/b/carlosfigueira/archive/2011/10/06/wcf-extensibility-wsdl-export-extension.aspx
http://blogs.msdn.com/b/carlosfigueira/archive/2011/04/05/wcf-extensibility-iendpointbehavior.aspx

Hope this helps!

Matt Warwick

Easy, right? Of course. But unfortunately, like most things in life, the trade-off of simplicity is some significant limitation in functionality and/or flexibility.

In the example case above, the “B2GResponse” record is a repeating element in the schema (maxOccurs=”unbounded”). While the TPE will very happily let you save and deploy the tracking profile, you will discover that a record with multiple “B2GResponse” nodes incurs a runtime error, as evidenced by this event log entry:

The “AmbiguousXPathException” thrown states “The result set for the XPath expression ‘<xpath expression for the target “PpsrBatchId” node>‘ contains more than a single node.”

Moreover, the actual node value is never recorded, even though BAM does (very politely) create the record despite the error (it just inserts NULL into the “PpsrBatchID” field):

Essentially the XLANG engine cannot map a repeating value to a single column in the BAM activity table. Makes sense, right? Even though we know in this case that “PpsrBatchID” node will contain the same value from every record, we can’t expect the BAM runtime to infer this for us. What would be nice is if the interface allowed you to specify a specific index for the node from which to draw the value from; it doesn’t (simplicity vs. functionality).

One obvious solution would be to re-design the message schema and elevate the “PpsrBatchId” value to a single node occurrence at the message level. However, we don’t always have this option to customise messages.

It doesn’t have to end here, though. Often the limitations offered with a simple GUI tool can be overcome by looking at what’s under the covers. In this case, with a little bit of “hacking”, we can manipulate BAM to parse the value out of a specific instance of a repeating field.

First we have to understand what the TPE does. When you save a tracking profile, it creates an XML file with a “*.btt” extension. This can then be deployed either through the TPE GUI or by the BTTDeploy command line tool:

“%BTSINSTALLPATH%tracking\bttdeploy.exe” “.\ ProcessRegistrationsResponse.v0.2.btt”

If we inspect the *.btt XML file, we can easily divine its structure:

Essentially, we see a “Dimension” element for each BAM activity field with a “DataLevel” sub-element that defines the source of the data. In the case of example TPE above:

<?xml
version=“1.0“
encoding=“utf-16“?>

<TrackingProfile
xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance“
xmlns:xsd=“http://www.w3.org/2001/XMLSchema“
VersionGuid=“00000000-0000-0000-0000-000000000000“
Name=“ProcessRegistrationsResponse“>

<Dimension
Name=“ActivityID“
DataType=“TraceID“ />

<Dimension
Name=“StartTime“
DataType=“DATETIME“>

<DataLevel
Name=“Rcv_RegistrationsResponseMsg“
SourceTypeSelected=“Orchestration Shape“
TargetAssemblyName=“MyClient.Ppsr.Orchestrations, Version=1.0.0.0, Culture=neutral, PublicKeyToken=d9cf9d5c9f7e687e“
OrchestrationReference=“MyClient.Ppsr.Orchestrations.ProcessRegistrationsResponse“
ShapeID=“5121a798-8e5d-4cc5-b0b7-ae68ece3992d“ />

</Dimension>

<Dimension
Name=“EndTime“
DataType=“DATETIME“>

<DataLevel
Name=“TRACE“
SourceTypeSelected=“Orchestration Shape“
TargetAssemblyName=“MyClient.Ppsr.Orchestrations, Version=1.0.0.0, Culture=neutral, PublicKeyToken=d9cf9d5c9f7e687e“
OrchestrationReference=“MyClient.Ppsr.Orchestrations.ProcessRegistrationsResponse“
ShapeID=“e1f75b0c-dfc4-4adb-b806-beab9156a04b“ />

</Dimension>

<Dimension
Name=“PpsrBatchID“
DataType=“NVARCHAR“>

<DataLevel
Name=“PpsrBatchID“
SourceTypeSelected=“Orchestration Payload“
TargetAssemblyName=“MyClient.Ppsr.Orchestrations, Version=1.0.0.0, Culture=neutral, PublicKeyToken=d9cf9d5c9f7e687e“
OrchestrationReference=“MyClient.Ppsr.Orchestrations.ProcessRegistrationsResponse“
ShapeID=“67801549-63ff-4db0-affa-abed2e387b22“
MessageName=“msgB2GResponsesIN“
MessagePart=“responses“
SchemaName=“MyClient.Ppsr.Schemas.Internal.PpsrDB.Veda.TableOperation_dbo_B2GResponse+Insert“
MessageDirection=“Out“
SomXPath=“/*[local-name()='&lt;Schema&gt;' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/TableOp/dbo/B2GResponse']/*[local-name()='Insert' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/TableOp/dbo/B2GResponse']/*[local-name()='Rows' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/TableOp/dbo/B2GResponse']/*[local-name()='B2GResponse' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/Types/Tables/dbo']/*[local-name()='PpsrBatchID' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/Types/Tables/dbo']“
XPath=“/*[local-name()='Insert' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/TableOp/dbo/B2GResponse']/*[local-name()='Rows' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/TableOp/dbo/B2GResponse']/*[local-name()='B2GResponse' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/Types/Tables/dbo']/*[local-name()='PpsrBatchID' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/Types/Tables/dbo']“ />

</Dimension>

<Dimension
Name=“TableInsertRequestSent“
DataType=“DATETIME“>

<DataLevel
Name=“Snd_InsertResponseSQL“
SourceTypeSelected=“Orchestration Shape“
TargetAssemblyName=“MyClient.Ppsr.Orchestrations, Version=1.0.0.0, Culture=neutral, PublicKeyToken=d9cf9d5c9f7e687e“
OrchestrationReference=“MyClient.Ppsr.Orchestrations.ProcessRegistrationsResponse“
ShapeID=“67801549-63ff-4db0-affa-abed2e387b22“ />

</Dimension>

</TrackingProfile>

Notice the highlighted XPath statement that defines where to retrieve the data from for the “PpsrBatchId” field. Here we have the opportunity to manually edit the XPath, instructing BAM that we only want the first record instance by using the XPath “position()” function:

“/*[local-name()='&lt;Schema&gt;' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/TableOp/dbo/B2GResponse']/*[
...
/*[local-name()='B2GResponse' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/Types/Tables/dbo' and position()=1]/*[local-name()='PpsrBatchID' and namespace-uri()='http://schemas.microsoft.com/Sql/2008/05/Types/Tables/dbo']“

Now we can save & deploy the BTT file, and the next time we run the process…

Bingo! We can now see the value entered in the table.

Lesson learned: when the simplicity of a GUI tool restricts your ability to do something slightly more advanced, it pays to dive under the covers and see what really makes it work. You’ll often be rewarded with the ability to customise the behaviour.

The main areas of improvement are:

  1.  Platform Support for Windows Server 8, SQL Server 2012 and Visual Studio 2011.

  2.  Improvements to the accelerator packs for HL7, HIPPA and SWIFT

  3.  Better adapter connectivity for DB2 and IBM systems

  4.  Performance improvements for batch processing, ordered send ports and scale-out host configurations.

  5.  Even tighter integration with the Windows Azure ServiceBus for cloud-scale integration solutions.

  6.  Changes to the licencing model to allow cloud-based hosting from a SPLA partner.

Fundamentally these aren’t big changes to the overall BizTalk Server platform, but it demonstrates an on-going commitment by Microsoft to on-premises integration solutions for customers that can’t (or wont) move their systems to the cloud in the future.

Further, by ensuring that BizTalk Server is 100% compatible with the Windows Azure ServiceBus (using the Azure relay & messaging bindings with the WCF adapter), Microsoft are positioning themselves really well for any combination of the following solutions:

• Pure On-premises Integration
• BizTalk Server
• Windows Server AppFabric

• Hybrid Integration
• BizTalk Server
• Windows Server AppFabric
• Windows Azure AppFabric

• Pure Cloud-based Integration
• Windows Azure AppFabric

I think this is a really smart strategy by Microsoft, and those integration consultancies who play in this space would be wise to maintain their on-premises BizTalk expertise so they can deliver all three types of solutions to their customers.

I know we are.

We know that the future of system integration will involve the cloud in one way or the other, and as integration specialists working with corporate & government clients, building Hybrid Integration Platforms using both Windows Azure AppFabric Middleware AND BizTalk Server is now our raison d’être.

However explaining the cloud to our customers has not been as easy as our decision to follow Microsoft into the cloud.

Interestingly, we’re finding that the main pushback isn’t technical because they generally already know that the cloud gives them immediate agility, better server utilisation and a managed infrastructure service.

The main resistance points that we’ve had are:

    1.  Perception of unquantifiable risks (information security, service uptime etc), which is tackled here.

    2.  Real unwillingness to uproot their business’ IT systems while everything is stable, which is tackled here.

    3.  Perception of unpredictable costs, which is tackled here.

The third & final resistance point is an interesting one, mainly because I genuinely think some of our customers genuinely LIKE buying servers and having control over their environments.  I think this is because provided they can quantify the expense of buying their own infrastructure (however high it is), and as long as they are still making money, why turn everything upside down and change the way they do things?

This is clearly bonkers, however, it’s one of the resistance points we face and we have to know how to deal with it.

Enter stage left the Windows Azure Discovery Pack!

As you can see it’s a current promotion by Microsoft that gives you a 6 month block of Azure hours for half-price, plus free training, plus free consulting (from Mexia or your preferred Windows Azure partner).

The website also says the offer expires June 30^th 2012, but Microsoft’s effervescent Azure marketing specialist Margaret Synan explained to me that if you buy a DP before Xmas 2011, you’ll have time to purchase another one before the offer goes away.  This means you’ll have effectively locked in the 50% discounted Azure pricing for 12 months!

Talk about a way to quantify your Azure costs.

The Discovery Pack is really worth considering if Azure goodness is on your company roadmap for 2012 and you want to control your costs whilst you climb aboard the juggernaut!

Get it while it’s hot, and tell them Mexia sent you!

Mexia participated in the Melbourne event early 2011 and we loved every bit of it.  You get free Azure developer training, one-on-one developer mentoring from Azure MVP’s such as Steven Nagy, and advice on how to commercialise your idea.

If you’re a Microsoft BizSpark member you can also pitch your idea to a panel of investors & technical gurus, all of whom have real world experience launching technical products into the marketplace.

It doesn’t get much better considering the price, and I heartily recommend the investment of your time to go along.