Following up on the BizTalk Adapter Pack on Framework 4.0

In this post, I mused about whether the BizTalk Adapter Pack 2.0 would actually work in Framework 4.0 applications -- not just in design-time. As it turns out, the answer is yes, albeit with two caveats and a potential issue.

These things you must do, or it won't work at all:

1. You must add the useLegacyV2RuntimeActivationPolicy="true" attribute to the startup node of your app.config file. Without this, you'll get an exception at runtime indicating that the mixed-mode assembly can't be loaded without additional configuration. This setting changes the way the Framework loads assemblies in side-by-side mode when they use APIs such as COM interop; marklio explains this better than I.
2. You must apply the machine.config changes in my prior post to the machine.config for both the 32 and 64 bit versions of the Framework, or you must change your build configuration to target a specific CPU architecture. You need to do this so the WCF architecture used by the Adapter Pack can see its binding extensions in the appropriate architecture.

The potential issue is that the Adapter Pack seems to be slower when run under Framework 4.0. The performance hit, based on my instrumentation in an application that I've just migrated, is about 20% or so for the SAP RFC adapter. I suspect this is because of changes between the 2.0 and 4.0 CLRs. Depending on what you use the Adapter Pack for, this may or may not have a major impact. SAP, for instance, is often slow enough on the RFC interface that a 20% increase is hardly noticeable.

Oh, and this trick works for both the full and client profiles of the Framework, again assuming you do both 1 and 2 above. 2 will usually be taken care of for you as most of the time, you'll be using architecture-specific DLLs for your ERP system's interface and you'll have already addressed this when you built the application.

SCCM is ConfigMgr and diagnosing BITS transfer errors

I'm told that SCCM should actually be referred to as ConfigMgr, as Microsoft doesn't actually own the SCCM trademark. Personally, not working for Microsoft, I could care less, but to make life easier on people, I'm going to start tagging posts with both terms.

On to the meat of things, which is how to diagnose ConfigMgr BITS transfer errors in Windows 7, and what a common error you might encounter is. When you configure a ConfigMgr site system for app distribution on a Server 2008 or 2008 R2 machine, it sets up IIS and enables BITS, the Background Intelligent Transfer Service. BITS is neat in that it can trickle data down to a machine, allowing it to download files without chewing up all of the user's apparent bandwidth. BITS works over http and https, which is why IIS is required for distribution points. BITS usually works pretty well, downloading packages quickly from the nearest distribution point when the user's network connection is idle.

Sometimes, however, a system may hang up on a BITS download. I'm not speaking here about the common failures of ConfigMgr downloads, such as IIS misconfigurations (WebDAV needs to be enabled and properly configured, for instance), location issues, impersonation issues (if a user is logged on, the distribution point is selected and accessed using the user's account as opposed to the computer account; this also means that if you have multiple sites, the site for the domain in which the user account lives is used), or the like. No, a BITS error appears when you begin a download and it stalls at a particular percent, seemingly at random. You can check on BITS problems in two ways -- using PowerShell, or by using bitsadmin. Either one needs to be run as an administrator to be used for ConfigMgr troubleshooting. Unfortunately, the PowerShell cmdlets for BITS are functionally useless, as they can basically only tell you a job's rough status. Therefore, we must delve into the deprecated bitsadmin command.

The first step is to find the job of interest. To do this, run "bitsadmin /list /allusers", which will list all of the BITS jobs on the system. ConfigMgr's jobs are named "CCMDTS Job" and will look something like this:


{7DDD0DB1-33BC-4EFE-BE0E-E904B362D55F} 'CCMDTS Job' TRANSFERRING 0 / 2 811168 /
7724032

The first field is the job's GUID. The second field is the name, while the third is the status. The fourth field lists the number of files completely downloaded out of the number of files in the job, and the fifth field lists the number of bytes downloaded out of the number of bytes in the job. This job is healthy; it's listed as status TRANSFERRING. A job can also be in the TRANSFERRED state, which means it's done (and ConfigMgr usually cleans those up for you), or in the ERROR state.

To find out why a job has ERRORed out, you can look at it with "bitsadmin /info GUID /verbose", where GUID is the GUID of the job. A list of files will appear under the JOB FILES header of the output. Each line shows the source file URL and the destination path. The error is usually on the last file downloaded, which is the first one in the list; any error will appear on that line. For instance, you might get an http 404 error, which means that the file isn't available on the distribution point. You can verify that by copy-pasting the source URL into any web browser. A 404 error can be due to a failure of the package to copy (though usually this'll result in ConfigMgr's LocationServices throwing a fit and blocking the download), but sometimes, it's a case of missing a specific fiddly bit in ConfigMgr setup on IIS 7.0.

You see, IIS 7.0 blocks certain extensions from being downloaded from the server. It essentially makes them invisible if they're not handled by a type handler, which is good if, for instance, your legacy web application uses an Access database. You probably wouldn't want that to be downloaded, and it probably needs to be accessible by the web application's service account, so IIS 7.0 just makes it impossible to download. These filters are set up in the %windir%\system32\inetsrv\config\applicationHost.config file. Search for the requestFiltering section, and you can see the entries. If any file in your ConfigMgr application package has an extension that's in that list -- perhaps because it uses an Access database -- LocationServices will allow the download to occur, but the BITS transfer will fail because the file is "unavailable," IIS having made it invisible. The fix is simple and documented here:

• Change the "allowed" "false" to "true" for the extensions in question, then 
• Run "iisreset /restart" from your nearest friendly command prompt

Be aware that this makes your IIS configuration less secure, though if you're like most people, your distribution point servers only run IIS for DP duties, so this is an irrelevant concern. Now that you've done this, you can go back to your machine and run "bitsadmin /restart GUID", which will force the job to restart from the error state. If nothing else is blocked, you're in business!

Other BITS errors to be aware of are:

• http 500: An Internal Server Error means you should check the Application event log on the DP server. If this is the first time it's happened, recycling the application pool for the DP site and restarting the BITS service on the server should fix the issue. If not, make sure BITS is installed properly and WebDAV is properly configured.
• http 403: Make sure the user attempting to access the DP has the required rights to do so; this means that the security on the distribution point location should permit proper groups (usually, DPMachine\Users) read access. This should really only be an issue in a cross-forest configuration, where the default configuration of the Users group on the DP machine doesn't include authenticated users from the other forest. My advice here is to not cross forest boundaries with your DPs, especially as AD Site boundaries behave strangely across forests. Avoiding cross-forest DPs is a key part of a KISS design for ConfigMgr; if you need to do this, set up different ConfigMgr sites in the forests and wire them together in a site hierarchy.

The tale of the mysterious Certificate Revocation Check failure in SCCM

One of the more fun applications in the Microsoft server set is System Center Configuration Manager, the new version of what was previously called Systems Management Server (SMS). SCCM is a godsend when it works, but it can exhibit some quirky behavior that's hard to diagnose.

On one computer today, I had an issue where I was trying to use my slick PXE automatic OS imaging function to reimage the machine. (I'll probably discuss my SCCM architecture at a later date, as it does some truly awesome things with PXE boot and task sequences for zero-touch image deployment.) I kept seeing that the task sequence was failing, as evidenced by what appeared to be a random reboot before the TS started. So, I dutifully looked in the SMSTS.log file for the source of the problem -- and it turned out to be an issue with certificate revocation list checking, specifically the dreaded WINHTTP_CALLBACK_STATUS_FLAG_CERT_REV_FAILED error. This error's particularly annoying for those of us with working certificate authority infrastructure, as it doesn't say anything about why the revocation check failed. Some of the time, at least in our environment, this is due to the CA randomly failing to publish its delta CRLs -- an issue easily addressed by restarting the CA. After doing this, I found the check was still failing, so I took at look at the CRL publication servers. We publish our CRLs to Active Directory and to two web sites, one internal and one external. All three of these were showing current, valid CRLs.

I was about to angrily disable CRL checking when I suddenly realized that the most recent delta CRL from the issuing CA has been published just a couple of hours ago. A quick check of the time on this newly unboxed computer revealed that it was a day and 16 hours off the current time, which would of course make CRL checking fail; as far as the machine's concerned, the CRL it downloaded wasn't yet valid. A quick fix of the time and date in the BIOS, a clearing of the PXE advertisement, and a reboot was all it took to get the task sequence humming again.

So, the moral of this story is, always check the dates and times of your client machines! Not only is this critical for Kerberos, but it can impact your PKI in ways that aren't immediately obvious.

Using Avantext technical manuals over the network, part 2

In my last award-winning post, I mentioned that it's possible to coerce Avantext into accessing files over the network and deploying silently -- a feat their technical support swears is impossible (and I'm pretty sure voids any support contract). Further investigation indicates that you need to do a few more things to make this work right.

1. First, for each CD, you need to edit the .ini file to which you point in the registry so that it uses the UNC path of that directory instead of a drive letter. For instance, for disc PR0101 (Piper Dakota), you'd want to replace all references to a drive letter (say, "D:") in the ini file to the UNC path to that network folder (\\Server\AvantextShareName\PR0101). Be sure to change the entries in both the [Path] section and the [Volume] section. Because we're changing a file that comes from the stock installation, I would highly recommend making a copy and changing the copy. You can always change the .reg file to point to any .ini file you want.
2. Make sure that the Users group has Modify access to the AVViewer\dat directory in your Avantext install directory. Avantext really likes to write to that directory, in flagrant violation of the Windows development guidelines. Likewise, if your users want to use the Order functionality in Avantext, you'll need to ensure they have Modify rights to the TechPubs\Orders directory, or that your .ini files from above point to a location to which they do have access.
3. You may need two .reg files, one for 32-bit systems and one for 64-bit systems, as Avantext is a 32-bit application. Its registry keys will show up in the Wow6432Node key on a 64-bit machine.
4. When you update Avantext, make sure you don't clobber the .ini file from above, or you'll be sorry.

One interesting note is that, at least on Windows 7 x64, you get an error about a file being used by another process when you exit the Avantext viewer. You can inform your users to just respond with "Cancel" to that dialog box.

Using Avantext technical manuals over the network

Everyone who's had the displeasure of dealing with aviation industry software knows that most of the vendors are stuck in the 1980s. A case in point is Avantext, the official Piper Aircraft technical manuals vendor. The Avantext software allows you to, in theory, access the technical manuals over a network, but in practice, makes this annoying by forcing you to map drives. Drive maps are the spawn of Satan (brittle unless mapped over DFS, often disconnected randomly, confusing to you and users when someone's "R drive" doesn't work, tough to deploy silently with, etc.), so I'm going to tell you how to get past this limitation.

When you install Avantext initially, you install it off a single CD. The content remains on the CD and registry keys are added to tell the viewer where to find the content. Now, if you're following the official directions, you need to repeatedly add each new CD from the network share where you copied them using a drive map. Instead, do this:

1. Open up regedit.
2. Go to the HKEY_LOCAL_MACHINE\SOFTWARE\Avantext\Products key.
3. Find your currently installed product and make the following changes:
a. Change the ContentLocation value to a UNC path pointing to the network location of your copy of the CD. Be sure to leave an ending slash on the path, or Avantext will say it can't find the CD.
b. Change the IniPath value to the UNC path of the .ini file in that directory.

Now, you've gotten Avantext to look at the network for a single CD. To add additional CDs, export this registry key, then open the .reg file in notepad. Edit the exported key to reflect the CD name (the CD name typically being the first four characters of the folder name) and adjust the UNC paths appropriately. When you're done, import the .reg file back into the registry and you should see all of your available publications.

Because you now have a .reg file that sets up Avantext, you can use a silent install of the client (with the /s switch) and a run of regedit to import the data. This lets you deploy Avantext over the network.

Please note that you shouldn't do this if you don't have the appropriate licensing for using Avantext over the network, and as always, you should ensure the security on the network folder is such that only licensed users can use the system.

Complex Exchange migrations, part 1

One fun thing about being where I'm at is that the previous IT regime was -- how shall I put this -- less than competent. That leaves me in the position of working with two forests and an Exchange 2003 implementation that likes to go down harder than a drunken baby in a roller derby. To fix this, I've got a plan of attack that involves migrating all of the users into the new, working, 2008 R2 functional level forest (we'll call it new.company.com) and onto a shiny new Exchange 2010 server (mail.company.com). Because this is a staged roll out that's happening at the same time as a Windows 7 implementation, we have to maintain coexistence and both forests for some time. That means that I'm doing a cross-forest Exchange migration from a legacy version, which is what I like to think of as "complex."

In the next posts on this topic, I'm going to walk you through how to pull this off in a mid-sized business. The basic steps are:

• Install Exchange 2010 in the new domain;
• Figure out how to synchronize the GALs for mail flow;
• Create cross-forest accounts and connectors;
• Script the movement of mailboxes;
• Clean up any messes that arise.

I'll assume you can figure out how to install 2010; it's pretty straightforward. I'll pick up with GAL synchronization and what exactly you need to create in the other domain to get mail working. I personally used ILM, but I know enough now about the schema to provide you with a PowerShell script to sync the GAL.

Adding the BizTalk Adapter Pack "Add Adapter Service Reference" plugin to Visual Studio 2010

Visual Studio 2010 is out at last, and it's awesome. It's got a slick new interface design, support for C# 4.0, and impressive integration with Team Foundation Server. What it doesn't have, however, is backwards compatibility out of the box with certain key components -- such as the BizTalk Adapter Pack and its required component, the WCF LOB Adapter. This means that you lose the ability to consume adapters with the Add Adapter Service Reference wizard. Now, you could wait for an official release that's compatible with VS 2010, or you can follow my instructions here and magically enable the Adapter Pack for 2010! Please note that these instructions involve editing machine.config and the registry, so remember to back things up first. They work for me, but they could break something on your machine.

Phase 1: The Registry 
The WCF LOB Adapter registers the Visual Studio add-in through the registry. It actually does it in an "unofficial" way, which is to say that it adds itself as
a Visual Studio feature package, then adds a reference to itself in the Menus key. To get the menu option to appear, you need to copy these registration entries from your Visual Studio 2008 registry tree to your Visual Studio 2010 one. Luckily for you, I've written a registry script that'll do that very thing. Copy the script below into a file with a .reg extension, then double-click on it to import it into the Registry (elevating, of course, if you're on Vista or 7). You may need to perform a few checks before importing it, such as:

• Removing the Wow6432Node and SysWOW64 references from the script if you're using a 32-bit operating system.
• Making sure you have the 64-bit version of the WCF LOB Adapter installed if you're using a 64-bit OS (which, I should note, is the only supported configuration)

The script follows.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\10.0\Packages\{fff9759c-767b-4327-b8c2-f2ff2e36144d}]

@="Microsoft.ServiceModel.Channels.Tools.PlugInPackage.PlugInPackage, Microsoft.Channels.Tools.PlugInPackage,Version=2.0.0.0,
Culture=neutral, PublicKeyToken=null"
"InprocServer32"="C:\\Windows\\SysWOW64\\mscoree.dll" "Class"="Microsoft.ServiceModel.Channels.Tools.PlugInPackage.PlugInPackage"

"CodeBase"="C:\\Program Files\\WCF LOB Adapter SDK\\Tools\\Microsoft.ServiceModel.Channels.Tools.PlugInPackage.dll"

"ID"=dword:00000001
"MinEdition"="Standard"
"ProductVersion"="2.0.0.0"
"ProductName"="Add Adapter Service Reference" "CompanyName"="Microsoft"

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\10.0\Menus]

"{fff9759c-767b-4327-b8c2-f2ff2e36144d}"=", 1000, 1"

Once this script is imported, when you run VS 2010, you'll now see the Add Adapter Service Reference menu item in the context menu of Solution Explorer. However, when you click on it, you'll be told there are no valid adapters installed. To solve this, we need to proceed to... 

Phase 2: machine.config 

Because VS 2010 runs as a Framework 4.0 application, the plugin binds using the 4.0 configuration files.You can verify this yourself by turning on assembly binding logging (http://msdn.microsoft.com/en-us/library/e74a18c4.aspx) and viewing the logs for devenv.exe's binds to Microsoft.ServiceModel.Channels.Tools.PlugInPackage.

Here's what that looks like in VS 2008, emphasis added:

*** Assembly Binder Log Entry (4/15/2010 @ 11:35:25 AM) ***

The operation was successful.

Bind result: hr = 0x0. The operation completed successfully.

Assembly manager loaded from: C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll

Running under executable C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\devenv.exe

--- A detailed error log follows.

=== Pre-bind state information ===

LOG: User = APIMEM\pchipman

LOG: Where-ref bind. Location = C:\Program Files\WCF LOB Adapter SDK\Tools\Microsoft.ServiceModel.Channels.Tools.PlugInPackage.dll

LOG: Appbase = file:///C:/Program Files (x86)/Microsoft Visual Studio 9.0/Common7/IDE/

LOG: Initial PrivatePath = NULL

LOG: Dynamic Base = NULL

LOG: Cache Base = NULL

LOG: AppName = NULL

Calling assembly : (Unknown).

===

LOG: This bind starts in LoadFrom load context.

WRN: Native image will not be probed in LoadFrom context. Native image will only be probed in default load context, like with Assembly.Load().

LOG: Using application configuration file: C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\devenv.exe.Config

LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v2.0.50727\config\machine.config.

LOG: Attempting download of new URL file:///C:/Program Files/WCF LOB Adapter SDK/Tools/Microsoft.ServiceModel.Channels.Tools.PlugInPackage.dll.

LOG: Assembly download was successful. Attempting setup of file: C:\Program Files\WCF LOB Adapter\SDK\Tools\Microsoft.ServiceModel.Channels.Tools.PlugInPackage.dll

LOG: Entering run-from-source setup phase.

LOG: Assembly Name is: Microsoft.ServiceModel.Channels.Tools.PlugInPackage, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35

LOG: Re-apply policy for where-ref bind.

LOG: Post-policy reference: Microsoft.ServiceModel.Channels.Tools.PlugInPackage, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35

LOG: Found assembly by looking in the GAC.

LOG: Switch from LoadFrom context to default context.

LOG: Binding succeeds. Returns assembly from C:\Windows\assembly\GAC_MSIL\Microsoft.ServiceModel.Channels.Tools.PlugInPackage\3.0.0.0__31bf3856ad364e35\Microsoft.ServiceModel.Channels.Tools.PlugInPackage.dll.

LOG: Assembly is loaded in default load context.

And what it looks like in VS 2010:

*** Assembly Binder Log Entry (4/15/2010 @ 10:53:05 AM) ***

The operation was successful.

Bind result: hr = 0x0. The operation completed successfully.

Assembly manager loaded from: C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll

Running under executable C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\devenv.exe

--- A detailed error log follows.

       

=== Pre-bind state information ===

LOG: Where-ref bind. Location = C:\Program Files\WCF LOB Adapter SDK\Tools\Microsoft.ServiceModel.Channels.Tools.PlugInPackage.dll

LOG: Appbase = file:///C:/Program Files (x86)/Microsoft Visual Studio 10.0/Common7/IDE/

LOG: Initial PrivatePath = NULL

LOG: Dynamic Base = NULL

LOG: Cache Base = NULL

LOG: AppName = devenv.exe

Calling assembly : (Unknown).

===

LOG: This bind starts in LoadFrom load context.

WRN: Native image will not be probed in LoadFrom context. Native image will only be probed in default load context, like with Assembly.Load().

LOG: Using application configuration file: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\devenv.exe.Config

LOG: Using host configuration file:

       

LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.

LOG: Attempting download of new URL file:///C:/Program Files/WCF LOB Adapter SDK/Tools/Microsoft.ServiceModel.Channels.Tools.PlugInPackage.dll.

LOG: Assembly download was successful. Attempting setup of file: C:\Program Files\WCF LOB Adapter\SDK\Tools\Microsoft.ServiceModel.Channels.Tools.PlugInPackage.dll

LOG: Entering run-from-source setup phase.

LOG: Assembly Name is: Microsoft.ServiceModel.Channels.Tools.PlugInPackage, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35

LOG: Re-apply policy for where-ref bind.

LOG: Post-policy reference: Microsoft.ServiceModel.Channels.Tools.PlugInPackage,            Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35

LOG: Found assembly by looking in the GAC.

LOG: Switch from LoadFrom context to default context.

LOG: Binding succeeds. Returns assembly from C:\Windows\assembly\GAC_MSIL\Microsoft.ServiceModel.Channels.Tools.PlugInPackage\3.0.0.0__31bf3856ad364e35\Microsoft.ServiceModel.Channels.Tools.PlugInPackage.dll.

LOG: Assembly is loaded in default load context.

As you can see, we're using the 4.0 configuration file in 2010, which means we need to copy the magic words from our 2.0 configuration file so our plugin can see the valid adapters. Note that this doesn't necessarily mean that 4.0 programs will be able to consume adapters -- that's an investigation for another day.

The magic words, in this case, happen to be located in the system.servicemodel part of the machine.config tree. You want to copy any Microsoft.Adapters references in the extensions/bindingElementExtensions and extensions/bindingExtensions nodes. You'll also need to copy any client/endpoint nodes that make similar references. For example, to get SAP bindings to work, you need to merge the following XML into your 4.0 machine.config file:

<system.serviceModel>

    <extensions>

        <bindingElementExtensions>

            <add name="sapAdapter" type="Microsoft.Adapters.SAP.SAPAdapterExtensionElement, Microsoft.Adapters.SAP, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>

        </bindingElementExtensions>

        <bindingExtensions>

            <add name="sapBinding" type="Microsoft.Adapters.SAP.SAPAdapterBindingSection, Microsoft.Adapters.SAP, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>

        </bindingExtensions>

    <client>

        <endpoint binding="sapBinding" contract="IMetadataExchange" name="sap"/>

    </client>

         

Now, you can use the Add Adapter Service Reference wizard in VS 2010 with your older Framework projects.

(Oh, and by the way, I want you to know how much effort it took to make this look pretty. Stupid HTML editor...)