Grandpa Ham – Grandpa Can do IT!

Attended combined NIEM National Training Event and Oasis Interoperability Summit in Baltimore last week. What a week!!

All of the following Items are from that event:
1. Participated in live demonstrations of interoperability by 11 separate commercial vendors, all using the DM-OPEN Backbone. Messages included Common Alerting Protocol sent from an actual Chorine sensor, NWS Tornado Warnings in CAP with the full polygon showing on maps used by multiple vendors, EDXL-DE wrapped Hospital Availability Messages, and EDXL-DE wrapped NIEM Amber Alert Messages, with accompanying Style sheet and reference base-64 encoded picture data used in full display. A professional videographer filmed the demonstration activities and interviewed key players. The edited video will be made available by OASIS. I will post the link when it is available.
2. Moderated NIEM NTE panel titled “Coordinating the Development and Adoption of Emergency Data Standards With the Ongoing Development of NIEM.” A format of 5 separate questions with short answer to each question by all panel members in turn was well received, both by the panel and the audience. Answers were lively and interesting. There were many audience questions as well. The NIEM organization recorded all panel sessions, so this panel will be available for review in its entirety.
3. Acted as a panel member in a second NIEM NTE panel titled: “Playing Well With Others”—NIEM and External Standards. This was a half session panel that stirred lots of interest and did not afford adequate time for all audience questions. Its recording will also be made available by the NIEM organization. Both panels made is clear that there is real cooperation between standards bodies and progress is being made to ensure that the value of all standards is recognized as a federation real capabilities. While some technical and “turf” issues need to be understood better, the folks involved look forward to the future with a positive attitude and a real belief in success.

Special Thanks to Donna Roy (NIEM Director) and her crew for a great event, and to Bill Kalin (Contractor to DHS Science and Tecnology) and Jane Harnad (OASIS) for organizing a superb demonstration and to all of the vendors for showing real interoperability in action. Standards do work!!!

DM-OPEN in particular is designed to be a great enabler for NIEM and NIEM based messaging. DM-OPEN is based on the EDXL-Distribution Element (DE) which is a NIEM approved external standard for “packaging” content for distribution. The following link is to a recorded presentation from 19 August 2009 (last week) to the DM-OPEN Special Interest Group SIG. This recorded presentation is from a live demonstration of the use of DM-OPEN to transport NIEM IEPD defined content from an originator to a separate display application. It also shows how (when a NIEM IEPD also has the associated style sheet that is now part of NIEM IEPD requirements) that the NIEM content and is associated presentation can be shipped in the same message to recipients. The recording is 40 minutes long. The recording is a little rough in spots, but the content is both illustrative and thought provoking.
http://www.disasterhelp.gov/disastermanagement/library/archive/open/090819present.wmv

The new DM-OPEN (out this fall) will be even more useful as it will allow queries using NIEM compatible keyword and type structures within the DE’s ValueListURN structure (a categorization scheme for metadata about message content). This will make retrieval and re-distribution of NIEM Content more practical and more efficient for DM-OPEN participant applications.

A raw demonstration of a very usable capability.

The DE offers implementers a unique opportunity: the ability to send message content within one contentObject tag and an XSLT for displaying that content in the originator’s desired format in another, all within the same message. On Wednesday August 19, I will demonstrate how this works on line to the DM-OPEN Special Interest Group (SIG) at noon Eastern Time via Microsoft Live Meeting. I will retrieve a DE created by the folks at Evolution Technologies from DM-OPEN, and then display it in a browser as the sender intends it to be seen. This is brand new to me so I will not have written all of the code to do it all “automagically.” (I may have it done by then but do not promise.) So, you will have to put up with some cut-and paste and manual file creation, but I will be doing it live so you can see the process and visualize for yourself how you might automate it in your system. And guess what? The content is from a NIEM IEPD.

(I sure hope I do not have a Bill Gates Moment! 🙂 )

Here are some Live Meeting Instructions and the call-in information:

IMPORTANT: If you have not logged into Live Meeting before, check out the following connection instructions and participant guidelines prior to next week’s meeting: http://www.disasterhelp.gov/disastermanagement/library/documents/LiveMtgInstruct.pdf

(1) Login to MS Live Meeting for visuals: The following login link can only be used 30 minutes prior to the scheduled meeting time:

https://www.livemeeting.com/cc/eiip/join?id=DMprogram&role=attend

(2) Call into the Conference Bridge number as follows: 1 (800)366-7242; Pin 3647 6736#.

I received e-mail asking about my ideas for incorporating external structures like IEEE 1512 or Cursor on Target (Cot) into NIEM.  I think my response may be of general interest:

I am not a believer in the “incorporation of external structures like 1512 or Cot in NIEM.”  The cost of merging the maintenance  and update process of multiple separate standards related organizations would be too high. The resulting governance process would induce such brittleness that progress would simply stop.   I am, however, a believer in using external standards in the IEPD process and in the use of external standards on their own when and wherever they make sense.  Developing a brand new NIEM structure to replace a well-defined standard that is already in use is ludicrous.  And I believe that the NIEM management team agrees with me when I say that.

There are actually four cases to consider that involve the use of external standards:

1. The external standard meets the mission of an IEPD on its own. In this case just use it.  Document it as the content of the exchange and do not spend time trying to “NIEMify” its content. (Examples are CAP1.1 and perhaps some of 1512).

2. The external standard is used to “wrap” NIEM content (likely case for EDXL-DE and Cot).  In this case again the standard is simply used to wrap a NIEM IEP (as defined in an IEPD for content).  There is no need to add the external standard elements into the internal content IEPD process.  The content IEPD should define how the external standard(s) could be used as a wrapping for transport, but that is different than defining the content itself and should be encapsulated in an entirely separate section of the IEPD from the content definition.

3. The external standard contains components that need to be used in an IEPD in combination with NIEM components (GML, KML, and perhaps 1512 are examples).  It would seem that we could simply add the appropriate namespace origin and mix them into our exchange schema with the content from a NIEM extension schema.  The problem here is that representations of these components are almost guaranteed to violate NIEM Naming and Design Rules (NDR).  For example: Even a simple type such as xsd:string must be converted to a nc:TextType, which is in turn a niem-xsd:string that acts as a proxy for the XSD type but adds an optional set of attributes to the string.  So, even a simpleType from schema becomes a complexType in NIEM.   Luckily, there is an out in the NDR called the adapter. An adapter is a simple, single element wrapper that is used in NIEM to contain the external component. It is defined in the NIEM NDR as follows:  “An adapter type is a NIEM conformant type that adapts external components for use within NIEM.  An adapter type creates a new class of object that embodies a single concept composed of external components.  A NIEM conformant schema defines an adapter type.” It basically adds the attributes at the adapter level that are needed for NIEM conformance without fussing with the representations and naming issues of its contained content.  An adapter can contain:
a) an entire external schema (assuming it wraps the root element type)
b) a complex component element type from a schema
c) a single type from the schema.
The use of adapters is described in section 7-7 of the NIEM NDR.  Serious users of NIEM should read the NDR in its entirety.  The final point on this discussion is very simple.  You do not incorporate an external structure into NIEM, but you can define a NIEM conformant adapter in your IEPD as a container for that external structure that will allow you to import the external structure “as-is.”

4. The NIEM is not complete and needs extension. There are concepts that are needed in NIEM that may exist in external standards already.  In this case the concepts (but not the standards themselves) actually do need to be “incorporated” into NIEM.  So, how do we bring in the concept without creating a duplication of the external standard?  We could “coordinate” with the external standards organization(s), each of which have their own representation for the concept. This is indeed a hard task, and creates brittleness in and out of NIEM.  It can be done to some extent, but it is not a good idea to put it into any form of formal governance.  The cost is just too high.  Luckily, NIEM already provides a workable solution: the use of Abstract Elements with Multiple Representations already exists for dates and units of measure.  This is extended to the use of external namespaces for code lists.  Why not do the exact same thing for concepts that originate in an external standard?
a) Bring in the concept.  Name and define it in accordance with NIEM NDR.
b) Identify the Concept as an abstract element with multiple representations.
c) Identify one of the representations as the one in the external standard using the external standard’s namespace. The NDR may require that it be wrapped in a single element Adapter.
d) Identify a NIEM compliant and/or other standard representations as appropriate.
This avoids the duplication of concept and organizes multiple representations of the concept in a manner that that both controls those representations and makes translation between representations easier to accomplish. Yet it does not induce “coupling” between NIEM and other standards organizations. The separation remains clear and clean.

Note: Items 1, 2, and 3 on this list are already in place as how you should work with NIEM.  I put them in this message as a way of organizing the different issues that may be in play. Number 4 is new.  It is a combination of my recent completion of the on-line Practical Implementer’s course for NIEM and discussions with FEMA’s Emergency management domain lead and a couple of Georgia Tech Research Institute (GTRI) representatives at the recent NIEM Business Architecture Council meeting in Baltimore.  I have also had off-line discussion with FEMA technical representatives and members of the OASIS Emergency Management Technical Committee.

A final note: Although they do conform to the NDR, not everyone likes adapters. They add one more tag to an already deep chain of tags in a NIEM compliant schema.  They allow the use of non-NIEM content (although it is content that is recognized by NIEM).  My take is that adapters make cooperation between NIEM and the external standards bodies possible without the intense combined governance that could have a negative impact on progress on all sides.

I have been studying the National Information Exchange Model (NIEM) fairly extensively over the last few weeks. I have even read the NIEM Naming and Design Rules document from beginning to end. I will admit that I went into it with something of a jaundiced view.  As a veteran contributor to the DoD data model and an outside observer of the GJXDM (recently), and a large scale IBM model (a long time ago), I have real reservations about the usability and maintainability of any all-knowing, all-seeing model.  I have, at least at this point, become a believer in NIEM.  Why?   Because NIEM accepts the notion that a federation between separately name-spaced models makes sense, both within NIEM, and with external standards defined outside the heavy NIEM NDR discipline (or defined with a different heavy discipline).   The notion of defining an Adapter for NIEM use of other standards is a brilliant concept.  This, combined with the Information Exchange Package Documentation (IEPD) methodology for documenting the contextual use of data used in exchanges has made me a fan.

The problem with this “federation of standards” concept is that it makes tools (and “auto-magic” validation) harder to build.  As a result there is a tendency to try and force all of the standards back into the all-knowing, all-seeing model. It is a seductive idea, but not a good idea.  Let’s look at a very simple example: EDXL Resource Management uses the Customer Information Quality (CIQ standard) for Person Names. This allows internationalization for all kinds of different Naming structures and for a wide variety of Addressing schemes.  NIEM (as a national model) is much more U.S. centric, particularly in the use of PersonName tag. Both CIQ and NIEM are appropriate in their respective namespaces (and the NIEM NDR respects this fact by allowing for the adapter wrapper for external standards).   If we try to combine the two standards by defining CIQ elements as NIEM elements directly in order to make the subschema generator work more easily, we blur important distinctions that were developed for good reason.

So, we need to use NIEM IEPD methods. They are excellent. But we must resist the desire to force single definitions for concepts that may appear to be the same, but actually differ due to the context in which they were defined.  In other words, do not force a merger of conceptual domains, unless they actually are the same.  NIEM lets us federate in the building of an IEPD.   We should take advantage of that capability.

I wrote a little ditty that explains the value of what FEMA’s Disaster management program offers to vendors, open source developers and even contract developers in the Emergency Management and Public Warning Domains. It is a question that users of such software might ask their vendors. Take a look.
See my Contact Info if you would like some help getting started

Are you OPEN? [1]

Can you connect using standards?
Are you open to all?
Or are you a silo?
Using “standards” to stall?

Our open web service
Connects all kinds of apps.
A middleware instance
To share more than “CAPs.”[2]

We have a web service
Based on EDXL [3]
That helps apps connect
Yet encapsulate well.

Hard wired integration
Is not what we do.
You connect via service
As captured by you.

You decide layout
And your design form
But connect to all others
Using standards as norm.

We make it straightforward.
Your connection is clean.
The boundaries work well.
You control what is seen.

You have the power.
We provide pipes,
For transferring data
Of all defined types.

With data described
Using DE [4]
So intelligent routing
Can come to be.

We provide access.
You set the rules
In the tags that you set
In the DE through your tools.

We then connect others
As desired by you.
And they get your data
As you want them to.

They can format the layout
In their own way.
Connected, yet separate
With their own say

Into how to display
And how to reuse
And so can all others
Unless you refuse.

You can work independent,
Yet use standards to share.
The best choice to ensure
Service to all; everywhere.

Gary A. Ham – May 14, 2009

[1] FEMA – Disaster Management Program – Open Platform for Emergency Networks
[2] OASIS Common Alerting Protocol
[3] OASIS Emergency Data Exchange Language
[4] OASIS EDXL Distribution Element

I have finally updated my resume to account for the work I have been doing for FEMA since October. Yes I am back to the Disaster Management Program except that I am working for the Program Office instead of on the development contract. A lot of the work is similar and I am glad I can participate. The mission is important. I am now under contract through Eye Street Software Corporation to provide systems engineering support to the Federal Emergency Management Agency (FEMA). Duties include:

• Requirements development and assessment of steps needed to decouple the current Disaster Management Interoperability Services (DMIS) capability into two separate and cooperative capabilities for FEMA. The Disaster Management – Open Platform for Emergency Networks (DM-OPEN) will be a stand alone, standards based Enterprise Service Bus for data communications interoperability across the full spectrum of responder organizations at the Federal, State, Local and Tribal levels. The Current DMIS Toolset will be transformed from its current client server implementation into a web-based framework (the DM-Framework) that houses access to a user-configurable set of emergency management applications.
• Systems Engineering Life Cycle Documentation and Federal Enterprise Architecture compliance management for the DM PMO.
• Response to stakeholder inquiries concerning DM-OPEN, the DM-Framework, and related data standards.
• Assistance to programmers connecting to DM-OPEN Interfaces.
• Liaison with the Organization for the Advancement of Structured Information Standards (OASIS), as well as other standards organizations and Federal programs that affect, or are affected by, the Disaster Management Program.

The new poller/poster for FEMA’s Open Platform for Emergency Networks is now on Source Forge. Lee Tincher from Evolution Technologies reworked some of my old code to make in work directly with both Oracle and MySQL databases. I must admit that his solution is much more robust than my original test code. On the other hand, I did write the foundation connection code. :–) Let’s hope we get a lot of interest. This could make emergency alert sharing in a non-proprietary environment actually work. Even better, lets make it work in a multi-proprietary environment using OASIS EDXL Standards for communication through a level playing field Government supplied middleware infrastructure. This was my dream when I worked on Disaster Management. Maybe it can actually happen.

Spent the last couple of weeks helping to design a single place in the logical data structure for all aspects of information a DMV might want to consider. The Customer Information Quality Specification from OASIS provided the basic structure, but a lot of new material is needed to make it work. The key is to allocate the data to the correct place in a hierarchy of Party Types and to the roles that are “playable” by each type. It is a thought provoking exercise. It also make you well aware of the potential data quality pitfalls of mis-allocating data to a less advantageous entity within the overall data architecture. (Note to readers: this is just the beginning ov some very cool stuff.)

Lee Tincher from Evolution Technologies is building some new open source connection code for DM-OPEN that adds to my previous test code. This addition will include database connection code for oracle and MySQL that will let developers retrieve Common Alerting Protocol (CAP) alerts from DM-OPEN and directly put them into a relational database for further processing. We plan to put the updated code up on the EM Forum site for others to use as desired. Cool stuff. Eventually we plan to add to this code so that it handles Emergency Data Exchange Language (EDXL) Distribution Element (DE), Resource Messaging (RM), and Hospital AVailability Exchange (HAVE). Emergency Managers every where – While I am of some assistance, Lee is doing real work that supports you! I am sure that he will appreciate your gratitude and support.