Grandpa Ham – Grandpa Can do IT!

I wrote the following Comment on a LinkedIn NIEM Thread and thought it might be worth sharing:

In my opinion (perhaps not so humble :-)) NIEM is very good if you use it right. But, if you think of NIEM as a standard by itself, it is NOT GOOD. Too much stuff. Hard to keep organized. It is actually incredible how well organized it is given its volume, but as a standard by itself, it is just too BIG to be anything but unwieldy.

Now, as a model for building standard exchanges, it has TREMENDOUS value. It is like a mine. You have to refine the ore through an appropriate standards development process. (The IEPD process works for this.) But the ore is super high quality when compared with building standards from scratch, because the NEIM model has been preprocessed in the sense that most definition and relationships are well defined and understood. Even so, you still have to refine NIEM input using the IEPD process to actually create a usable exchange or a useable standard exchange.

Bottom line, if you expect NIEM to be “auto-reusable,” you will be disappointed. But, If you use NIEM appropriately, it has a lot to offer in the way of documented data structure and definition.

There have been at least two situations that I have heard of recently that claim NIEM Compliance and External Standards Conformance in the same breath. While it can be done, neither actually did so. In one case, NIEM elements were mixed into non-NIEM schemas, but the NIEM attributes were removed. This is clearly not the correct approach (although at least NIEM concept re-use was achieved). In another approach, external standard concepts were “NEIMified” in a tool and mixed with NIEM elements in a combined IEPD without the use of adapters. This approach is NIEM conforming, but NOT standards conforming. This approach can claim to use input from existing standards but cannot claim to actually use those standards. Especially, this approach cannot claim to conform to them. I firmly believe in NEIM as a methodology and as an excellent model for concept re-use, but building a NIEM conforming schema is not the same as building a standard. A standard involves acceptance of the exchange schema by a formal standards body or by a wide body of users as a reusable exchange. When you build a NIEM complient IEPD you have 3 options:

1. Build it as a NIEM IEPD and ignore other standards.
2. Build an IEPD and use pieces of external standards but ignore validation or conformance to the the standards that are reused.
3. Build an IEPD with standards conforming components in adapters and add any other needed data using NIEM want list methods or current NIEM elements .

In all three cases, the result is not a standard until it is recognized as one, either de facto or through a recognized standards body. So, the IEPD is just the beginning. Before it becomes a standard, it must be recognized, either by a formal standards body, or through industry dominant re-use.

Bottom line: If you choose path 2, you should not advertise it as standards conforming. While you can claim NIEM Conformance, you cannot honestly claim conformance to the external standards used as input in any way whatsoever.

I have been chosen to present a new twist on the reuse of data definitions at the National Information Exchange Model – National Training Event (NIEM NTE) in August. Data definition reuse questions that have been asked before include:
1. Is it OK to Use an External Standard without using NIEM?
2. How do I (or is it allowable to) encapsulate the use of an External Standard inside a NIEM conforming schema?
3. What is the best way to (or should I) bring concepts from external schemas into NIEM (to “NIEMify” them)?
These question have answers, although not everyone agrees on all of them.

My question is different. What if there was a way to reuse NIEM conforming structures inside an external standard? I plan to show an example of how it could work, using NIEM enumerated values (facets, codelists, and schema subsets if necessary) to populate EDXL-DE metadata around an IPAWS Profile CAP message and/or a NIEM IEPD for Amber alerting.

It ought to be an interesting discussion.

The Fiscal Year 2010 “Interoperable Communications Grant Program, Guidance and Application Toolkit” has just been published. My first question on seeing the grant language was, Did they mandate real interoperable data standards for software purchased using grant money?

They did. From Page 20:

Grant-funded systems, developmental activities, or services related to emergency response information sharing should conform as much as possible with the OASIS Emergency Data Exchange Language (EDXL) suite of data messaging standards and National Incident Management System (NIMS) guidelines. Additional information on data messaging standards and their applicability may be found at www.oasis-open.org. The NIMS Supporting Technology Evaluation Program (NIMS STEP) provides objective evaluations of commercial software and hardware products, and reports on product conformity to standards and NIMS guidelines. Findings from evaluations may be accessed through the Responder Knowledge Base (RKB) website to assist grantees in making purchases. More information on the NIMS STEP can be found at https://www.rkb.us/contentdetail.cfm?content_id=219711.

And Again from page 28 under Technology:

National Information Exchange Model (NIEM). FEMA requires all grantees to use the latest NIEM specifications and guidelines regarding the use of Extensible Markup Language (XML) for all grant awards. Further information about the required use of NIEM specifications and guidelines is available at http://www.niem.gov.

NIEM is XML. EDXL is XML. What gives? Who has precedence? Why is EDXL mentioned in the Funding Restrictions section and NIEM in the Administrative Requirements section?

In reality, you can ignore the apparent confusion. The requirements are valid and complimentary. For the most part, EDXL standards are accepted by NIEM as “approved external standards.” So you do not violate the NIEM requirements by using them, provided you use them as-is, in their entirety. If you use use individual elements (or a subset of elements) from an EDXL schema) in a way that does not validate against one of the schema standards, you are actually violating both EDXL and NIEM unless you document the use of those elements using the formal NIEM Information Exchange Package Documentation (IEPD) methodology as defined at niem.gov. So if you want to use a system that uses EDXL-Common Alerting Protocol (CAP), EDXL- Distribution Element (EDXL-DE), EDXL-Resource Messaging (EDXL-RM), and/or EDXL-Hospital Availability (EDXL-HAVE), go ahead. You are within the terms of the grant language. But if you modify (aka “improve”) the standards in any way, you must go through a formal IEPD process.

If, however you have requirements for information exchange that are not met by existing standards, NIEM offers you the opportunity to reuse existing NIEM IEPDs, build a new IEPD from existing NIEM data definition resources, or build an IEPD from a combination of data definition resources. It is a well-defined process that is designed to maximize reuse and minimize redundancy in data structure definitions supporting emergency management dat exchange requirements.

So, to summarize, if the software you are considering for purchase/development with your grant money reuses EDXL Exchange Standards and/or NIEM IEPDs, you are home free. If not, the system needs to define its exchanges with other systems following NIEM IEPD development rules as found at NIEM.gov

Attended combined NIEM National Training Event and Oasis Interoperability Summit in Baltimore last week. What a week!!

All of the following Items are from that event:
1. Participated in live demonstrations of interoperability by 11 separate commercial vendors, all using the DM-OPEN Backbone. Messages included Common Alerting Protocol sent from an actual Chorine sensor, NWS Tornado Warnings in CAP with the full polygon showing on maps used by multiple vendors, EDXL-DE wrapped Hospital Availability Messages, and EDXL-DE wrapped NIEM Amber Alert Messages, with accompanying Style sheet and reference base-64 encoded picture data used in full display. A professional videographer filmed the demonstration activities and interviewed key players. The edited video will be made available by OASIS. I will post the link when it is available.
2. Moderated NIEM NTE panel titled “Coordinating the Development and Adoption of Emergency Data Standards With the Ongoing Development of NIEM.” A format of 5 separate questions with short answer to each question by all panel members in turn was well received, both by the panel and the audience. Answers were lively and interesting. There were many audience questions as well. The NIEM organization recorded all panel sessions, so this panel will be available for review in its entirety.
3. Acted as a panel member in a second NIEM NTE panel titled: “Playing Well With Others”—NIEM and External Standards. This was a half session panel that stirred lots of interest and did not afford adequate time for all audience questions. Its recording will also be made available by the NIEM organization. Both panels made is clear that there is real cooperation between standards bodies and progress is being made to ensure that the value of all standards is recognized as a federation real capabilities. While some technical and “turf” issues need to be understood better, the folks involved look forward to the future with a positive attitude and a real belief in success.

Special Thanks to Donna Roy (NIEM Director) and her crew for a great event, and to Bill Kalin (Contractor to DHS Science and Tecnology) and Jane Harnad (OASIS) for organizing a superb demonstration and to all of the vendors for showing real interoperability in action. Standards do work!!!

DM-OPEN in particular is designed to be a great enabler for NIEM and NIEM based messaging. DM-OPEN is based on the EDXL-Distribution Element (DE) which is a NIEM approved external standard for “packaging” content for distribution. The following link is to a recorded presentation from 19 August 2009 (last week) to the DM-OPEN Special Interest Group SIG. This recorded presentation is from a live demonstration of the use of DM-OPEN to transport NIEM IEPD defined content from an originator to a separate display application. It also shows how (when a NIEM IEPD also has the associated style sheet that is now part of NIEM IEPD requirements) that the NIEM content and is associated presentation can be shipped in the same message to recipients. The recording is 40 minutes long. The recording is a little rough in spots, but the content is both illustrative and thought provoking.
http://www.disasterhelp.gov/disastermanagement/library/archive/open/090819present.wmv

The new DM-OPEN (out this fall) will be even more useful as it will allow queries using NIEM compatible keyword and type structures within the DE’s ValueListURN structure (a categorization scheme for metadata about message content). This will make retrieval and re-distribution of NIEM Content more practical and more efficient for DM-OPEN participant applications.

I have been studying the National Information Exchange Model (NIEM) fairly extensively over the last few weeks. I have even read the NIEM Naming and Design Rules document from beginning to end. I will admit that I went into it with something of a jaundiced view.  As a veteran contributor to the DoD data model and an outside observer of the GJXDM (recently), and a large scale IBM model (a long time ago), I have real reservations about the usability and maintainability of any all-knowing, all-seeing model.  I have, at least at this point, become a believer in NIEM.  Why?   Because NIEM accepts the notion that a federation between separately name-spaced models makes sense, both within NIEM, and with external standards defined outside the heavy NIEM NDR discipline (or defined with a different heavy discipline).   The notion of defining an Adapter for NIEM use of other standards is a brilliant concept.  This, combined with the Information Exchange Package Documentation (IEPD) methodology for documenting the contextual use of data used in exchanges has made me a fan.

The problem with this “federation of standards” concept is that it makes tools (and “auto-magic” validation) harder to build.  As a result there is a tendency to try and force all of the standards back into the all-knowing, all-seeing model. It is a seductive idea, but not a good idea.  Let’s look at a very simple example: EDXL Resource Management uses the Customer Information Quality (CIQ standard) for Person Names. This allows internationalization for all kinds of different Naming structures and for a wide variety of Addressing schemes.  NIEM (as a national model) is much more U.S. centric, particularly in the use of PersonName tag. Both CIQ and NIEM are appropriate in their respective namespaces (and the NIEM NDR respects this fact by allowing for the adapter wrapper for external standards).   If we try to combine the two standards by defining CIQ elements as NIEM elements directly in order to make the subschema generator work more easily, we blur important distinctions that were developed for good reason.

So, we need to use NIEM IEPD methods. They are excellent. But we must resist the desire to force single definitions for concepts that may appear to be the same, but actually differ due to the context in which they were defined.  In other words, do not force a merger of conceptual domains, unless they actually are the same.  NIEM lets us federate in the building of an IEPD.   We should take advantage of that capability.

Interesting phenomena.  I put out a tweet likening the NIEM Naming and Design Rules Document to a cross between Leviticus and IRS regulations.  Almost immediately, I was followed by 1) a Bible Study Group asking for money 2) a NIEM consultant, and 3) a Tax Consultant.  Apparently they each have some sort of bot in place that looks for key words. They then follow in tthe hope of being followed.  It worked for the NIEM consultant, not the others. Shortly thereafter, I was folowed by ithe IJIS Institute, a justice related non-profit with a strong interest in NIEM.  I now follow them as well.