Presentation on theme: "Quick Guide to CAM Blueprints"— Presentation transcript:
1 Quick Guide to CAM Blueprints Intelligence Commanders TrainingQuick Guide to CAM BlueprintsUsing blueprints to develop XML exchange templatesand schemaAuthor:David RR WebberChair OASIS CAM TCJanuary, 2010(CAM – Content Assembly Mechanism Specification)Foundations of Intelligence1
2 IntroductionConfiguring formal information exchanges using XML transactions today requires a set of exchange schema that describe those interactionsAn exchange schema provides the structural layout for the XML transaction along with rules detailing the content expectedExchange schema development requires knowledge of XML best practices, industry specifications, data modelling techniques and guidelines for exchanging XML transactionsHow does the CAM blueprint approach enable a shift away from a purely manual process today to one that can be highly automated, consistent and repeatable?Introducing concepts and overall needs
3 Solution Approach Top Down development Allows developers to quickly outline the information exchangeProvides XML dictionaries of canonical reusable componentsCAM tool then expands the details and fill-in the blanksReview in visual editor to confirm or extend resultsIterative approach until completeAutomated exchange package generationSchemas, XML, documentation, mapping crosswalkTest generation of example XML with rules validationPrinciples and Rules evaluatorProvides best practices for interoperabilityChecks for specific schema techniques and detailsDictionary driven reuseProvides industry components / Enterprise Data ModelEnsures consistency of definition and useRoll your own dictionariesQuick sketch of main aspects of blueprinting approach
4 Agenda Blueprints and Dictionaries Generating Exchange Artifacts Exchange Assembly Process Steps OverviewLeveraging re-use – canonical XML dictionariesIntroducing Exchange Assembly BlueprintsModelling information structures in XMLUsing namespaces and prefixesConstructing your Blueprint templatesRunning the ExpanderFinishing your Exchange TemplateGenerating Exchange ArtifactsNDR evaluation, Exchange schema, mapping crosswalk, XML instances, realistic data use, business rules documentationSummaryUse this slide to format text lists with bullets
5 Exchange Assembly Process Overview 1BusinessModellingRequirementsCharterWorkGroupProjectInceptionCAM EnabledDictionaryGeneration /SelectionModel ExchangeBlueprintsDictionary2WebbasedDesktopExcelSearch toolsImport XSDNaming / NDR / spell checkGenerate DictionaryImportEDMExchange Content Modelling3Expand and Evaluate ResultsExpand, Review, RefineCompleted Exchange StructuresNDR Evalu- ation4dictionariesReview / Tailor Exchange StructuresEach swim lane represents sets of tasks and actions in the process of developing an information exchange. Complete package of artifacts required are generated by CAM tools, also testing and verification reports. This then is ready to publish so that implementers can align their application systems to the specifications and business requirements. In NIEM parlance this is the IEPD – Information Exchange Package Definitions. Notice also the EDM – Enterprise Data Model components can be presented as dictionaries.Completed Exchange artifactsXMLexamplesXSDschemasSubsetSchemaCrosswalkMappingSpreadsheetRulesReportRenderingStylesheetTest Examples and Rules5Test CasesValidation EngineSchema GenerationGenerate Package Artifacts andTest themXSD Schema WriterCross Reference Mapping6Exchange Package
6 Leveraging Re-use : Canonical XML Dictionaries Intelligence Commanders TrainingLeveraging Re-use : Canonical XML DictionariesWhat is a canonical dictionaryStyles of dictionaryHow do I obtain a dictionary?How do I use it with CAM?Foundations of Intelligence6
7 Canonical XML dictionary A collection of distinct components that represent discreet business information for an application domainIncludes singleton components and combinations of related components together as sub-assembliesInformation is represented in a simple neutral conceptual data format that captures the critical concepts about the data e.g. name, description, content type, contextual usage pattern, hierarchyWikipedia definition:
8 XSD Schema v CAM Dictionary There are generally two “styles” of XSD schemaDistinct information exchange schema structuresCollections of schema model components, possibly abstract, and possibly recursiveCAM dictionaries built from distinct exchange structures generally mimic those components directlyParent / Child sets are visible in dictionary as ABIE/BBIE pairings“children” column in dictionary contains direct descendents“context” column reflects component relationshipsCAM dictionaries built from models with abstract and recursive parts may have fragments of components onlyAssembly of exchange structures using references to piece parts, rather than complete structure assembliesReuse is then around these extension part collectionsContext may need to be overridden to meet desired use patternExamples of two styles of schema – exchange schema: OASIS standard specifications; model schema: NIEM model schema.Notice that dictionary components are to facilitate assembly – so may have duplicates, or merged components sets (e.g. ABIE may appear more than once, or BBIEs only after first ABIE occurrence to use them. This reflects how the dictionary was built – e.g. single step from one CAM template, or multiple merging steps from many CAM templates.
9 Available XML Dictionaries LEXS 3.1.4 dictionaryOASIS EDXL dictionaryOASIS EML dictionaryNIEM 2.1 dictionariesCBRN dictionaryEmergency dictionaryFamily dictionaryImmigration dictionaryInfrastructure dictionaryIntelligence dictionaryJustice dictionaryMaritime dictionaryScreening dictionaryTrade dictionaryNIEM core dictionaryImmigration blueprintPackaged with CAM editorsee dictionary folder of install+ spreadsheet+ blueprint samplesXMLXMLXMLNote: Those marked in bold are model style dictionaries with recursive components.Available from download sitedirect link:+ includes spreadsheets and sample blueprintSampling of available dictionaries. Those marked in bold are model style dictionaries with recursive components.XMLXMLXMLXMLXMLXML
10 Drag and Drop dictionary into Spreadsheet Item namesParent item indicatorList of Child itemsDrag and Drop XML dictionary format into Microsoft ExcelChild item indicatorThis shows the components and concepts that underpin the dictionary, viewed as a spreadsheet. This provides a familiar interface for business data analysts to work with. Simply take the XML dictionary file and drag and drop it into Microsoft Excel, click “OK”, and it will open as a workbook.Item PropertiesUnique ID
11 Creating your own Dictionaries The CAM toolkit includes dictionary generation toolscan harvest components from existing exchange schemawill import model components from ERwin modelling toolcan ingest and use subset schema or component schemaA separate companion Quick Guide describes the dictionary creation processes and toolsLook for existing dictionary packs from Sourceforge files location of CAM download
12 Using DictionariesDetermine your business information exchange components at conceptual levelSearch and locate candidate components from appropriate domain dictionary collectionsCatalogue the parts to be usedDictionary components can be referenced individually or as collections by an assembly blueprint that puts them all together to create a complete information exchangeComponents can be selected from multiple dictionariesNote any new extension pieces as neededA control file configures which physical dictionary files to reference and associates one or more particular namespace prefixes with each dictionaryThe download packs available from Sourceforge contain everything you need to get started; dictionaries, control files, sample blueprints and instructions.
13 Dictionary Control File Associates physical dictionary files to namespace prefix detailsThis is only required if you are setting up your own new dictionaries from scratch. You can of course review control files to see what dictionaries are referenced.Control files should be manually edited in a XML editor tool to make configuration changes as needed (e.g. arrowed text areas).
14 Introducing Exchange Assembly Blueprints Intelligence Commanders TrainingIntroducing Exchange Assembly BlueprintsWhat is a blueprint templateExample blueprintsBlueprint tools overviewHow do I create a blueprint?How do I use it with CAM?Foundations of Intelligence14
15 Introducing Blueprints A blueprint is the outline of your information exchange and the top level sets of components neededAllows rapid iterative exchange structure designStart with a basic idea of what is needed, extend and refine as you clarify groups of components and their relationshipsCan import components from one or more domain dictionary collectionsAllows you to focus on the conceptual and hierarchical information model components and their purposeThe low level specific information details will be filled in for you by the expander tool from the matching dictionary definitionsReview results and add local extensions / exclusions as neededBlueprints themselves also have high re-use valueCan be sub-assemblies and patterns not just exchange models
16 Example Blueprints OASIS EDXL HAVE message LEXS messaging blueprint Business functional componentsLEXS messaging blueprintReusable messaging envelope constructsIndividual componentMessage handling,delivery and controlPayload goes hereTop level sets of business information componentsthese examples available from CAM editor install package~ CAMeditor\eclipse\workspace\CAMEditor\dictionary\blueprints\
18 How do I create a blueprint? Use CAM Eclipse visual editorcreate basic blueprint from scratch or from XML instanceClone and edit an existing blueprintuse a XML editor to manipulate raw XML of blueprint (e.g. search and replace namespace details)modify blueprint using CAM Eclipse visual editorBeware of namespaces!You need to pre-declare namespaces in the blueprint – that may appear later in the finished expanded exchangeUse CAM Eclipse visual editor root node menu to add namespace definitionsTip: use your XML editor to cut and paste namespaces into raw blueprint XML file directlyA blueprint is nothing more than a regular CAM template stripped of everything except the structure section and a top level outline.
19 Blueprint Structure Overview Short Name for ExchangeRoot node name and optional namespaceXML container elements to facilitate use patternsDictionary component referencesA blueprint is nothing more than a regular CAM template stripped of everything except the structure section and a top level outline.Dictionary Type annotation reference(note red paper clip icon)Namespace declarations
20 Adding Components from a Dictionary Understanding the 4 basic types of component referencing availableDirect component (BBIE)Direct collection (ABIE)Indirect type (extension)Indirect sub-component (contextual re-use)Control of the XML structure hierarchyGrouping collections of componentsExtending an existing componentBuilding new extension from existing componentsNamespaces – when to use them
21 Example XML Techniques (Immigration Alien Alert example blueprint)Direct collections referencing (ABIE)Grouping collection of componentsDirect component referencing (BBIE)Building new component from existing onesIndirect type extension via annotation Type(red paper clip icon denotes annotation)Namespaces – root namespace optional (aaa)Dictionary component namespaces are required
22 Making New Blueprint from scratch Assign root elementOpen new template21Complete structure by adding elements and dictionary components3
23 Running Blueprint Expander Intelligence Commanders TrainingRunning Blueprint ExpanderBlueprint check listUsing Expander tool dialogueReviewing resultsHints and TipsFoundations of Intelligence23
24 Using Your Blueprints What you need - Getting Started CAM Eclipse visual editorBlueprint templateDictionary or dictionaries of componentsDictionary control fileContains linkage of namespace prefix to each dictionaryContains physical filename path of each dictionaryGetting StartedInstall CAM editor tool from Sourceforge.netOpen example blueprint from EDXL folder~ CAMeditor\eclipse\workspace\CAMEditor\dictionary\blueprints\EDXLTry blueprint expander with control fileEDXL-dictionary-control-file.xml
25 Running Blueprint Expander This example is available in the NIEM dictionaries download pack from SourceforgeSelect Dictionary Control FilePick output result filename (default is-expanded.cam)Determine rule generation selections from dictionary definitionsNote: entries are “sticky” so are remembered for next time
27 Reviewing Results Exchange Structure items 2 1 Rules for each item DomainComponents Namespaces3
28 Hints and Tips Blueprint Remedy Node not expanded, or removed! Override default dictionary structure context definition using an annotation Type reference to the node instead – see Alien alert example blueprint – aaa:OfficerDetailsAdded parent and child combination. All children added instead of just one.Expander inserts all children of parent. Either delete unwanted children from expanded result (see next section), or create custom parent node, and pick children components directly – see Alien alert blueprint example.Node not expandedMake sure namespace prefix and spelling exactly matches dictionary and dictionary control file prefixes.Undefined namespace prefix error when opening expanded template in CAM editorExpander added dictionary components from another namespace. Declare that namespace prefix back in blueprint and re-run expander againExpanded result missing all desired sub-componentsCreate your own blueprint extension parent and combine parts to create desired result – see Alien alert example – aaa:FurtherDetailsExpanded result has too many sub-componentsIn expanded result mark unneeded parts with “excludeTree” rules, then run File / Export / Compress to strip out; or use extension technique above to pick only few child elements instead of whole parentSome of these techniques are blueprint ones, others involve the expanded result template. Next section covers editing in more details.
29 Completing the Exchange Template Intelligence Commanders TrainingCompleting the Exchange TemplateAdding nodesDeleting nodesEditing RulesAdding annotationsFoundations of Intelligence29
30 Exchange Template Editor 21Rules for each itemCan add new custom domain Items3May need to add a namespace for new items4
31 Deleting Exchange Components Select item to be removed1Right click for popup menu2From Rules exclude item or tree34After marking all items to be excluded, save template, then runFiles / Export / Compress template to physically remove them.
32 Editing Rules 1 Select item to add new rule Right click for popup menu 2Use rule wizard to complete3
33 Adding annotations 1 Select item to add new annotation Right click for popup menu2Use annotation wizard to complete3
35 Exchange Generation Steps EXCHANGE TEMPLATEStructureRulesDocumentationToolkit for exchange artifacts generationRun template Evaluation ReportCompare to industry dictionary- create mapping spreadsheet- create crosswalk xmlGenerate business rules reportGenerate exchange XSD schemaGenerate XML test instance(s)123Suite of menu options and tools in desktop CAM toolkit editor(each run custom xslt scripts on the exchangeCAM template xml)45CAM Template
36 Using NDR Evaluation Report Provides scoring, alerts, warnings and potential issues including:Naming and content model conventionsNaming and restriction consistency checksInteroperability enablers/inhibitors checksRules integrity and duplicatesStatistics on exchange sizeSpell checking on component namesAccess from Tools menu, Evaluate Template option
37 Evaluation Report (NDR) example Part of an example report for LEXS getDataItemRequest template displayed in toolkit HTML viewer
38 Compare to industry dictionary References industry dictionary of names and propertiesMatches on physical namesReports mapping detailsCompatible with Microsoft ExcelReport can be used to do spell checkingGenerates crosswalk xml fileWhen NIEM-dictionary selected:generates NIEM SSGT wantlist.xml fileprovides % re-use statistics for NIEM components( note NIEM-dictionary.xml is located in install directory~ CAMeditor\eclipse\workspace\CAMEditor\dictionary folder )
39 Example cross-reference spreadsheet Formatted view in Microsoft Excel of import of cross-reference report details (from generated XML file)
40 Generate Documentation and Schema Create HTML report of exchange schema details and associated content and business rulesReport layout and content designed to be reviewed by business analystsSchema:Generate XSD schema for exchangeCustomizable exchange folder layout management by namespace for extension, subset and exchange schema componentsWrites XSD schema in syntax that is clear, simple and compatible with deployment tooling environments
41 Business Rules Documentation Part of the example rules for LEXS getDataItemRequest template displayed in toolkit HTML viewer
42 Export Exchange to XSD Schema Export Template to Exchange XSD:Completed Exchange StructureSet Exchange Output OptionComplete set of exchange schemas generated
43 Exchange Schema Generated Each namespace file is import for those specific type definitionsSet of XSD files with filename and namespace suffixReviewing XSD results in a schema editor tool
44 XML Testing Examples Generation Support for software development testing processDesigned to allow creation of concrete realistic examples not just random value basedHinting system allows insertion of actual test system values into XML examplesCan create both valid and invalid examples to support unit testing of application softwareExclude capability allows generator to create examples that contain only a portion of the entire exchangeControl over random seed value used allows re- generation of identical test cases
45 XML example generation wizard View of CAM toolkit with LEXS getDataItemRequest and dialogue for XML test example generator tool
46 Running validation rules tests Built-in CAM validation engine allows testing of XML instances against actual exchange rules (CAMV).Critical to ensure that the exchange validates actual live production example scenarios correctlyAllows deployed solution to match exchange schema detailsErrors can be reviewed interactively in exchange visual interfacePost-processing of validation results allows unit regression tests to be created with reporting of errors, warnings and information level notes
47 Run Exchange Template Pick XML test case to validate Run validation Review results in visual editor
48 Example Exchange Packaging Details Package ArtefactDescriptionCAM Generates FilesNDR Evaluation ReportEvaluation of exchange template compared to set of Naming and Design Rules (NDR) and interoperability factors / principlesCrosswalk XMLItemized list of each dictionary component element and attribute included in the exchange.Exchange SchemaBase document schema that defines the full XML structure for the exchange and is generally named after the exchange itself.Extension SchemaSpecification for extended components—separate local name-spaces of components not contained in dictionarySample XML InstanceExample instance(s)DocumentationMaster DocumentationThe Master Document is the main document for which all of the context and details around the exchange are explained. This document includes, the overview, as well as details surrounding the exchange, business drivers and requirementsBusiness rulesBusiness rules in plain or structured English, generated by CAM development tool.Mapping to DictionaryMapping of domain components, tagged with constraints (i.e., cardinality, etc.) to dictionary components as a spreadsheet.Extended componentsComponents created because they were not in dictionary—may be part of mapping spreadsheet and include structure and definitions of new components.
49 Summary Dictionary driven exchanges Blueprint enabled reuse Intelligence Commanders TrainingSummaryDictionary driven exchangesBlueprint enabled reuseAutomated exchange package generationAlignment to NDR Principles and RulesTesting and validation supportFoundations of Intelligence49
50 Review Top Down development NDR Principles and Rules Reference dictionary componentsCreate exchange blueprintRun Expander toolRefine desired structure in visual editorNDR Principles and RulesBest practices for interoperability and schema techniquesDictionary driven reuseSelect Enterprise Data Model and industry componentsEnsures consistency of definition and useAutomated exchange package generationSchemas, XML, documentation, mapping crosswalkTest generated example XML with rules validation
51 Links and Resources CAM Toolkit download NIEM dictionaries and sample blueprintNIEM tools and IEPD tools tips and hints