1. Collaborative development & Source Code Repository
October, 2015
SDMX Tools Task Force

2. Objectives of this presentation
Eurostat:
What is CITnet
SDMX tools set at CITnet (Confluence, Jira, Stash/Git, Nexus)
How to work with CITnet GIT repositories (example of SDMX-RI Tutorial)
How to deal with dependencies (example of SDMX-RI Tutorial)
Participants:
Comment on the proposal
Collaborative development and governance
October, 2015
SDMX Tools Task Force

3. What is CITnet?
October, 2015
SDMX Tools Task Force

4. CITNet tools for collaborative development
CITNet provides:
Source Version Control Software
Issue tracker
Code explorer and Collaborative Code Review
Software code repository management
Continuous integration
Agile Iteration Management and Planning
Collaborative documentation
October, 2015
SDMX Tools Task Force

5. SDMX tools at CITnet Uses Stash from Atlassian
Project URL for each tool
E.g. SDMX-RI:
dmxsource/
3d77a
GIT URLs format:
Notes:
1. Stash is highly integrated with CITnet Jira
2. One needs a CITnet account to use Stash / Jira
3. A single account for both Stash and Jira
October, 2015
SDMX Tools Task Force

6. How we develop Project phases, planning and execution
Each requirement should be traceable and test case covered
October, 2015
SDMX Tools Task Force

7. Branching strategy Main branches in most projects
Release branches X.Y.Z
JIRA related branches: e.g. SDMXRI-XY
SdmxSource (JAVA) – special case
October, 2015
SDMX Tools Task Force

8. Practical and ready to use
Agile and KTC approach
Practical and ready to use
Confluence
Jira
Git, Nexus
Know The Customer approach
October, 2015
SDMX Tools Task Force

9. Development & supporting tools
October, 2015
SDMX Tools Task Force

10. Collaboration Projects & Status
B. SDMX related actions ( )
Collaboration Projects & Status
Project
Status
SDMX-RI & .STAT (OECD)
Under arrangement
SDMX-RI & ISTAT
Established
SdmxSource & SDMX-RI & DSWS
Transformation service
STRUVAL
Ongoing, planned Q1 2016
SdmxSource & SDMX Converter & EDIT
SDMX-RI & PC-Axis Sweden
Under arrangement (on hold)
October, 2015
SDMX Tools Task Force

11. Collaborative development in CITnet
User registers in CITNet (request registration)
Requests to have access to a project (Stash/GIT)
ESTAT includes the user to a group having Read access
Registered users can
clone, browse and fork any repository within the project
create and contribute to pull requests targeting any of these repositories
Request merge
October, 2015
SDMX Tools Task Force

12. Collaborative development
Link
Shared development welcome
Ask for access and inform on scope and planning
Deliver and request merge
Eurostat:
Assess the possibilities to merge and publish as part of the tools
In case of tools discuss with the system suplier how to publish and ensure evolution
System supplier
Deliver project supporting documentation
Ensure bug fixing and support
The expert group: Create guidelines and governance structure
October, 2015
SDMX Tools Task Force

13. Governance proposal – simple and practical
Define
Actors
Roles
Responsibilities
Scope
Time
Procedure
Ensure
Evolution & Quality
Transparency
Information sharing
October, 2015
SDMX Tools Task Force

14. Items for discussion (1/4)
Collaborative development
Proposal: Based on already established projects, launch a pilot and work on best practices
Discussion:
Recommendations:
Best practices
Proposal: Expert group to create guidelines, project related templates and governance structure
October, 2015
SDMX Tools Task Force

15. Items for discussion (2/4)
SDMX development
Accessibility and usability
Proposal: Use CITnet for information sharing and development; Store all related documentation there and give access rights
Discussion:
Recommendations:
Software sharing
Proposal: Updated JoinUP; use Nexus for binaries; use Stash/Git for source code;
October, 2015
SDMX Tools Task Force

16. Items for discussion (3/4)
SDMX tools governance
ESTAT SDMX tools set
Proposal: Owned, maintained and supported by ESTAT; Use-cases relevant to the community implemented by ESTAT
Discussion:
Recommendations:
Enhanced ESTAT SDMX tools
Proposal: Bug fixes and support by the system supplier; Eurostat assesses the implementation; Merge if feasible; Maintain based on community needs
October, 2015
SDMX Tools Task Force

17. Items for discussion (4/4)
SDMX tools governance
Other SDMX tools and frameworks (not based on Eurostat tools)
Proposal: Keep informed; Assess integration based on user need and relevant use-cases for the community; Merge if feasible
Discussion:
Recommendations:
October, 2015
SDMX Tools Task Force

18. Thank you! Questions?
October, 2015
SDMX Tools Task Force

19. Working with SDMXRI repositories
Step 1. Get the URL to clone.
Visit a repository. E.g. MappingAssistant
Click on “Clone” button.
Copy the HTTP URL.
October, 2015
SDMX Tools Task Force

20. Working with SDMXRI repositories
Step 2. Clone
Using a GIT tool (e.g. Git Extensions, SourceTree)
Or command line:
Important note: When cloning we are using the URL from Step 1
git clone –b <existing branch name>
October, 2015
SDMX Tools Task Force

21. Working with SDMX-RI repositories
Making changes
Fork
Clone the fork repository.
Make code changes
Add and commit changes
Push to CITnet (and other remotes)
Create a Pull request
Discuss, possibly go back to step 2
The changes are merged
October, 2015
SDMX Tools Task Force

22. Working with SDMX-RI repositories
Fork
From Stash
Click at Fork link.
To fork you go via Stash to the repository you want to fork and click “Fork” link.
2. A form will appear, where you can select
a. the project (usually your own personal project space)
b. the name of the forked repository and if we want enable fork syncing, which normally we want.
3. After clicking “Fork repository” the new forked repository will be created and will be shown in the browser.
After this step you can clone the forked repository.
4. Then we can clone as we did before.
If we already have a local repository with changes then in most cases we can just change the remote.
October, 2015
SDMX Tools Task Force

23. Working with SDMX-RI repositories
How to create a branch ?
Using CITnet Stash or JIRA
And then Clone that branch
GIT tools
Command line
After a Clone.
We can also create a branch.
There are a few ways to create a branch. E.g. from Stash or JIRA, or some GIT tool e.g. GitExtensions or Visual Studio or the command line.
You need to check out to the branch after creating it.
There is also a single command do to that, using –b parameter in git checkout to create and change to the new branch.
git branch <branch name> # create a branch
git checkout <branch name> # change to a branch
October, 2015
SDMX Tools Task Force

24. Working with SDMX-RI repositories
Make codes changes
October, 2015
SDMX Tools Task Force

25. Working with SDMX-RI repositories
Add modified/new files
Commit
git add path/to/modified/file.cs
git commit –m”SDMXRI-xyz Implementation of the make coffee interface.”
Staging:
First we stage files to be committed.
Then we commit them..
Note: for those with experience in centralized source control software this only commits them locally..
October, 2015
SDMX Tools Task Force

26. Working with SDMX-RI repositories
Push changes to CITnet
If the branch does not yet exist on CITnet
Else just push
git push –u origin <branch name>
Push changes:
To apply the changes from the local repos to the CITnet repository e.g. the forked repository we need to push them.
If the branch does not exist on CITnet we need to tell GIT to create it on CITnet when pushing. Else we just push the changes.
git push
October, 2015
SDMX Tools Task Force

27. Working with SDMX-RI repositories
Create a pull request
Using Stash
Discuss
To create a pull request and discuss about it we need to use Stash.
October, 2015
SDMX Tools Task Force

28. SDMX-RI Dependencies 3rd party dependencies available via NuGET
For NuGet 2.7+ deps downloaded automatically
For NuGet prior Enable NuGET Restore
SDMX-RI uses NuGet for almost all dependencies.
2. Normally if you are using the latest NuGET then dependencies should be downloaded automatically else NuGet Restore will be needed.
3. Since the current target is Visual Studio 2010, “NuGet Restore” may be needed even for Visual Studio 2013.
October, 2015
SDMX Tools Task Force

29. SDMX-RI Dependencies Inter SDMX-RI Dependencies
Only when building from GIT
NuGet is used for resolving SDMX-RI libraries
All SDMX-RI libraries produce NuGet packages
No central repository (yet)
Needs a local NuGet repository
Correct order and correct branch matching needed
Dependency version == branch name
E.g. DataRetriever v5.3.0 is in branch in the dr.net repository
Internal dependencies:
1. Inter-SRI dependencies we mean the dependencies that one SDMX-RI project has to another, e.g. Data Retriever depends on SdmxSource and NSI WS depends on both SdmxSource and Data Retriever.
2. While the SDMX-RI packaged source code uses NuGET only for external dependencies, when using GIT the inter-SDMX-RI dependencies are also resolved with NuGet.
3. This difference exists because the packaged source code aims for simplicity, simply link to a DLL, while when developing SDMX-RI NuGet is very useful, but it has complexities.
4. The complexities are: there is no public central repository. NuGet does not have a default local repository like for example maven.
So in order to work with GIT code one has to create a local repository, which could a folder on the local/shared disk or full web based repository.
October, 2015
SDMX Tools Task Force

30. Inter SDMX-RI Dependencies
Repository
NuGet package name
SDMXRI Dependencies
sdmxsource.net
SdmxSource.NuGet
None
estat_sdmxsource_extension.net
Estat.SdmxSource.Extension.NuGet
commonui.net
EstatCommonUI.NuGet
pcaxis.net
PcAxis.NuGet
common.net
MappingStoreRetrieval.NuGet
dr.net
DataRetriever.NuGet
October, 2015
SDMX Tools Task Force

31. Inter SDMX-RI Dependencies
Repository
NuGet package name
SDMXRI Dependencies
sr.net
NuGet.StructureRetriever
MappingStoreRetrieval.NuGet
SdmxSource.NuGet
Estat.SdmxSource.Extension.NuGet
PcAxis.NuGet
MappingAssistant
EstatCommonUI.NuGet
DataRetriever.NuGet
October, 2015
SDMX Tools Task Force

32. Inter SDMX-RI Dependencies
Repository
SDMXRI Dependencies
NSIClient.NET
SdmxSource.NuGet
Estat.SdmxSource.Extension.NuGet
NSIWS.NET
MappingStoreRetrieval.NuGet
DataRetriever.NuGet
NuGet.StructureRetriever
PcAxis.NuGet
October, 2015
SDMX Tools Task Force

33. Inter SDMX-RI Dependencies
Repository
SDMXRI Dependencies
TestAuthConfig.NET
MappingStoreRetrieval.NuGet
SdmxSource.NuGet
Estat.SdmxSource.Extension.NuGet
EstatCommonUI.NuGet
TestClient.NET
DataRetriever.NuGet
NuGet.StructureRetriever
PcAxis.NuGet
October, 2015
SDMX Tools Task Force

34. Dev Environment (.NET)
Visual Studio 2013 Professional/Community or better
Re-Sharper might help with SdmxSource... But not free
GIT
Basic support build in Visual Studio 2013.
We use MSysGit + Git Extension.
API Documentation
Doxygen
Sandcastle Help File Builder
Ideal Dev Environment:
An ideal development environment for developing SDMX RI and SdmxSource would consist of :
1. Visual Studio, preferably Visual Studio 2013 professional/community or better.
Currently we target Visual Studio 2010 but we develop in both in 2010 and 2013.
2. Because of the size of SdmxSource some tool like ReSharper does help navigating e.g. finding an interfaces implementing classes or refactoring. But it is not free and requires annual subscription.
3. To access GIT repositories.
a. Visual Studio 2013 provides some basic support
b. we use msysgit (Command line) and GitExtensions which has a nice GUI and Windows Explorer and Visual Studio integration.
4. If someone is interested in producing API documentation. There are two solutions:
a. One use Doxygen which can produce very nice documentation with diagram and such but currently it cannot handle SdmxSource… b.Sandcastle Help File Builder is another option which is used for SdmxSource.
October, 2015
SDMX Tools Task Force

35. Dev Environment (Databases)
SQL Server >= 2005, Oracle >=10 and MySQL >= 5.5
For SDMX-RI related development.
Tools
SQL Server Management Studio 2014
Oracle SQL Developer 4.1
MySQL Workbench 6.3.
For SDMXRI development, currently the minimum supported versions are :
1. SQL Server 2005,
2. Oracle 10
3. MySQL 5.5.
But since those are very old versions one can of course use a more recent version.
Each DB vendor provides an environment to assist with database development and maintenance:
1. Microsoft provides SQL Server Management Studio,
2. Oracle provides the SQL Developer for Oracle
3. MySQL there is Workbench.
Of course there are many other tools that could be used with more or less functionality.
October, 2015
SDMX Tools Task Force

36. Support strategy
CITnet Jira tool employed as single (ESTAT) repository of
Bugs
Support requests
Suggestions for enhancements
Other CITNet elements
Source code repository
Wikis
Support levels
1st level: central problem registration and reporting
2nd level: problem analysis and resolution
3rd level: bug-fixing and new feature analysis (per product)
October, 2015
SDMX Tools Task Force

37. SDMX Tools in CITnet / Project structure
Current/
Change
Branch
GIT Clone URL
Create Pull
Request
Clicking to a repository e.g. MappingAssistant we see the source code of Mapping Assistant.
2. Above that we see the current branch and clicking at it allows us to select another the branch.
By default the master branch is selected.
3. On the top right we can get the “clone URL” by clicking at the “Clone” button.
4. You can create a pull request by clicking at the” Create pull request.”
5. You can create a fork by clicking at the fork link.
Fork
Files (e.g. Source Code.)
October, 2015
SDMX Tools Task Force

38. Branching in Stash/Git
Selecting a branch
Creating a branch
Stash/Git
The branch menu allows to search and select branch.
It is also possible to search and select a tag.
Jira
October, 2015
SDMX Tools Task Force