3Version control Quality control: “Something adheres to some criteria” date xbatch no. xingredients xcalories x℮ gram xQuality control:“Somethingadheres tosome criteria”Without a properly definedsomething there is no basiswhatsoever for quality control.So, industrial QA always definessomething: e.g. all peanut butterjars have a unique number on them.We should add unique version no.sto data and tools tooSubversion (svn) does this for usSvn is a necessary(but not sufficient)requirement for achievingqualitydate √batch no. √ingredients √calories √℮ gram √
4SubVersion SubVersion is a web-based system for version control It allows a worldwide community of people to collaborate onlineYou can think of a SubVersion repository asan ftp drive with a memorysharepoint environment for any file format (not just *.doc, *.xls).Everyone can add/alter/remove and retrieve dataBut nothing can be added/altered/removed without being noticedbecause SubVersion logs everything whichguarantees a safe way of collaborating without need for editorsThis is exactly how Wikipedia operates:no editorial boardsimply anyone can participate, but not unnoticedeverything is loggedSo SubVersion is a kind of ‘wikitoolia’ (or ‘toolipedia’ if you like)
5SubVersionWikipedia keeps the history of all modifications, and so does Subversion
6SubVersionSubVersion servers are for instance hosted by SourceForge.orgDeltares also hosts SubVersion serversOpenEarth has a free ride on a Deltares serverEveryone is invited to launch their own server for internal useThe OpenEarth server is for open toolsproof-of-concept for subsequent internal clones by partnersoperational server for open dataSubVersion was originally meant for computer code.Recently it is also used for model input schematisationsAnd now OpenEarth has adopted it to hostraw data + scripts as too
7SubVersionSubVersion is a tool to synchronize a local copy with a server copyA SubVersion client is needed to :Get updates from the server to a local copy: checkoutPut updates from a local copy onto a server: commitThe server mother-copy can be viewed with a web-browser onlyThe server copy cannot directly be usedThe server copy only shows the latest version, not the historyA SubVersion clientcan get a local copy to work withcan show the history of all modificationsUnlike Sharepoint: multiple people can work simultaneouslySubVersion makes sure that changes are merged without conflicts.
8REPOSITORY basics One central mother-copy is synchronized two-way with many copieson local PC’s worldwide.centralmother copyinternetsynchronizemany localPCs worldwidea local copyE:\a local copyD:\a local copyD:\a local copyF:\a local copyF:\a local copyF:\a local copyE:\a local copyD:\
12Repository usernameGet username and password: register atWhy, OpenEarth is open, right? Yes, but closed communityFor best quality all actions are logged (just like wikipedia)Nothing can be lost, only temporarily disabledSo anyone can be allowed to joinEvery file is logged …… and every line in every file is logged.
13REPOSITORY browse (WebSVN look) WebSvn offers a special web-based code syntax highlighting. This is not the real file url though.
14REPOSITORY browse (WebSVN look 2) The direct acces url does not offer syntax highlighting, but this is the real file url.
15other browser looks exist too (example 1) REPOSITORY browse (plain look)other browser looks exist too (example 1)
16other browser looks exist too (example 2) REPOSITORY browse (sourceforge look)other browser looks exist too (example 2)
17REPOSITORY checkout Not handy to get files one by one with browser Get them all at once with free programJargon:Checkoutbrowsecentral database: svn.oss.deltares.nlcheckoutupdatecommitadddeletelocal copyD:\ E:\ F:\copy
18REPOSITORY checkoutDownload and install Tortoise (http://tortoisesvn.net/)
19REPOSITORY checkout Make a checkout in e.g. F:\checkouts\ No need to back this up, it’s only a copy ...After installation of TortoiseSVN and rebooting, new icons show-up when right-mouse clicking on a directory.
20REPOSITORY checkout Copy url from browser (case sensitive!) Make directory tree of local copy mimics serverFor WebSVN interface click button
21REPOSITORY checkout Wait for checkout to finish. OpenEarthTools is a few 100 Mb.
22REPOSITORY checkout Do NOT use the option “Create a repository here” This is meantfor hosting amother copy onyour machine
23REPOSITORY commit browse central database: svn.oss.deltares.nl checkoutupdatecommitadddeletelocal copyD:\ E:\ F:\copy
24REPOSITORY commitA commit is needed to put local changes into the central server.This is needed if you locally did a meaningfull …modificationdeletetionadditionmoverenameEach action isindicated with aunique icon in yourwindows explorerNon-meaningfull modifications (e.g. tests)can be reverted (undo all local changes).folders files
26REPOSITORY commitAlways put a small remark to describe to comunity members what you modified (“did a lot” is not specific enough).
27REPOSITORY update browse central database: svn.oss.deltares.nl checkoutupdatecommitadddeletelocal copyD:\ E:\ F:\copy
28REPOSITORY updateGet updates from others from the server to your local copy.Do this regular to benefit from improvements.For active OpenEarthTools use: daily!Always do this before a commit.
29REPOSITORY updateSubVersion: open source version control system. Users ‘commit’ their files in one central database (regular update local copy). Every commit gets a unique revision number. Per commit one can add a comment to indicate what was changed.
30REPOSITORY statistics TOOLS:> 5000 commits!~ 100 active users~ 1000 user requests(incl XBeach)DATA:> 5000 commits!~ 100 active usersEU MICORE deliverablesBwN deliverablesStatistics: Per project or tool one may create a separate ‘repository’. Bundling reusable tools into 1 central repository has huge advantages (sharing, collaborating, learning). OpenEarth tools, as used in Building with Nature, is open source en freeware.
32REPOSITORY add a raw dataset There are 2 copies of 1 file on your PC:Visible working copy, for editingHidden shadow copy, to detect changesBefore adding a file to the server, a shadow copy must be created.Allowsforofflineworking
33REPOSITORY add a raw dataset Now the addition must be simply be committed as any change
34REPOSITORY add a raw dataset OpenEarthRawData is very big: don’t make a full checkoutTo add a thing, first make an empty checkout of the destination.
35REPOSITORY delete browse central database: svn.oss.deltares.nl checkoutupdatecommitadddeletelocal copyD:\ E:\ F:\copy
36REPOSITORY add a raw dataset There are 2 copies of 1 file on your PC:Visible working copy, for editingHidden shadow copy, to detect changesWhen deleting a file on the server, your shadow copy be informedAllowsforworkingoffline
37REPOSITORY add a raw dataset Now the deletion must be simply be committed as any change
39REPOSITORY copyAgain: first inform shadow copy locally, then commit to server …Drag with right-mouse button
40OpenEarthTools Tools are stored under https://svn.oss.deltares.nl/repos/openearthtools/trunk/Tools are stored per languageMatlab has main activitybut Python and R toolbox are starting to grow now.
41OpenEarthTools: Matlab: start The Matlab toolbox has to be added to your matlab pathDo not use addpathThis wil also add all hidden (.svn) directories (a Matlab bug)This does not add the required JAVA paths (netCDF-4)Run oetsettings.m insteadYou have to do this every time you start MatlabSo it’s easier to make a shortcut button this, right-mouse click on the shortcuts toolbar to add one
42OpenEarthTools: Matlab: help You can find the available Matlab tools the same way to you find official Matlab tools using use help, doc or lookfor>> help oetsettings>> doc googleplot>> lookfor convertcoordinates
44OpenEarthRawData Raw data are stored under https://svn.oss.deltares.nl/repos/openearthrawdata/trunk/Data are stored with copyright holder as main directory.This allowscopyright holders to maintain their own datacopyright holders to shift easily from private to open sourceusers to identify whom to acknowlegdeData should also containdedicated processing scripts (if not in OpenEarthTools)url file to web sourceINSPIRE XML meta-data file
45OpenEarthModels Model input specifications are stored under https://svn.oss.deltares.nl/repos/openearthmodels/trunk/Model input schematisations are stored with copyright holder as main directory.This allowscopyright holders to maintain their own datacopyright holders to shift easily from private to open sourceusers to identify whom to acknowlegde
46Exercise The repository is supposed to be working anytime Do not play with the actual repositoryAll advanced users will by annoyed by thisBut then, how I can I learn how to work with it?Solution: use the sandboxPlay around at the highest level as much as you likeAnd clean up afterwards (delete)With your browser:https://svn.oss.deltares.nl/repos/openearthtools/sandboxNow delete the addition you made inAnd check the log file, to see what colleagues did.
49netCDF/OPeNDAP how to use: opendap.deltares.nl: www.OpenEarth.eu: Simple OPeNDAP primerSome more sophistication: subsetting with OPeNDAPThese are tutorials that subset the North Sea from a wordwideDigital Elevation Model (DEM)