Presentation on theme: "Audit Control Environment Mike Smorul UMIACS. Issues surrounding asserting integrity Threats to Integrity of Digital Archives –Hardware/media degradation."— Presentation transcript:
Issues surrounding asserting integrity Threats to Integrity of Digital Archives –Hardware/media degradation –Hardware/software malfunction –Operational errors –Security breaches, malicious alterations –Technology evolution –Object transformation (format obsolescence) –Infrequent access to most data
Using Hashes to Monitor Files Strong hashes can assert a file has not changed How to manage millions of hashes? How do you prove the hash value hasn’t changed? How do you prove a hash value was issued at a given time?
Audit Control Environment (ACE) Solves the problem of storing and verifying hashes. Secures hashes by issuing token for each file/hash to me monitored. Tokens contain a cryptographic proof that allows for 3 rd party auditing. One number stored externally is used to audit tokens and hashes.
Hash Authentication Hash 1Hash 6Hash 5Hash 3Hash 4Hash 2 Intermediate Hash Value Previous Round Hash CSI (one hash value) Challenge Hash IHV Gather Hashes During Round Create Merkel Tree For Supplied Hashes Link to previous roundGenerate proof for hash
How to scale? Two layers of Merkel tree 1.Short rounds(seconds), that generate Cryptographic Summary Information(CSI). 2.Each successive round includes previous CSI 3.Second, daily rounds comprised of all CSI’s for previous day. Daily tree root, called Witness can validate all CSI’s for a day. –Only 365/year generated. Very manageable! Two components, an Integrity Management Service(IMS), and Audit Manager(AM) were developed.
Components Integrity Management Service (remote) –Runs all hash aggregation, round generation, witness publication. –Stores CSI values –Generate proofs from CSI to witness –ims.umiacs.umd.edu Audit Manager (local) –Monitors local files –Determines audit policy –One or more per archive –Locally stores hashes and tokens
ACE Audit Audit Local Files: Audit Manager periodically scans all files and compares stored digests with computed digests. Assume valid hashes in database Audit Local Manager: Manager computes round summary for each digest using that digest and its token. This is compared to value stored on the IMS. Assume IMS returns valid summary information, do not trust hashes in database IMS Audit: Round summaries are used to compute witness values. These are compared with offsite witness values. Do not trust IMS, force IMS to prove its CSIs link to a witness
Audit Manager Downloadable, one or more per archive Monitors local files Simple Requirements –Java 1.6+ –Tomcat –MySQL Managed by archivist/librarian after install Monitor multiple collections on different architectures Hides all the complexity you just saw!
ACE Audit Manager Dashboard for collection monitoring
Logging All events logged –Files offline, corrupt, re-available –Audit times, last seen
Error Reporting View all files that are not perfectly intact Remove files from monitoring. View events associated with files
Audit Manager Features Compare collections to hash list –Same/different names, same/differing digests Export collection details –Hash export, wget crawl export JSON interface for embedding statistics in 3 rd party websites Gateway to data
Fun with hashes Ensure everything was uploaded –Accepts a bag-it style manifest (hash + path) Extract all the duplicates –More exist than you think Ensure collection replicas are complete Locate renamed files
Performance Audit Manager (1.1beta3) 1.25 million false digests (no bytes read) –Registration: 3h, 6m (112 files/s) –Audit: 1h, 15m (277 files/s) 1.25 million false data files (1.25Tb data) –Registration: 5h, 7m (67.8 files/s, 67.8MBytes/s) –Audit: 4h, 30m (77.2 files/s, 77.2MBytes/s) In practice, bottleneck tends to occur at archival resource, not AM.
Chronopolis test Three sites –UMD, NCAR, SDSC Three ACE AM installations Independent monitoring at all sites 30 day audit policy Over 17Tb and 5.5 million files
Chronopolis Test ProviderFilesSize(GB)Time(h)Files/sMB/s CDL46,7624,29120:32.6359.44 SIO-GDC197,7188156:498.0534.00 ICPSR4,830,6256,957122:4810.9316.11 NC-State608,4245,46532:145.2448.22
ACE Summary High performance, Scalable 3 rd party auditable Version 1.0 publically available –Support for local files, SRB, iRODS http://adapt.umiacs.umd.edu/ace Standalone client available