Presentation is loading. Please wait.

Presentation is loading. Please wait.

Your Results From Disk to Device (Preview) Amit Chourasia Mona Wong-Barnum Michael L. Norman San Diego Supercomputer Center, UCSD Presented at: XSEDE 13,

Similar presentations


Presentation on theme: "Your Results From Disk to Device (Preview) Amit Chourasia Mona Wong-Barnum Michael L. Norman San Diego Supercomputer Center, UCSD Presented at: XSEDE 13,"— Presentation transcript:

1 Your Results From Disk to Device (Preview) Amit Chourasia Mona Wong-Barnum Michael L. Norman San Diego Supercomputer Center, UCSD Presented at: XSEDE 13, San Diego, CA, Jul 23, 2013

2 Stream Encode Explore Disseminate My Experiments

3 Computation Cycle Compute/Ingest Post Process Analyze & Save Monitor ? Validate ? Abstract ?

4 Derived Products MonitoringValidationAbstraction

5 Direct AccessDirect Sharing Not All Members Have HPC Access Security Implications On HPC Derived Products

6 Concise derived products Plots & Text Documents Image Sequences & Videos Share with the research group 1.Download derived products 2.Share with collaborators a. snippets b.Set up a webpage – Add content –

7 Derived Visual Content Survey Derived content for a single research group in respective domain

8 Pitfalls in sharing derived content Download Upload Round Trip Video Encoding Complexity Web Setup Process Duplication Scalability

9 Direct Access Not All Members Have HPC Access Direct Access Not All Members Have HPC Access Direct Sharing Security Implications On HPC Direct Sharing Security Implications On HPC Gaps Download Upload Video Encoding Web Setup Process Duplication Scalability Direct Transfer Easy SharingReusableUbiquitous SeedMe.org

10 Why not use existing tools? FeaturesSeedMeYouTube & VimeoFlickr SharingPublic, Group, PrivatePublic, PrivatePublic Upload MethodCommand Line, API, WebAPI, Web Meta DataRichLimited Download SupportYesVimeo onlyNo CopyrightUnencumberedEncumbered Still ImagesYesNoYes Periodic Image UpdateYesNo Image Sequence to VideoYesNo Frame RateArbitrary29.97 fpsUnknown Video ResolutionArbitraryUpto 4KUpto HD DesirableLimitingCrippling

11 SeedMe Architecture Ingestion (Atomic or Recurring) Instant Processing (Text, Images) Notification Responsive Content Access Batch Processing (Videos)

12 SeedMe Hardware Ingest Temp Storage (10TB) Encode Cluster (16 nodes) : Apple Compressor + ffmpeg Web Cluster (2-8 nodes) : Drupal Mirrored Storage (50 TB) : ZFS Desktops Phones Tablets 20 gbe 20 gbe 20 gbe Ingestion (Atomic or Recurring) Instant Processing (Text, Images) Notification Responsive Content Access Batch Processing (Videos)

13 Compressor Encoding Benchmark

14 SeedMe Collection Ticker*PlotsImage Sequences*Videos * Recurring: periodic update Meta Data Access: Public, Group, Private Collaborators

15 Create New Collection Set Access Control (Public, Group, Private) Specify collaborators* and meta data Upload/Update content http/https, ftp, Globus Online Interactively on Web, Command Line or REST Services View Collection on Web SeedMe: Interaction Process

16 SeedMe: Interaction Modality Command Line Interface (using curl) Web Interface (WYSIWYG) RESTful API (integrate your tool chain) – JSON

17 Content Videos Permissions Plot Files Doc Files Authentication Text Ticker Image Sequences Meta Data { " ": "api_key": "XXXXXXXXXXX", ”view_permission": "public", ”viewer_ s": "title": "simulation XYZ", "description": "input conditions", "credits": "Visservices group, SDSC", "key_values": { “key1”: "v1”, "key2": "v2”}, "expire_date": " ", "tags": [ "x", "y", "z"], "ticker": ["time step 100", "time step 200"], "files":{ ”docs" : ["a.pdf", ”b.txt"], "plots" : ["a.png", "b.jpg", "c.tiff"], "image_sequences" : { "ash": ["ash1.jpg", "ash2.jpg", "ash3.jpg"], "air": ["air1.jpg", "air2.jpg", "air3.jpg"] }, "movies": ["m1.mov", "m2.mov »] }

18 Application Scenarios Monitor progress with text tickers and images Encode and stream videos from image sequences Share & collect visualization results Use the collections for education and outreach Content accessible on most devices

19 Work in progress How it Works Examples #1 & #2#1 #2 Periodic update at simulation run time from a Magneto Hydrodynamics Simulation Courtesy of Mahidhar Tatineni and Homa Karimabadi, UCSD

20 Future Work Production SeedMe Service Example Scripts & Documentation Globus integration with SeedMe

21 Acknowledgements National Science Foundation This material is based upon work supported by the National Science Foundation under Grant No. OCI Andrew Ferbert, Michael Dwyer, Kristen Levy & Doug Weimer, SDSC : Hardware/software setup Mahidhar Tatineni, SDSC & Homa Karimabadi, UCSD: in-situ tests Apple Inc. : Provided test hardware/software on loan during project conceptualization phase

22 Sign up Invitation For notification and early access (we won’t spam) Contact: sdsc.edu

23

24 Persistence? Results Move to own resource Move to XSEDE resource Move to YouTube/Flickr, Vimeo Move to Dropbox/Cloud FreemiumPURGE


Download ppt "Your Results From Disk to Device (Preview) Amit Chourasia Mona Wong-Barnum Michael L. Norman San Diego Supercomputer Center, UCSD Presented at: XSEDE 13,"

Similar presentations


Ads by Google