1. OpenStorage API part II
OpenStorage API Training
OpenStorage API Training 1

2. Image Creation and Retrieval
stsp_create_image() stsp_write_image() stsp_close_image() stsp_open_image() stsp_flush()
OpenStorage API Training

3. Image Creation and Retrieval
How a new image is initialized
stsp_create_image(). Pass in sts_lsu_t (server handle + LSU name), sts_image_def_t described previously, and pendingflag. Pendingflag is a hint (not definitive) – indicates whether this process intends to create another image after this one on same LSU.
In the initialized image’s sts_image_info_t, set STS_II_IMAGE_PENDING, STS_II_BUSY_WRITE. Ignore STS_II_FILES_PENDING and STS_II_FILES_CREATED.
OpenStorage API Training

4. Image Creation and Retrieval
How image data is written
stsp_write_image(). Passed in: image handle, stat structure, buffer ointer, length, offset in image. Return bytes successfully written hrough bytewritten. See Section VI, Image Content Description, for discussion of stat structure.
Argument len is n * STS_BLOCK_SIZE.
stsp_write_image() need not be synchronous, despite appearance.
Optimize data transmission to storage server! If data is buffered, and an error subsequently occurs, return error from stsp_write_image() or stsp_close_image(), whichever occurs first.
stsp_flush() (optional) – flush any buffered data to storage server
As image is written, update size in sts_image_info_t.imo_size. Update statistics in sts_lsu_info_t (lsu_used, lsu_used_phys, possibly lsu_capacity).
OpenStorage API Training

5. Image Creation and Retrieval
Normal completion, complete image
stsp_close_image() is called with flag argument STS_CLOSEF_IH_COMPLETE.
Clear STS_II_IMAGE_PENDING, set STS_II_IMAGE_CREATED. Clear STS_II_BUSY_WRITE.
Image cannot be reopened for writing.
OpenStorage API Training

6. Image Creation and Retrieval
Normal completion, incomplete image
Occurs during checkpoint/restart.
stsp_close_image() is called with flag either clear (STS_CLOSEF_IH_INCOMPLETE).
Clear STS_II_BUSY_WRITE.
Image can be reopened for writing.
OpenStorage API Training

7. Image Creation and Retrieval
Abnormal completion
Connection dropped before stsp_close_image() is called.
Clear STS_II_BUSY_WRITE. If image is broken (cannot be written to any more) clear STS_II_IMAGE_PENDING.
OpenStorage API Training

8. Image Deletion – stsp_delete_image()
How an image is deleted
At present, image deletion can only be initiated by the DPA.
stsp_delete_image(). Pass in sts_lsu_t (server handle + LSU name), sts_image_def_t containing image name, asyncflag. asyncflag means deletion can be asynchronous (no return code from actual deletion).
OpenStorage API Training

9. Image Deletion – stsp_delete_image()
Space reclamation associated with image deletion
Defined by LSU properties described previously (STS_LSUF_STORAGE_*).
OpenStorage API Training

10. Image Retrieval – stsp_read_image()
Steps in image retrieval
stsp_open_image(). Image must be complete (STS_II_IMAGE_COMPLETE). Pass sts_lsu_t (server handle + LSU name), sts_image_def_t, mode STS_O_READ. Return image handle.
Iterate stsp_read_image(). Pass image handle, buffer pointer, offset, length. Return bytes successfully read through bytesread. Argument len is n * STS_BLOCK_SIZE
stsp_close_image().
OpenStorage API Training

11. Image Duplication (Optimized)
stsp_open_target_server() stsp_copy_extent() stsp_copy_image()
OpenStorage API Training

12. Image Duplication (Optimized)
Concept of optimized duplication
Image copy is implemented by storage server, instead of application reading source image and writing to target image.
Image data flows directly from source storage server to target storage server.
OpenStorage API Training

13. Image Duplication (Optimized)
How optimized image copy differs from normal image creation
Image creation: stsp_open_server(), stsp_create_image(), iterate stsp_write_image(), stsp_close_image(), stsp_close_server().
Optimized copy: either
a. stsp_open_target_server(), stsp_create_image(), stsp_open_server(), stsp_open_image(), iterate stsp_copy_extent(), stsp_close_image(source), stsp_close_image(target), stsp_close_server(source), stsp_close_server(target). This copies image extent by extent. Or
b. stsp_open_target_server(), stsp_open_server(), stsp_copy_image(), stsp_close_server(source), stsp_close_server(target). This copies entire image in one operation.
c. Whole-image copy can also use asynchronous interfaces stsp_async_copy_image() and stsp_named_async_copy_image() (see Part III, Events and Asynchronous Operations).
OpenStorage API Training

14. Image Duplication (Optimized)
stsp_open_target_server() similar to stsp_open_server(), but contains server handle for the source storage server. The server parameters refer to the target storage server. This allows the source storage server to open a server handle to the target storage server. The returned server handle is only valid for stsp_create_image(), stsp_open_image(), stsp_copy_image(), stsp_async_copy_image(), stsp_named_async_copy_image(), and stsp_close_server().
stsp_copy_extent(): arguments are source and target image handles, source and target offset, length. Return bytes successfully copied through bytescopied. Target image handle may refer to image on local or remote storage server. Argument len is n * STS_BLOCK_SIZE. flags argument allows to_offset and len arguments to be overridden; STS_CX_APPEND means append to target image, STS_CX_REMAINDER means that len encompasses all of the source image beyond from_offset. These are conveniences for the OST application.
If image duplication is supported, stsp_open_target_server() and either stsp_copy_extent() or whole_image copy must be implemented. Set server capability flags in sts_server_info_t.srv_cap[], and set srv_flags.STS_SRV_IMAGE_COPY.
OpenStorage API Training

15. Virtual Image Creation – stsp_include_in_image()
Concept of virtual images, and how a virtual image differs from a normal image
A virtual image consists of references to other images, which must be stored in the same storage server.
Likely implementation: the storage server creates a map for the virtual image, which contains pointers to extents in other images. Extents have reference counts so that if an image is deleted and some of its extents are referenced by other images, those extents are not freed.
Don't implement stsp_include_in_image() by copying data. Do that with stsp_copy_extent() instead.
OpenStorage API Training

16. Virtual Image Creation – stsp_include_in_image()
Creating a virtual image
Initialize with stsp_create_image() in the usual way.
Specify extents with stsp_include_in_image(). This is similar to stsp_copy_extent(), with source + target image handles and offsets, and length. Argument length is n * STS_BLOCK_SIZE.
stsp_close_image() when done.
OpenStorage API Training

17. Image MetaData
Don't confuse with image content description, described in Section 6; image content description is also known as metadata. stsp_read_image_meta() stsp_write_image_meta()
OpenStorage API Training

18. Image MetaData
Image metadata can be written during, before, or after image data.
Metadata is always r/w irrespective of image access mode.
In particular this means metadata can be updated after image is complete.
Also metadata can be written if image is open for read only.
Metadata content is defined by application.
Typically not more than a few MB.
Obvious implementation is to store metadata in a file, associated with image (no need to dedup).
OpenStorage API Training

19. Concurrent Image Update
Storage server/plugin is not responsible for protecting badly behaved applications.
For example: two processes write same image concurrently.
Or: one process deletes image while another reads it.
Storage server should execute all calls as received and return errors as necessary.
OpenStorage API Training

20. Building a Plugin and Storage Server
Header Files
Plugin only needs to explicitly include stspi.h.
stspi.h includes stsi.h.
stsi.h includes stsplat.h, which is platform-specific.
To get correct stsplat.h, put a –I in makefile for the correct platform under sts/include/platforms, e.g. sts/include/platforms/linux.
STS_VERSION (defaulted in stsi.h) determines build version.
Must define STS_EXTERNAL_VENDOR in build environment
OpenStorage API Training

21. Required Interfaces
All those discussed in Part I. All others are optional. See Plugin Writer's Guide (SDK) for authoritative list.
OpenStorage API Training

22. Option Interfaces Currently used by NetBackup
6.5.3: stsp_open_target_server(), stsp_copy_extent()
6.5.4: (subject to change) stsp_include_in_image(), stsp_named_async_copy_image(), stsp_open_evchannel(), stsp_get_event(), stsp_delete_event(), stsp_close_evchannel(), stsp_named_async_wait(), stsp_named_async_cancel(), stsp_named_async_copy_image(), stsp_label_lsu()
OpenStorage API Training

23. Major Design Decisions – Storage Server
All of these are strongly vendor-dependent.
Nature of LSUs – relationship with physical storage.
How are LSUs configured? Tool and documentation. STS-P-I does not support LSU configuration, at present.
What LSU properties will be advertised (sts_lsu_info_t).
How to represent dedup attributes of storage, if any.
OpenStorage API Training

24. Major Design Decisions – Storage Server
How images will be stored and accessed. Files, blocks, blobs, …?
How storage is reclaimed when an image is deleted.
How to store LSU information and image information.
What server properties will be advertised (sts_server_info_t).
Storage server naming convention.
Which optional services in STS-P-I to support.
Which event model to support (push/pull/stored/volatile), if any.
How to implement optimized image copy, if supported.
How to implement virtual synthetic images, if supported.
Handling abnormal conditions from plugin e.g. dropped connection.
Handling concurrent connections.
OpenStorage API Training

25. Major Design Decisions – Plugin
Also strongly vendor-dependent.
RPC mechanism to storage server.
Write buffering for stsp_write_image() – performance is critical.
Try not to store state information on plugin host. Instead store state in the storage server. Plugin may store non-critical state information that is meaningful only on that host (however, must be synchronized for multiple plugin instances on same host).
Where possible, considering allowing different versions of the same API interface to share some common source.
Multithreading considerations.
Which API version(s) to support.
For multiple versions, all in one binary, or one binary each version?
OpenStorage API Training

26. Presentation Identifier Goes Here