ePubs

The open archive for STFC research publications

Full Record Details

Persistent URL http://purl.org/net/epubs/work/53578
Record Status Checked
Record Id 53578
Title Towards a scalable, open-standards service for brokering cross-protocol data transfers across multiple sources and sinks
Contributors
Abstract Data Transfer Service (DTS) is an open-source project that is developing a documentcentric message model for describing a bulk data transfer activity, with an accompanying set of loosely coupled and platform-independent components for brokering the transfer of data between a wide range of (potentially incompatible) storage resources as scheduled, fault-tolerant batch jobs. The architecture scales from small embedded eployments on a single computer to large distributed deployments through an expandable workernode pool controlled through message-orientated middleware. Data access and transfer efficiency are maximized through the strategic placement of worker nodes at or between particular data sources/sinks. The design is inherently asynchronous, and, when third-party transfer is not available, it side-steps the bandwidth, concurrency and scalability limitations associated with buffering bytes directly through intermediary client applications. It aims to address geographical/topological deployment concerns by allowing service hosting to be either centralized (as part of a shared service) or confined to a single institution or domain. Established design patterns and open-source components are coupled with a proposal for a document-centric and open-standards-based messaging protocol. As part of the development of the message protocol, a bulk data copy activity document is proposed for the first time.
Organisation ESC , STFC , ESC-SCT
Keywords data transfer , open standards , asynchronous , third party , scalable , cross-protocol
Funding Information
Related Research Object(s):
Licence Information:
Language English (EN)
Type Details URI(s) Local file(s) Year
Journal Article Philos T Roy Soc A 368, no. 2 (2010): 1926, 4115-4131. doi:10.1098/rsta.2010.0148 2010