ePubs
The open archive for STFC research publications
Home
About ePubs
Content Policies
News
Help
Privacy/Cookies
Suggest an Enhancement
Contact ePubs
Full Record Details
Persistent URL
http://purl.org/net/epubs/work/53578
Record Status
Checked
Record Id
53578
Title
Towards a scalable, open-standards service for brokering cross-protocol data transfers across multiple sources and sinks
Contributors
D Meredith (STFC Daresbury Lab.) (Pr.Au.)
,
S Crouch (Southampton U.)
,
G Galang (Victorian eResearch Strategic Initiative)
,
M Jiang (STFC Daresbury Lab.)
,
N Nguyen (Sydney U.)
,
P Turner (Sydney U.)
Abstract
Data Transfer Service (DTS) is an open-source project that is developing a documentcentric message model for describing a bulk data transfer activity, with an accompanying set of loosely coupled and platform-independent components for brokering the transfer of data between a wide range of (potentially incompatible) storage resources as scheduled, fault-tolerant batch jobs. The architecture scales from small embedded eployments on a single computer to large distributed deployments through an expandable workernode pool controlled through message-orientated middleware. Data access and transfer efficiency are maximized through the strategic placement of worker nodes at or between particular data sources/sinks. The design is inherently asynchronous, and, when third-party transfer is not available, it side-steps the bandwidth, concurrency and scalability limitations associated with buffering bytes directly through intermediary client applications. It aims to address geographical/topological deployment concerns by allowing service hosting to be either centralized (as part of a shared service) or confined to a single institution or domain. Established design patterns and open-source components are coupled with a proposal for a document-centric and open-standards-based messaging protocol. As part of the development of the message protocol, a bulk data copy activity document is proposed for the first time.
Organisation
ESC
,
STFC
,
ESC-SCT
Keywords
data transfer
,
open standards
,
asynchronous
,
third party
,
scalable
,
cross-protocol
Funding Information
Related Research Object(s):
Licence Information:
Language
English (EN)
Type
Details
URI(s)
Local file(s)
Year
Journal Article
Philos T Roy Soc A
368, no. 2 (2010): 1926, 4115-4131.
doi:10.1098/rsta.2010.0148
2010
Showing record 1 of 1
Recent Additions
Browse Organisations
Browse Journals/Series
Login to add & manage publications and access information for OA publishing
Username:
Password:
Useful Links
Chadwick & RAL Libraries
SHERPA FACT
SHERPA RoMEO
SHERPA JULIET
Journal Checker Tool
Google Scholar