This coordination between RENCI, DICE, ITS, and the libraries is helping solidify data management support to the Carolina community. Along with support for distributed data management and sharing, we work to develop innovative mechanisms and options for storing and managing research data as well as curation techniques for longer term storage and retrieval of that data.

As the interconnected nature of seemingly separate disciplines becomes more evident, and as discoveries are increasingly pulled from sources of data that were apparently unrelated, the demand for seamless access to disparate data collections is growing. Data sharing across platforms and across administrative and disciplinary boundaries is supported by the Integrated Rule-Oriented Data System (iRODS) data grid technology (https://www.irods.org/index.php/Introduction_to_iRODS). RENCI partners with the DICE Center (http://dice.unc.edu/) at UNC-CH, developers of iRODS, to offer iRODS data grid support for data management, sharing, publishing, preservation.

An iRODS data grid allows a global file system view of distributed data, with access controls that enable the owner of the data to permit access to other users (or not). Effectively, iRODS provides a virtual global user space: IRODS users can ingest, access, share, and publish their distributed data on a variety of computer systems, without ever having a computer account on those systems.

The RENCI data grid is up and helping North Carolina researchers assemble shared data collections through use of data grid technology.

IRODS rules are based on a growing set of microservices and allow data administrators to implement data management policy, site-wise, collection-wise, file-wise, user-wise, etc. The flexibility of policy implementation allows iRODS to be tailored to very specific data management needs, controlled by the data owners, even when the data resides physically at a remote location.

RENCI at UNC-CH manages the UNC mirror node of the National Archives and Record Administration (NARA) Transcontinental Persistent Archive Prototype (TPAP), a NARA data grid based on iRODS technology.

The iRODS system is middleware for sharing data distributed across heterogeneous resources. The permanent database that is part of the system, the iCAT metadata catalogue, contains metadata that is used for the mappings between underlying physical infrastructure and the global, logical digital objects in the iRODS collections.

An iRODS data grid is implemented through the installation of software servers at every location where data is stored. Each server maps from the protocol of its own specific storage system to the standard operations supported by the data grid (some 80 operations for the remote manipulation of data and metadata). One server is iCAT-enabled and linked to the metadata catalogue; the other servers communicate with this server to access global metadata.

Peculiarities of storage systems and access methods, locations of data, user authentication and access across disparate systems are all transparent to the user. A user can access files from an online file system, near-line tapes, relational databases, sensor data streams and the web without worrying about location or system access protocols and without establishing separate accounts or passwords/certificates to the underlying computer systems.

These functionalities, which “virtualize” data storage and user access, are implemented with the mappings and profile metadata contained in the iCAT. The iCAT can be implemented in any of several database technologies (Oracle, Postgresql, mySQL) and manages all state information as well as user-defined descriptive metadata.

Practically speaking, clients communicate with any iRODS server in a given data grid, then that server contacts the iCAT and transacts any data creation, transfer, or modification. Functionalities in iRODS are implemented through rules invoked by a rule engine; the rules are composed of microservices that are provided in the iRODS system.

iRODS Data Grid Architecture

The modular nature of the iRODS rules allows fine customization of data curation, and data management policy can be implemented at several levels: that of a site, a collection, a user group, a user, or even a file. Proprietors of data collections can implement their own policy by composing microservices into rules and can verify that their policy is being enforced, even when the data is stored remotely (and/or is distributed).

Many iRODS microservices are currently under development; users can compose rules out of existing microservices or can write their own specialized microservices to implement specific policy.