The basics

This is not a section I really want to write, but I don’t know of a high-level answer to “when we say repository, this is what we mean”. I spent a little time looking around for a summary, but more often than not I found more questions (or, perhaps more useful yet inappropriate for my purposes, technology-based answers rather than use-driven), so I’ve taken a stab at addressing what I believe are some key issues:

Repositories are a collection of services, with well-defined interfaces, for storing and managing data (both content and metadata) in a format-neutral, display-independent manner way. Repositories can be used as preservation repositories, as access repositories, as centralized aggregations of far-flung data, etc and operate on any scale for any audience. Furthermore, there are existing standards and agreements about what it means to be a certain type of repository (TDR, OAIS, etc). All of these repositories, however, share some common services — whether implemented as software, external processes, or manual processes.

Some essential repository services are:

• Identifier services, which may include assignment + registration
• Storage services (although the content stored may be only pointers to the “actual” content)
• Content identification, matching identifiers to content items
• Ingest workflows
• Access mechanisms

Without these services in place, a repository system would face some difficult obstacles in creating and providing value-added services. Repositories may provide multiple flavors of these services, some of which may be defined in generally accepted standards, models, and specifications.

Other basic services which operate on top of the above services are fairly common in most well-developed repository frameworks include:

• Dissemination services, to transform repository data into other forms + formats
• Authorization services

More advanced services may include:

• preservation services, including checksum (generation + verification), file format migration, support for models like LOCKSS
• relationship services, using an RDF triplestore or similar, offering SPARQL endpoints, interferencing, etc
• discovery services, using Lucene/Solr/etc, to provide relevancy, optimized user experience, drill-down faceting

These more advanced services are likely separate applications in the repository ecosystem and are generally useful utilities independent of any repository system. Repositories generally integrate with these external applications in a modular, mix-and-match manner using well-defined interfaces.

Fedora

One approach to repository services is the “repository-in-a-box” model, where you can install and configure a base set of services provided by a single application. Within this group of services, Fedora provides a very basic implementation of the core repository services (vs a full-stack application like DSpace, which provides production-ready user interfaces). Fedora bills itself as a Flexible, Extensible Digital Object Repository Architecture.

• Identifier services, through PIDGen which provides sequential identifiers per-namespace
• maps http uris to deferenceable uris to files
• REST + SOAP APIs for Ingest + Delivery
• Dissemination services using WSDL
• Authorization using XACML (and authentication using a number of plugins)
• Integrates with the Mulgara triplestore and a Lucene index (by default)

Fedora provides a many opportunities for customization and enhancements through custom development:

• the Fedora REST, SOAP, and triple-store APIs allow developers to build on top of low-level services, which may include access interfaces, administrative interfaces, or otherwise
• the Fedora application provides Java Messaging Services (JMS) events when objects within the repository are created, deleted, or modified, and developers can build applications that listen to these events and trigger actions (Shelver <http://yourmediashelf.com/blog/2010/03/01/blacklight-activefedora-and-shelver-interplay-between-searching-managing-and-indexing-in-a-repository-solution/>, fedora-workflow <http://github.com/cbeer/fedora-workflow>, etc)
• the Fedora application is build modularly, and Java developers are able to develop and use components as needed, if they conform to the Fedora interfaces

As services go beyond the basic, common applications present in institutional repositories, enhanced repository services require custom development or supplemental services outside of the repository services. For most, this includes integration with a more advanced search provider (like Solr). At some point, additional services can blur the lines between the repository services and front-end user interfaces (which have to respond to local customization to meet user needs).

Repository-independent services, or third-party services, require some wrapper to make them interoperable with the Fedora APIs, which makes integration with existing technology more difficult. Even Duraspace’s Duracloud offering is (currently) built as separate services with some possibility of storage-level integration. Preservation support services will bypass the repository APIs and provide those services against the file system instead.

Considering the services Fedora doesn’t provide or the obstacles Fedora creates in integration, many ask why they should start using Fedora anyway. The strongest response to this, I believe, is that it provides a common structure to basic repository services, while at the same time not creating major obstacles to future expansion or migration outside Fedora. Out of the box, Fedora provides a set of “training wheels” (ht Mike Giarlo <http://lackoftalent.org/michael/blog/>) for repository services development that can be removed when unnecessary, but in the meantime offers structure for the creation of new repositories and support for repository services as needed.

CDL Microservices

Another approach to repository services are “microservices” like those designed by the California Digital Library (CDL), provide standards and specifications for individual repository services, which form a structure for standardized, mix-and-match repository services that can integrate, interoperate and take advantage of existing technology independent of a repository application like Fedora. This, conceivably, allows all domain developers to take advantage of these common projects without using a specific technology. CDL provides microservices specifications for:

• identifier assignment + registration, using NOID, which can act as a CLI tool or a CGI service
• file-system structures, using the Pairtree convention
• data exchange and verification, using BagIt
• access standards, using the ARK URL format

The standards are developed inline the “UNIX philosophy”:

Write programs that do one thing and do it well. Write programs to work together. — Doug McIlroy

These basic services can be organized and crafted using the existing capabilities in web servers, file systems, etc. More advanced services can act within this structure, using individual standards when needed. While significant development and customization may be required to get a microservices architecture to a useable state, the end result is more flexible and targeted to an institutions needs.

Flexing Fedora

These two approaches are certainly not incompatible, and Fedora is quite capable of using some of these micro-services standards under the hood (replacing custom developed approaches to these basic services). By taking this approach, Fedora could act as a management application on top of generic repository data, allow both Fedora-based and microservices-based services to operate on the data, and make it easier to reach around Fedora when necessary (or, go so far as to remove it entirely).

What follows is a short summary of on-going work in this area, which mostly focus on removing the Fedora-centric definitions of /how/ or /where/ services act. The majority of these ideas build on new developments and best practices (since Fedora was initially created) in the repository community as a result increased adoption or awareness of issues. Where available, I’ve included links to projects in-the-works.

Some of this work is quite easy to do:

• integration of NOID identifier services by creating a web-services consumer for Fedora identifier assignment <http://gist.github.com/273584>
• replacing the custom, timestamp-hash file store with a Pairtree structure (the prototype is limited, however, by Fedora’s hard-coded distinction between object and datastream filestores <http://gist.github.com/280020>
• using memento http headers to provide versioning <http://www.fedora-commons.org/jira/browse/FCREPO-604>

Other projects that are more involved, and require more work than just creating new modules for Fedora:

• BagIt and SWORD ingest and dissemination options to replace the custom Atom structure <http://fedora-commons.org/confluence/display/FCSVCS/SWORD-Fedora+1.2>
• Integration of arbitrary ingest of structured data (perhaps similar to CDL’s 7train <http://seventrain.sourceforge.net/>?)
• Pluggable authn/authz, through the FESL project, JAAS should provide a pluggable authentication backend <http://www.fedora-commons.org/confluence/display/DEV/Fedora+Enhanced+Security+Layer>
• support for arbitrary RDF metadata, forget RELS-EXT/RELS-INT — force that kind of decision into a disseminator and use a seamless API to pull back RDF triples (/object/{pid}/relationships) <http://www.fedora-commons.org/confluence/display/DEV/Supporting+the+Semantic+Web+and+Linked+Data>

More advanced microservices integration is highly involved and would require a major re-work of the application:

• Two-way messaging queues (or file alteration monitors, or database update hooks) to allow Fedora to receive updates
• decreased reliance on self-generated registries, I think the situation is getting better, but I’m not sure its fully there..
• pluggable storage modules with intelligent filtering, routing, multiplexing, and rules mechanisms — the Akubra project may be doing (part of?) this <http://www.fedora-commons.org/confluence/display/AKUBRA/Akubra+Project>
• workflow support hooks, to allow integration and automation of workflow tools (possibly a result of Hydra?)
  

A federated repository ultimately needs three things:

1. A common API among the participating repositories,
2. A search index that covers all the repositories, and
3. A resolver to translate a search result back to the originating repository

Common API

For bVault, the common API is the set of web services exposed by Fedora, and the metadata translation dissemination service behind that, which allows a client to receive a particular metadata format, regardless of the underlying schema. This is an important feature, because it allows individual repositories to use whichever metadata format is most natural to their needs, while seamlessly generating interoperable metadata.

Search index

The exact methods employed to generate a spanning search index are essentially arbitrary. Solr provides some distributed/sharded search capabilities, but the index could also operate on a pub/sub model where repositories push content out to a master search index, or with a search engine like crawler using OAI-PMH endpoints for the repository. Because the search index is loosely coupled to the whole system, it ultimately is an architectural decision rather than a technical one

Distributed Resolver

Now that we have a way to discover items within a repository, the interface needs a way to extract the content from the origin. For this, we need a way to resolve a unique resource identifier (URI!) back to its source. Again, the method is somewhat arbitrary, but for this project, we elected to require unique namespaces for each repository (quite reasonable, considering the application).

To do this, I’ve slipped a namespace resolver into the client’s API call to allow the interface to act independently from the source of the content. For a simple API call, like listDatastreams, we have:

public function listDatastreams($pid, $asOfDateTime = null) {
      return Fedora_Repository::get('API-A', $pid)->listDatastreams(array('pid' => $pid,
                    'asOfDateTime' => $asOfDateTime));
}

This requests the API-A binding appropriate to the current persistant identifier (pid):

/**
  * Retrieves a Fedora Repository that can provide the $type endpoint for the PID/prefix $prefix
  *
  * @param string $type
  * @param string $prefix
  * @return Fedora_Repository
  */
static public function get($type, $prefix = '') {
     global $objManager;

     $arrRepository = $objManager->resolve($prefix);
     $objClient = new stdClass;

     if(count($arrRepository) == 1) {
           $objClient = $arrRepository[0]->getSoapClient($type);
     } else {
           $arrKey = array_rand($arrRepository, count($arrRepository));

           foreach($arrKey as $key) {
               $objClient = $arrRepository[$key]->getSoapClient($type);
               if($objClient !== false) {
                     break;
               }
           }
      }

      if($objClient instanceof SoapClient) {
            return $objClient;
      } else {
            return false;
      }
}

Creating a distributed repository doesn’t cost much now, and if you design it right, you can benefit from the potential for redundancy and mirroring immediately, even before there is a federated network to tap into.

The full source is available from the bVault Fedora PHP library.