Digital asset management is hard. Many people have solved many parts of the problem, but for a reasonably complex use-case, many of the existing solutions just aren’t there yet, especially within a vendor-driven world for a niche market within a niche market, which is concerned with all levels and life-cycles of an asset (from production, to reuse, to archiving and back again), which is almost certainly not a profitable market given public broadcasting budgets. I believe this is an ideal area for the development of open source solutions based on some existing works of open source software.
The “easy” part in the DAM ecosystem, I would argue, is archiving the material and ensuring its long-term preservation (and accessibility!). I’ve done a couple projects and prototypes now based on the Fedora Commons repository architecture, and it seems to be a promising platform for this kind of development. Objects and datastreams are stored on the file-system, which IT staff are traditional prepared to manage (vs some unique database structure almost certainly obfuscated in layers of (de-)normalization). Fedora will happily manage security policies, object relationships, data transformation services, and (shortly) more advanced file system interactions, which exposing a (relatively) consistent HTTP interface.
Discovery interfaces are probably the next easiest piece, having been examined and developed out of the information sciences communities. Using a combination like Solr and Blacklight (deployed successfully for WGBH’s Open Vault website), one can rapidly create interfaces to the underlying content that satisfy the many use cases. With Solr, you get a bunch of discovery mechanisms and options, including relevancy, term highlighting, faceting, etc.
From here, we start getting into the hard parts. Ingest and metadata editing is difficult to solve well in a content- and use-case- agnostic way, which is the approach most Systems seem to take. While the need for a generic asset management view is important (and solved!), if the collection of services fail to meet the needs of the users, encouraging adoption (nicely) is problematic. By using infrastructure elements with open and well-documented APIs, developers can extend and customize the user experiences to match the underlying data and processes. This is an area for which the adoption and support for open source projects can encourage sustainable development of these interfaces.
It seems like, after clearing these obstacles, many systems fail to account for the use and re-use of these objects within the media communities. Few systems account for batch encoding video and audio for web distribution, one-click publishing systems to blogs, social networking sites, or video portals, integration into broadcasting chains, etc — for very good reasons, there simply isn’t the incentive when faced with large upfront development costs for unique development. Given an open source platform, however, that supports (and encourages) sharable development of solutions, maybe we could start finding answers to these persistent problems (without re-inventing the wheel!).
I believe most of the core infrastructure pieces are there:
- Fedora, as I mentioned, which provides preservation and management services;
- Solr, which provides a discovery framework (and associated metadata extraction utilities like Tika);
- Blacklight, which provides discovery and access services;
- ESB or other workflow solutions like Camel, Ruote, or otherwise;
- Generic metadata editing options, like XForms, Django, etc;
- Open standards that allow for publishing and reuse (Atom, MediaRSS, RDF, ???);
- FFMPEG, which offers encoding and transcode services.
It isn’t an extensive development problem, these are well-established communities in their fields, it’s a simple matter of getting initial momentum in tying the complex pieces together and creating interesting and useful services on top.
So, why aren’t we doing this? Money, time, lack of a collaborative/communicative culture, and apathy (and acceptance) of second-rate, buggy commercial solutions that fail to address all aspects of a media objects life-cycle as it goes from the rapid iterations in production to many different distribution channels back to relative obscurity in an archival context (until a new production pulls it out again). Without full support, no step in the process can realize the potential of the content and have the incentive to put in the hard work to ingest and describe the asset.
Chris, thank you for identifying the issues and articulating them so well. I especially appreciate your point about the inadequacy of vendor-driven systems, and I agree that open source and standards-based solutions could be within reach. It’s frustrating to see top management continuing to spend big money on proprietary systems that fail to meet the range of our real DAM and archival requirements, while dismissing open source approaches as hippy culture or something. While it is understandable that top managers may not know a great deal about the technologies and practices involved, I think they should have more confidence in their own staff and the knowledge we have throughout the public media and moving image archival community. They should definitely listen to Chris Beer.
The lack of a collaborative culture is something we’ve been trying to address, and we need to bear down on it. I think we have a great opportunity to get serious stuff done over the next year or two. Many pieces seem to be moving into place, so let’s see if we can fit them together.
Keep up the great work,
Jack
Thanks Jack — I’m hoping one of the results of this series is to demonstrate that digital asset management is more than just storing files, but an active process of providing services and value on top of the content (which is where vendor-solutions seem to fail, and for which an open source approach could demonstrate real value).
I’m very interested to see how public media addresses the culture problem, and I can only hope it is addressed beyond technology and beyond additional work so it can become ingrained in what public media actually is. We’ll see, I guess.