​Summary – Evaluation of the Co-Lab Initiative from 2016-2017 to 2018-2019

Corporate Planning and Accountability
Library and Archives Canada (LAC)

May 23, 2019

On this page
Related

Brief Overview of the Initiative

The Co-Lab initiative was launched in April 2018. It presents an innovative approach to public service programming as it experiments with a new crowdsourcing web application for transcribing, translating, tagging and describing LAC’s holdings. The purpose of the initiative is to increase the digital content of LAC’s collections, their accessibility and discoverability, and to engage Canadians with the collection.

The initiative is part of LAC’s Public Services Program and is managed by the Exhibitions and Online Content Division of Public Services Branch. The Innovation and Chief Information Officer Branch was involved in the development of the Co-Lab web application and provides on-going technical support.

Questions and Evaluation Results

  • Question #1: Is the design of the Co-Lab initiative sound?

    Result: Achieved. Generally, the Co-Lab design is sound. Co-Lab management has clearly defined the purpose of the initiative and has established objectives that are consistent with LAC’s mission and priorities. The value that the initiative creates for LAC, the Co-Lab users and the Canadian public is clearly articulated.

  • Question #2: Is the design of the Co-Lab initiative consistent with best practices?

    Result: Partially. The design of the initiative is consistent with crowdsourcing best practices in cultural institutions. However, apart from the IT business requirements for the development of the web-application, little documentation demonstrates the rationale for the initiative and for the selection of the approach.

    In addition, certain aspects of the Co-Lab users’ role need further clarification. Finally, there is an indication that users need more support to further community development, enable self-management and ensure good community relations.

  • Question #3: How successful has the initiative been so far?

    Result: Partially. The implementation of the initiative has gone well. The initiative has generated interest from the public and a user community has began to form. The experience of users and Co-Lab staff has been positive. Co-Lab Challenges have been well received, participation has been increasing steadily since the launch of the initiative and the rate of task completion has been relatively stable. Nevertheless, Co-Lab does not have a robust reporting system in place yet and performance measures are still under development. The evaluation results provide Co-Lab management with an opportunity to make some adjustments to the initiative in order to improve it, to be better positioned to demonstrate the performance of the initiative and the attainment of results over time.

Recommendations

Recommendations, management response, and action plan
RecommendationsManagement Response and Action Plan
  1. As the initiative evolves, document the strategic thinking around Co-Lab and its future directions.
  1. Create a Co-Lab strategy, which includes the vision and objectives for the Initiative and expected results (short and medium term results).
  2. Develop an implementation plan with key milestones and timelines including requested resources ($ and FTEs).
  3. Document decision-making through Minutes of meetings/ Record of decisions.
  1. Define and document what success for the initiative is and how it will be demonstrated.

In Co-Lab strategy:

  1. Define what constitutes success for the initiative.
  2. Identify methods and data needed to demonstrate level of success.
  1.  
    1. Ensure that the reporting system currently being developed identifies meaningful performance measures that include output and outcome indicators.
    2. Ensure that consistent performance data is gathered as the initiative evolves to ensure that progress towards expected results can be demonstrated over time.
    3. Document the rationale for any major changes to performance measurement.
    1. Identify performance indicators that measure both outputs and outcomes.
    2. Monitor performance indicators on a quarterly basis.
    3. Document changes made to the performance metrics on an as needed basis.

Considerations for Improvements

Consideration for action 1: Define and document what constitutes an “inappropriate contribution”. Implement clear guidelines for user behavior and for treatment of violations.

Consideration for action 2: More support should be made available to the Co-Lab user community to ensure that it has what it needs to collaborate and manage itself effectively.

Evaluation Methods

The following qualitative and quantitative methods and data triangulation were used in the evaluation:

  • Administrative and financial documentation of the initiative, performance statistics and other relevant internal and external documentation were consulted;
  • Eight key informant interviews were carried out with managers and staff involved in the conceptualization, management and delivery of the initiative; and
  • A survey of 124 Co-Lab registered users was administered.

The focus of the evaluation was on assessing the design of the initiative, the effectiveness of its implementation to date and the resource utilization. This methodology is consistent with the Treasury Board Policy on Results (2016).

Limitations

  1. Limited documentation and background on what defines success for crowdsourcing initiatives or projects;
  2. No known published evaluation studies of crowdsourcing experimentation and projects in cultural institutions to guide the evaluation; and
  3. Limited information on the budget and costing of cultural sector crowdsourcing initiatives, so no comparative analysis could be performed as to the appropriateness of resources dedicated to Co-Lab.
​