ARCHIVED - About Us

Audits and Evaluations

Evaluation Framework for Modernization

Previous | Table of Contents

4.0 Monitoring and Evaluation Plan

4.1 Performance measurement plan

Strategy and Key issues

The table below identifies key outputs and outcomes along with their associated indicators, sources and measurement frequency. The performance indicators will enable management to assess progress and performance in achieving their goals. The indicators presented in this section of the report were developed through discussions with key informants. They are linked with specific outputs, immediate outcomes and intermediate outcomes of the logic model.

Table 3: Performance Indicators for Modernization Initiative
Output/ Outcome	Indicator	Data Source & Method	Responsible	Frequency
1. Plans and strategies	Number of policy suites developed (Acquisition, Collections Management, and Resource Discovery)	- Administrative data	Sub-Committee Leads	Quarterly
2. Work processes redefined	Extent to which work processes at LAC have incorporated the principles of Modernization	- Document review - Key informant interviews	DG Steering Committee	Semi-Annually
3. Horizontal collaboration	Extent to which policies and processes are in place to support internal and external collaboration	- Document review - Key informant interviews	DG Steering Committee	Semi-Annually
4. Employee engagement	Extent to which employees actively support Modernization activities	- Employee survey	DG Steering Committee	Semi-Annually
5. Policy-driven organization	Degree of coherence between policy decisions taken and their implementation	- Document review - Key informant interviews	DG Steering Committee	Annually
6. Professional bureaucracy	Degree to which on-going monitoring and lessons learned are incorporated into subsequent plans and actions	- Document review - Key informant interviews	DG Steering Committee	Annually

The Head of Evaluation will present an update to the Evaluation Committee on the progress of the implementation of the framework every six months once the framework is approved.

Reporting Strategy

Progress on the Modernization Initiative will be reported through the following mechanisms/documents:

• Regular updates to Management Board by the DG Steering Committee on Modernization
• A Departmental Performance Report (DPR) to be presented by LAC each year to the Treasury Board Secretariat (TBS); and
• Program Evaluation reports and updates to be presented to the Library and Archives Evaluation Committee for approval and subsequent release to the public.

In addition, on-going reporting of Modernization activities will be carried out using various mechanisms such as the LAC website, the LAC intranet and messages from the Librarian and Archivist of Canada.

4.2 Evaluation Plan

The five-year evaluation cycle for the Modernization Initiative spans the fiscal years 2010-2011 to 2015-2016. A formative evaluation will be conducted in 2012-2013 to assess the performance measurement strategy, identify any issues that may arise, and assess data availability for the summative evaluation to follow in 2015-2016.

The summative evaluation will be conducted to assess overall progress in reaching the goals and objectives for the initiative. This evaluation will fully incorporate findings from ongoing performance measurement and will follow the Evaluation Standards presented in the Treasury Board Evaluation Standards in the Government of Canada (TBS, 2004). There are three primary areas that will be considered, namely relevance, success and cost-effectiveness.

Key Evaluation Issues and Questions

Relevance - does the initiative continue to be consistent with departmental and government-wide priorities and does it realistically address an actual need?

Success - is the initiative effective in meeting its objectives, within budget and without unwanted outcomes?

Cost-effectiveness - Are the most appropriate and efficient means being used to achieve objectives, relative to alternative design and delivery approaches?

Evaluation Timeframes

The timeframes set out below are normal for most program evaluation cycles. The formative evaluation of the Modernization Initiative would be conducted in the fiscal year 2012-2013; the summative evaluation would occur in the fiscal year 2015-2016. Table 4 below summarizes the time frame applicable to evaluation activities.

Table 4: Evaluation activities timeframes
2010-2011	2011-2012	2012-2013	2014-2015	2014-2015	2015-2016
Develop Framework
		Formative evaluation
					Summative evaluation

Formative and Summative Evaluations

Table 5 below contains the evaluation framework that will guide the formative and summative evaluations for the Modernization Initiative. The framework addresses the three primary areas of relevance, success and cost-effectiveness, and contains a total of seven questions that will help to guide and focus the evaluation. For each evaluation question, the framework describes the indicators and data sources that will be used to address the question. All of the issues/questions will be addressed in both the formative and summative evaluations, however, the focus of the formative evaluation will be to assess if progress is being made in each of the respective areas.

Table 5: Evaluation Framework
Issues/Questions	Indicators	Data Analysis Methods	Evaluation Activity
			Formative	Summative
Relevance
1. To what extent does the Modernization Initiative continue to align with the priorities of Library and Archives Canada and the Government of Canada?	- Degree of alignment between Modernization activities and priorities of LAC and the Government of Canada	- Document review - Key informant interviews with LAC staff and key stakeholders	X	X
2. Do activities being undertaken under the Modernization Initiative continue to be relevant to the public? Do these activities address the evolving needs of stakeholders and clients?	- Evidence of public and stakeholder understanding of and acceptance of the need for Modernization	- Document review - Key informant interviews with LAC staff and key stakeholders and clients	X	X
Success
3. Is the Modernization Initiative being implemented as expected?	- Extent to which Modernization activities are implemented as planned	- Document review - Key informant interviews with LAC staff and key stakeholders	X	X
4. To what degree have the expected immediate and intermediate outcomes of the Modernization Initiative been achieved?	- Extent to which employees actively support Modernization activities - Extent to which work processes at LAC have incorporated the principles of Modernization - Degree of coherence between policy decisions taken and their implementation - Degree to which on-going monitoring and lessons learned are incorporated into subsequent plans and actions	- Employee survey - Document review - Key informant interviews with LAC staff and key stakeholders	X	X
5. Is the performance measurement strategy being carried out as planned?	- Degree of alignment of performance measurement plan with actual performance activities undertaken	- Document review - Key informant interviews with LAC staff	X	X
6. How is the performance data that is collected being used?	- Evidence of use of performance data in decision making	- Document review - Key informant interviews with LAC staff	X	X
Cost-effectiveness
7. Have more effective / efficient ways of achieving the expected outcomes of the Modernization Initiative become apparent?	- Evidence of more effective / efficient ways of achieving the expected outcomes are apparent to staff and key stakeholders	- Document review - Key informant interviews with LAC staff	X	X

Evaluation costs

The estimated costs for the formative and summative evaluations are provided in the table below. These are normal costs for contracted services.

Table 6: Evaluation costs
Study	Fiscal Year	Total O&M costs
Formative Evaluation	2012-2013	$25,000
Summative Evaluation	2015-2016	$75,000
Total		$100,000

Previous | Table of Contents