Thematic report on learning and development

447. The Learning and Development Strategy details the vision, aims and objectives of the Learning and Development function showing it aims to work towards results for staff (people) and Crown Office and Procurator Fiscal Service (organisation). Examples include:

• 'we intend to continue to develop and to ensure that our contribution …… continue to meet business needs'
• '…. ensuring that our people are equipped to provide a high quality service to communities… '
• 'Learning and development is central to performance management …. we aim to improve our organisational effectiveness in all areas of our business ….'
• 'We will provide a range of options which will allow managers at all levels to become more effective in their roles'
• 'We acknowledge the critical role of leadership in the organisation and will develop a menu of option to provide the support our senior managers need to continuously improve their skills'

448. In order to measure whether these are being met and whether the learning and development function is adding value to the organisation there has to be a form of evaluation built in to the learning and development process.

Evaluation Strategy

449. Learning and Development Division issued their latest 'Evaluation Strategy' in November 2009. It was developed to 'reinforce commitment to evaluation' and it stated that the Division 'recognises that evaluation is a key element which underpins the learning and development process'. The strategy can be found on the Learning and Development home page on the intranet although not all Learning and Development staff were fully familiar with it.

450. The strategy describes four levels of evaluation according to the well known Kirkpatrick Model devised 'by Donald Kirkpatrick in 1959 and revised in 2009' ³⁰:

• Level 1 - Reactions
• Level 2 - Learning
• Level 3 - Behaviour
• Level 4 - Results

451. For each level there is guidance in the strategy as to possible methods of conducting evaluation. In addition there is guidance available on evaluation techniques in the Standards Manual ³¹ which confusingly refers to a 2008 evaluation strategy with 5 levels.

452. We found that the current practices did not fulfil the aims of the evaluation strategy ³². A 'reaction' evaluation sheet (level 1) might give an indication of the quality of presentations and arrangements on the day but it does not provide any meaningful information about whether any differences in ability or attitude were found after the training. Although delegates might demonstrate an understanding of the training at the course (level 2) it is not until the training is put into practice at the workplace that it can be said to be effective. For such information to be obtained a variety of sources need to be interrogated. For example, assessment of delegates' knowledge/understanding before the course followed up by line management assessment on whether changes to performance/attitude have resulted. It is accepted that such an assessment is resource intensive, taking the time of both manager and delegate and requires drive and determination to ensure that this follow up is carried out and analysed. That is not to say that the effort required is not worthwhile.

453. It is also possible to assess effectiveness of learning and development through measuring various aspects of the business environment such as:

• reduction in errors
• increase in productivity
• achievement of targets
• staff motivation and attitudes, etc

454. To do this the benefits can be measured as long as an association can be made with a specific course and that the measures are realistic and achievable. This would help satisfy level 4 of the Kirkpatrick model but such evaluation may require more corporate input.

Evidence of Training Outcomes

Investors in People ( IIP)

455. The Crown Office and Procurator Fiscal Service was re-accredited with the Investors in People Award at silver level in 2009. Part of the assessment related to learning and development. The report concluded that the introduction of the College has ensured that there is a culture of continuous learning within the Crown Office and the Procurator Fiscal Service.

456. Senior management indicated that they wished to use the IIP standard as a 'catalyst and framework, which will help the Crown Office and Procurator Fiscal Service improve performance through training and development of all Crown Office and Procurator Fiscal Service people to meet the needs of the business ³³'.

Civil Service People Survey

457. The results of the Civil Service People Survey 2009 provide information on how the Learning and Development function is viewed. These were published in February 2010. There was an overall 74% response rate from staff in the Crown office and Procurator Fiscal Service. The employee engagement index was 63% against a median of 59% civil service wide which was within the top quarter of civil service bodies participating.

458. The relevant questions and responses in relation to learning and development are as follows:

• 74% gave a positive response to being able to access the right learning and development opportunities when they need to compared to 63% civil service wide.
• 56% gave a positive response to agreeing that learning and development activities they have completed in the last 12 months have helped to improve their performance compared to 51% civil service wide.
• 51% gave a positive response to feeling that there are opportunities to develop their career in the Crown Office and Procurator Fiscal Service compared to 38% civil service wide.
• 46% gave a positive response to agreeing that learning and development activities they have completed while working for the Crown Office and Procurator Fiscal Service helped them to develop their career compared to 44% civil service wide.

459. The above results are extremely positive compared to other organisations that participated and in fact in relation to learning and development the Crown Office and Procurator Fiscal Service was positioned in 20th place from 96 organisations. The results also highlight that there is room for improvement in ensuring alignment of training currently offered to business needs and career opportunities.

Inspectorate Survey

460. In our questionnaire we asked whether the most recent course staff had attended achieved its specific aims. 45 (or 76%) replied that the aims were met, three (or 6%) replied that they were not met and the remainder did not specify. Examples of comments in addition to the yes/no answers are as follows:

• "I had to use knowledge gained immediately after the course." (Q32, 3 - Course - Disclosure)
• "It was one of the best training courses I have had, benefiting from a small number of diverse and experienced participants and skilled and enthusiastic trainers." (Q32, 7 - Course - Advocacy 3)
• "Those attending had a wide range of experience in dealing with the subject matter; no way the course could have satisfied everyone's needs." (Q32, 17 - Course - Deaths 2)
• "As there is a lot of change in the department it was good to have the skills to cope with this, it assisted me in the thought process of managing change." (Q32, 24 - Course - Managing Change)
• "The course was entirely successful in providing me with information about domestic abuse and I subsequently felt more confident in dealing with these types of cases." (Q32, 39 - Course - Domestic Abuse)
• "The modules that we have completed so far have been very beneficial to me as a new line manager." (Q32, 43 - Course - Certificate in First Line Management)
• "Highlighted the key legislation and our responsibilities as members of the civil service in promoting diversity." (Q32, 48 - Course - Diversity)
• "I have found that all the courses I have attended have been of benefit to me in my everyday work." (Q32, 53 - Course unknown)

461. It is clear from the above results that for the majority of respondents the aims and objectives of each course are met and given the lapsed time since the course they appear to have put into practice what has been learned.

462. Given that there is little follow up evaluation to determine whether staff are more effective and efficient in their job as a result of training provided it is difficult to ascertain whether aims in the Learning and Development Strategy such as providing a 'high quality service to communities' or 'managers being more effective in their roles' is achieved as a result of specific training provided to staff.

463. Despite the lack of systems in place to carry out higher level evaluation which could provide evidence of the value of the training input in terms of learning outcomes, our own survey did show a level of satisfaction with Learning and Development activity as the following comments show.

464. We asked whether staff felt the service provided by Learning and Development Division adds value to the Crown Office and Procurator Fiscal Service. Examples of positive comments are as follows:

• "…. following the introduction of the Prosecution College I felt that training was being delivered more frequently and was focussed. With the introduction of sponsorship and accredited courses it has increased the value of Crown Office and Procurator Fiscal Service to the staff and their long term prospects." (Q33, 1)
• "Since the division has been brought to the Scottish Prosecution College I feel they have provided good accommodation and great trainers with different teaching styles." (Q33, 5)
• "I feel this in-house training is a valuable and cost efficient tool for the service and with a few improvements ie courses available to everyone interested can only improve the service as a whole." (Q33, 12)

465. Although a more negative slant from one lawyer:

• "I think that the value is quite limited for professional lawyers seeking to improve as lawyers and to develop their career within Crown Office and Procurator Fiscal Service …." (Q33, 47)

466. There was some confirmation of alignment of the training received to the business needs of the Service:

• "By attending the courses that are relevant to my job, enables me to carry out my duties more effectively and efficiently." (Q33, 20)
• "I do feel that the service provided is tailored to respond to the evolving need of the service as a whole." (Q33, 25)
• "I think the Learning and Development Division has a very difficult job to service the Service with limited space, time and resources. It is particularly challenged by the changes in legislation and requirements (eg disclosure) which can need disseminated very urgently and unexpectedly, disrupting other training ….." (Q33, 31)
• "Courses seem more focussed on the "real life" scenarios which staff have to deal with. Contributors tend to be those with practical experience who can understand the issues encountered on a day to day basis." (Q33, 33).

467. Not everyone agreed however:

• "I feel that training could be tailored more to meet the needs of the job in hand eg why send someone on training in relation to solemn matters when they work in a summary team and vice versa. Also I feel that there should be a register kept of all those who have been on specific courses so that those who haven't can tap into those resources." (Comments, 17)

468. There was general approval for the new College and training staff:

• "Any time I have attended a training course I have found it useful and helpful. The quality of training is always high and I think it's important to do this outwith your usual place of work to aid concentration." (Q33, 36)
• "The courses I have been on have all been excellently organised…." (Q33, 39)
• "Without doubt, since the opening of the Scottish Prosecution College the training has been first class (in my opinion)." (Q33, 45)

469. There is some consensus among those in the training profession that meaningful evaluation is not about a snapshot view of the training experience but rather a longer term study taking into account not only the direct input of training but also other factors that helped to reinforce the learning. Learning and Development Division are not alone in finding this difficult to achieve and indeed this may be a task for the wider organisation at strategic level (with Learning and Development Division involvement).

Recommendation - That Learning and Development Division introduce a system to carry evaluation of training at level 3 (Kirkpatrick model) and in conjunction with strategic leaders that evaluation of training at level 4 is also considered.

Suggestions from Benchmarking

Crown Prosecution Service ( CPS)

470. Trainers in the Crown Prosecution Service Leadership and Learning Division in York advised us that they were planning to implement a new system to better evaluate their courses. The IT system for carrying this out is ACHIEVE. It is planned as follows:

• Level 1 evaluation form would be issued for completion at the end of the course, following the stated aims and objectives of the course. The form would include questions about the delegate's state of knowledge pre and post course. The system could be pre-set to send reminders to ensure replies were received.
• Level 2 evaluation requires the tutor to provide feedback as to how each delegate performed in the group exercise and how the skill of the group as a whole improved after the exercise.
• Level 3 evaluation involves e-mailing the delegate three months after training to score themselves on how they had performed in relevant tasks since the training event. The delegate's line manager would view the delegate's own scoring and then complete a separate scoring on their view of the delegate's performance post training.

Police

471. The Scottish Police College told us that they evaluate in three separate stages but acknowledge that not all their courses are evaluated to each level.

• Level 1 - a reaction evaluation is used. The results of this may produce an action plan.
• Level 2 - some courses are followed by a focus group to obtain more in-depth feedback.
• Level 3 - feedback is sought after six months in a percentage of courses. Previously this was done by visiting the forces concerned and interviewing officers face to face. This was considered very resource intensive so the present practice is to use 'survey monkey' - an IT tool to carry out an online survey. However we were told that the response rate for this exercise is so poor that consideration is now being given to telephoning delegates for their feedback. This may be of interest to Learning and Development Division as they have recently acquired 'survey monkey' and are considering its use in the same way.

Conclusions

472. Overall the findings relating to Investors in People, Civil Service People Survey and our own staff survey provide some very positive people results. The comments provided in the course of this inspection tend to indicate that learning has taken place as a result of the training input offered at Learning and Development Division.

473. Given these results Learning and Development Division should be encouraged to introduce more in depth evaluation of courses to measure the extent of individual course effectiveness. This can then be fed into future planning of the prospectus and the Learning and Development Strategy and will help to ensure that the main objectives of Learning and Development can be measured and ultimately achieved. It should also help with measurement of any cost/benefit analysis that is carried out.