Thematic report on learning and development

141. Assuming that learning needs have been identified we then looked at how this was taken to Learning and Development Division and translated into a suitable training package to meet those needs.

142. It was not possible to examine all of the courses on the prospectus. Nor did we consider it appropriate to observe and make comment on the delivery of training as such, although we are familiar with the methods of delivery having attended courses at the Prosecution College in the recent past. We were given access to the shared folder containing all course materials and this aided our inspection.

143. We selected a few different courses to examine whether standard processes (that is to say the processes outlined in the Standards Manual referred to below) had been followed. These were Sexual Offences Act, Disclosure, Deaths, and Cashiers' Best Practice (e-learning). Our examination involved discussion with the lead trainer involved and a review of the course materials.

144. Where trainers had moved on from Learning and Development Division there was no record available in the Division as to the history of the course in terms of its design. For the purposes of our inspection we relied on evidence from interviews with trainers where we were unable to view materials from the early stages of this process. This was particularly the case for training materials which had been taken on and adapted from the former training division.

Standards

145. A Standards Manual is available to staff in Learning and Development describing 'end to end' processes for producing Learning and Development products. It is stored on the shared drive for access by all staff in the Division. It was to be subject to a change control process but appears not to have been updated since July 2008.

146. The trainers we interviewed claimed to be aware of the existence of the Standards Manual but some said that it was not always followed. Others said it was not used at all. One trainer claimed that 'industry standards' were followed, referring to recognised standards promoted by the Chartered Institute of Personnel and Development. Another referred to following National Occupational Standards for accreditation purposes. That said, of the courses that we examined during the course of this inspection it was clear that some but not all steps in the process mapped out in the manual had been completed.

147. The manual makes it clear that not all courses would have to follow all of the steps in the process. It would depend on the size and scope of the training package concerned. The main steps in the process are listed as follows:-

1. Learning Needs Analysis

2. Course Design

3. Quality Assurance - Sign off

4. Evaluation

5. Change Control/Version Control

6. Lessons Learned Log

Recommendation - That the Learning and Development Standards Manual is revised and updated for use by all staff in Learning and Development Division, that its use is encouraged and monitored, and that it remains updated to provide a full and comprehensive guide to all staff in Learning and Development Division. Where National Occupational Standards are followed for course accreditation it should be for the Head of Learning and Development to determine if these are sufficient to meet the standards specified in the manual.

Learning Needs Analysis

148. The Standards Manual states that a document should be prepared which accurately reflects the information supplied by the course sponsor, expert or policy or practice 'lead' who has identified the learning need (as described in the previous chapter). Guidance is given as to how to supplement the basic information with further information gathering techniques such as workshops, questionnaires, telephone surveys and existing documents to obtain the fine detail required. There may be differences in approach to conducting a learning needs analysis depending on the topic.

149. Our research also suggests that a proper learning needs analysis may also identify the most appropriate time to deliver the training and the target audience. The analysis of training needs should not only consider essential content but also acknowledge that people have different learning styles and address that issue by offering a variety of learning media where possible.

150. From our course sample inspection we observed a full and detailed Learning Needs Analysis for the new Sexual Offences Act training which was prepared following direction from the project board. This document identified different target audiences and methods of learning for those groups. Face to face training was considered necessary only for legal and precognition staff working in Area Sexual Offences teams, National Sex Crimes Unit, High Court as well as all Crown Counsel and the head of Victim Information and Advice.

151. Legal staff and Victim Information and Advice staff working outwith the specialised teams would receive e-learning and other guidance cascaded by circulars. Timing of courses to the target audience was planned over a seven month period up to December 2010, thereafter the material being incorporated into the existing two day Sexual Offences course.

152. The implication of the training plan for the new legislative provisions is that any member of legal or precognition staff who has already attended the sexual offences course and who is moved to their Area sexual offences team after December 2010 will not receive the face to face training on the new legislation but only the cascaded guidance.

153. This may be of some concern in the Service. One senior member of staff made this comment -

• "….. some of the training on new developments is restricted to those staff actually working in that particular area at that time (Solemn Renewal springs to mind, and I understand plans for the new Sexual Offences Act are similar) leaving knowledge gaps among some staff and reducing the flexibility of the workforce as the wash up courses finish shortly after the main tranche of courses. This means that not all reforms are fully embedded." (Q7, 55)

154. We highlight this concern as expressed to us and suggest that the matter be kept under review. Some staff who previously attended the two day core sexual offences course may find that they need to repeat the course to appreciate the complex changes to the law in this field. We therefore make no formal recommendation in this regard.

155. We learned that for disclosure training, the lead trainer was involved in the project at an early stage attending the disclosure reference group chaired by the course sponsor. In this way the consideration of the content and timing of training was an integral part of the project. The target audience was decided and priority given to those working in High Court units first as they would be subject to impact of the new procedure before it was rolled out to Sheriff and Jury teams. 'Wash-up' sessions for those who missed the training were also planned before the course then remained on the prospectus as a core vocational course.

156. We were not clear if there was a formal Learning Needs Analysis for this course as we were not shown any pre-course planning materials. Nevertheless the final course materials show clearly stated aims and objectives for the course and a range of learning 'interventions' including power point presentations, handouts and case study examples for practical workshop exercises.

157. The Deaths courses were based on earlier course materials as was the Cashiers' Best Practice e-learning course so a less formal approach was taken.

Course Design

158. The manual describes steps to be taken in designing a course, referring to both a high level and low level design. High level design gives an overview of what the course covers in terms of content. Low level design is a much more detailed document specifying what is to be covered. From these documents the trainer should then be able to produce a document entitled "Aims & Objectives" and the course materials are developed following the specifications of the aims and objectives. The trainers bring to this exercise their expertise and knowledge about how people learn and apply different methods of learning to the package, such as PowerPoint presentations and class exercises. We found that all courses on the Learning and Development prospectus do have clearly stated aims and objectives and indeed these are accessible for all staff in the Service by accessing the IDEAL page on the intranet.

159. It is at the design stage in the process that the diversity proofing tool available on the Crown Office and Procurator Fiscal Service intranet is to be used. All courses should be subject to diversity proofing to ensure that:

• "Everyone has the right to be comfortable in the training environment' and 'Learners should feel at ease when attending a training course and should be able to participate without fear of being ridiculed."

160. We were unable to verify whether any diversity proofing exercise was carried out in relation to any of the courses we looked at. The Head of Learning and Development confirmed that it was not possible to evidence that equality impact assessments had been carried out although we were assured that trainers displayed an awareness of equality and diversity issues in their work. For inspection purposes we consider that this type of process is essential to evidence and record.

Quality Assurance

161. The manual specifies that various levels of quality assurance should be built in to the process. The stages are named as QA3, QA4 and QA6. QA3 is an internal review by another member of staff, examining in particular that appropriate standards have been applied, grammar and spelling are correct, the layout is appropriate and that the aims and objectives are met by the contents. QA4 - a group of both internal and external reviewers, previously agreed by the course sponsor check the materials for procedural and technical accuracy. QA6 - final sign off when any previous amendments have been carried out following earlier stages. A pilot may be run if thought to be appropriate in addition to the quality assurance steps outlined. After the product 'sign off' the materials were to be stored on the shared drive as the baseline product.

162. Without access to actual materials of the pre-course work carried out we cannot say if the specific and detailed steps outlined in the manual were carried out for the courses we looked at. This is with the exception of the new Sexual Offences Act training to which we did gain access (because it was in the process of being designed during our inspection) and which did incorporate all the process steps described in the manual. We are however assured by the trainers that some sort of quality assurance process was undertaken for each course although not perhaps to the degree specified in the Standards Manual. It would appear therefore that there is not a consistent approach and much depends on those involved.

163. The quality assurance procedure for the deaths courses involved a single 'check over' by the lead in Policy Division with no further revision of materials. The course sponsor or portfolio owner was not directly involved with this process.

164. For disclosure, we were advised that quality assurance steps were undertaken with both the policy lead and reference group involved in revising materials.

165. For the Cashiers' course a sample of staff and managers were asked to take part in a trial and evaluation. These reviewers contained a group of staff unfamiliar with cashiers' duties to determine if the product was easily understood. After final changes were agreed by the course sponsors the product was launched.

Recommendation - That record keeping of design and quality assurance steps are maintained for audit/inspection purposes, particularly to show that an Equality Impact Assessment has been carried out for the package.

Evaluation

166. The manual refers to a 2008 strategy for evaluation. This highlights the need for Learning and Development to provide positive, measurable evidence of the contribution of training to performance improvement and achieving specific business needs and objectives. Evaluation can be carried out at different levels. The strategy referred to the five levels identified by Kirkpatrick ⁸:-

• Level 0 - Pre-training assessment of knowledge - a useful benchmark against which to measure knowledge, skills, behaviour, and attitude after training has taken place.
• Level 1 - Reaction - an immediate response at the end of a course (sometimes referred to as "happy sheets").
• Level 2 - Learning -this can be established during a course by a range of methods such as quizzes, practical tests, case study exercises.
• Level 3 - Behaviour - this involves the line manager and/or delegate providing feedback as to the impact of the course.
• Level 4 - Results - at corporate level, a measurement of the impact of a training course on the organisation or section of it.

167. The manual recommends that the trainer developing the product should incorporate a plan as to how the product will be evaluated, and this should be done in consultation with the course sponsor. High level evaluation - that is to say at level 3 or 4 was NOT mandatory. If the course sponsor requested this level of evaluation it was suggested that this could be done by either telephone follow up or by calling together a focus group of delegates who had attended in the previous 3-6 months to find out how effective the training had been for their work.

168. Lower level evaluation by way of reaction sheets is mandatory. It is not clear from the manual if level 2 evaluation was mandatory.

169. The results of all evaluations, at whatever level carried out, were to be collated into an event report providing a synopsis of the views expressed in the reaction sheets. A fuller report might be prepared following a focus group.

170. As we have already said, the manual has not been updated since July 2008. We are aware that a new evaluation strategy was developed in Learning and Development in 2009 and further work was being undertaken in this regard as we inspected. Despite a re-working of the evaluation strategy the actual practice of evaluation appeared not to have changed in the time we were inspecting.

171. Some trainers told us that their practice is to carry out a quick poll of knowledge/skill at the start of a course, which might satisfy Kirkpatrick's level 0. We specifically asked in our survey if there was any pre-course evaluation carried out in the last course staff had attended at the Scottish Prosecution College (either carried out before the delegate attended, or at the start of the course). Most respondents indicated that there was no analysis of their knowledge/attitude before the course. 25 respondents indicated that such an analysis was carried out at the start of courses such as Diversity, Certificate in First Line Management and Advocacy courses.

172. Managers were asked in their questionnaire if they had carried out a pre course skills/knowledge assessment for any of their staff. Two indicated that this had taken place, one for the Certificate in First Line Management course, the other did not give details of the course. 23 (or 92%) out of 25 managers said they had carried out no such exercise.

173. We were able to view all the course materials for courses on the prospectus including reaction sheets and the event reports summarising the feedback. It appeared that all courses were evaluated at this basic level. Trainers advised that they used the feedback from these sheets to make minor changes to the course. We were advised that the evaluation sheets and the event reports were seen by the Head of Learning and Development and then filed.

174. For the deaths courses, although there was no indication at the planning stage of how the deaths courses would be monitored and evaluated, standard 'reaction' sheets were used to evaluate the courses. The lead trainer indicated that the course was continually monitored in light of feedback from the reaction sheets and in light of any changes in policy/practice.

175. For the disclosure courses, evaluation was carried out by way of 'level 1' reaction sheets. Although there was not a 'pilot' as such, following some feedback contained in the reaction sheets after the first course was rolled out, the lead trainer made some changes to one of the case scenarios to better focus the learning point.

176. The cashiers' course had both pre and post learning evaluation sheets for completion. Although the course is an e-learning package it is supplemented by a booklet containing questions to check knowledge and to be retained as a reference. The delegate's line manager is involved in confirming that learning has occurred as they receive a booklet containing the answers to the questions. In this way evaluation is taken to the second level.

177. 'Level 2' evaluation is incorporated into a number of courses in terms of a knowledge check. Many course materials we viewed contained case study scenarios, quizzes, or involved some 'doing' to confirm the learning (such as the advocacy courses, mock trials).

178. More formal evaluation by way of assessment is incorporated into some of the externally acquired courses such as the Institute of Leadership and Management courses at level 3 (Certificate in First Line Management). Here delegates must confirm their learning by completing and submitting work which is formally assessed by accredited assessors and this work goes towards the module to attain a formal qualification.

179. Beyond this we found no evidence of any higher level evaluation. This means that 'level 3' follow up evaluation - after a period of time - say 3 or 6 months to ascertain if learning had been effective in the workplace is not presently carried out. This was confirmed by managers in Learning and Development and by the results of our survey.

180. Of course the Standards Manual advises that such high level evaluation is not mandatory and should only be carried out if the course sponsor requires it to be done. It is accepted that carrying out any higher level evaluation is resource intensive.

181. We were advised that some type of follow up evaluation was attempted previously. This involved sending an e-mail to the delegate and line manager. It was reportedly difficult to obtain a response and this may be one reason for its discontinuation. We discuss the merits of higher level evaluation in Chapter 11 - Results.

Change Control/Version Control

182. The manual made provision for a formal request to make a change to a 'signed off' training product. This request was to be directed to head of training and relevant stakeholders. Any changes to a baseline course would be recorded by way of a version control document stored with the course materials. This system was not in use as far as we could determine.

183. Indeed the trainers we interviewed told us that they made minor changes to format themselves, although significant changes to the materials would be checked with the course sponsor. No firm criteria could be supplied as to what might be referable to the course sponsor (assuming there was one). The 'change' forms suggested in the manual were not mentioned.

184. After the initial Deaths 2 course was delivered the lead trainer contacted the course sponsor suggesting changes to the format of the course and broadening the remit. This was agreed by the sponsor. This proposed change was at the instance of the trainer rather than as a result of feedback from the delegates at the first course.

185. With regard to the disclosure course, following some feedback contained in the reaction sheets after the first course was rolled out, the lead trainer made some changes to one of the case scenarios to better focus the learning point.

186. Not all changes to materials were at the instance of the trainers. We were told that often changes are required due to a change in legislation, case law or other policy/practice guidance. This was particularly the case for disclosure where guidance was being continually updated.

187. Some trainers expressed frustration that not all changes in legal policy or HR policy or practice were brought to their attention. While we understand that frustration, ultimately the lead trainers are responsible for the accuracy of the product they deliver. We understand that Learning and Development Division try to be proactive in communicating with internal partners although some groups or forums are set up without the knowledge of the Division and some changes in policy or practice are not communicated.

188. Where changes are made for example in policy or practice guidance these are generally brought to the attention of all staff by way of news items advertised on the PF Eye (the Crown Office and Procurator Fiscal Service intranet home page). If trainers miss the 'What's New' information on PF Eye they can access archived news. We suggest that as a matter of routine that these items are checked by all trainers for matters relevant to the courses in which they are 'lead' trainers.

189. We heard anecdotal reports from both trainers and delegates about instances when training materials were out of date and this was raised during the course by delegates who were aware of new developments before the trainers were. These instances appear to be exceptional rather than commonplace. None of the respondents to our survey reported this experience.

190. We did find version control templates in some but not all course materials. Where such a template was with the course materials it was blank. Indeed looking over all the course materials stored the only course where the template showed a number of changes recorded and dated was the diversity course. Since we were told of changes to baseline products by trainers we assume that the version control template had not been used to record these amendments.

191. Examples of changes reported by trainers include:

• providing more information on assessment criteria for ' CFLM' (Certificate in First Line Management) course
• changing the format of some of the ' CFLM' modules
• redesign of part of 'Cashiers' Best Practice'
• more practical based sessions in the 'Deaths' course
• amendments to the final case exercise in 'Disclosure Schedules'
• introducing a practical exercise first day of 'Advocacy 1' rather than full day of presentations
• cut back on some presentations to avoid repetition in 'Vulnerable Witnesses' course

Recommendation - That version control records are completed and maintained for all course materials.

Lessons Learned Log

192. This is described in the manual as a mandatory step in process following 'Prince 2 project management methodology'. A 'lessons learned' log was to be used to record good and bad experiences to inform any improvements.

193. This does not seem to be in use at all. It is perhaps a formal process which does not sit well for some courses. If the purpose in logging lessons learned is to share them with fellow trainers and use for future planning what might work better is a living document relating to all courses, stored in the shared drive, to which additions could be made by any trainer at any time. Such a document might be used in joint trainers' meetings to be the basis of discussions on practical problems and good practice.

Recommendation - That consideration is given to the reintroduction of 'Lessons Learned' logs and the use of these to assist in discussions between trainers at their regular meetings.

Other Standards - National Occupational Standards

194. The Learning and Development Manager advised us that it was the aim of Learning and Development to map courses to National Occupational Standards ( NOS) where appropriate.

195. NOS are statements of competent performance, agreed in the sector at a national level and often form the basis of qualifications such as NVQ (National Vocational Qualifications) and SVQ (Scottish Vocational Qualifications). Since Learning and Development now have a number of courses available on their prospectus that are linked to qualifications, mapping their courses to NOS in this way is essential for those courses.

196. There are many companies that provide advice to the public and private sector about how to standardise their procedures within an industry or sector. Learning and Development consult one such public sector organisation - Skills for Justice in relation to their administrative and management courses. Skills for Justice is the sector skills council and standards setting body for the justice sector. Whilst mapping to national occupational standards might be suitable for some of the administrative and management courses delivered at the College it was not clear whether these would apply to legal courses. We were told that this had not yet been considered by the legal training team.

197. One matter not mentioned in the Standards Manual but which we encountered during our benchmarking activity was the question of set reviews of course materials. At the Scottish Police College at Tulliallan we were advised it is policy to carry out a full review of each course every two years, even if there have been minor amendments during that period. At the review the course is examined for continuing relevance and to ascertain if it is still fit for purpose.

198. Similarly at the Crown Prosecution Service leadership and learning division course materials are given to the operational 'lead' for the topic covered by the course so that materials are given an external check for accuracy on a regular basis.

199. This appears to be a sensible practice and we commend it for consideration by Learning and Development Division. Such review might touch on any new case law or legislation since the last review, any identified practice gaps following work monitoring (for example by reference to the latest examples of identified good and bad practice picked up at operational level) and brought to the attention of the portfolio owner and/or policy lead for the topic.

Recommendation - That consideration is given to introducing a mandatory review of each course at specified regular intervals. The minimum period between reviews should be fixed in consultation with the course sponsor at the design stage.

Conclusions

200. Whether using a bespoke Standards Manual or National Occupational Standards as a framework for the process involved in the design and maintenance of learning products we believe it is desirable that a consistency of approach is demonstrated and monitored by management in Learning and Development Division. Given that National Occupational Standards are not universally applicable a Standards Manual would provide such consistency. Such a manual could be a significant resource at times of staff turnover in the Division. Indeed it might form part of the induction process for new trainers in Learning and Development.