Weeks 12 & 13 (cont.) - Summary of results

Data analyses of Student Survey (for usability)

Table 4 below indicates that overall, 72% of students responded in a positive manner to the 20 selected standards from Items 1, 2 and 4 . However 18% disagreed or strongly disagreed. These responses relate to Standards 1.8, 2.1, 2.6, 2.9, 2.10, 2.13, 4.1, 3.2 and 4.5 . Finally 9% of responses indicated N/A, referring to Standards 2.9, 2.10 and 2.14.

Table 4 – Summary of Student Feedback s
hown as % frequencies (n=5) graph

As well as responses to Standards 1, 2 and 4, there was an area at the bottom of the online questionnaire for respondents to write general comments. All responded and feedback was positive.

Data analyses of Staff survey (for quality)

Table 11 below indicates that overall, 88% of all staff responded in a positive manner to the 20 selectd standards from Items 1, 2 and 4. However 11% indicated a N/A response. Appendix 10 identified one role made these responses and they relate to Standard 1.6, 1.7 and 1.8, and all of Items 2 and 4.

Table 11 – Summary of All Staff only Results shown as % frequencies (n=9) graph

Table 12 below indicates that overall, 100% of online staff responded in a positive manner to the 20 selected standards in Items 1, 2 & 3 (see also Appendix 9).

Table 12 – Summary of Online Staff only Results shown as % frequencies (n=5) graph

Table 13 below indicates that overall, feedback from the Curriculum & Academic Services staff member indicated that 95% of the selected standards in Items 1, 2 & 4 (see also Appendices 10 and 11) were N/A (not applicable).

Table 13 – Summary of CAS Staff only Results shown as % frequencies (n=1) graph

As well as responses to structured questions, there was an area for each Item for respondents were able to write comments.

The full version of results are available - click here to access.

Week 3 - eLearning guidelines for quality

This week’s task is to identify 2 eLearning guidelines relevant to my area of practice (with the School of Business staff teaching fully online and/or in a blended delivery environment).

eLearning guidelines

In the past year as the eLearning Advisor & Developer I have identified the following "eLearning Guidelines for New Zealand" that I wish to specifically to address relating to quality in online teaching and learning.

1. 1.1 Teaching staff: learning design: learner/centred
   
   TD5: has a representative sample of students tested the elearning materials and any necessary modifications been made?
   
   
2. 1.2 Teaching staff: teaching relationships: collaboration
   
   TT13: does the teacher evaluate the elearning during their course to identify its effectiveness and how to improve it?
   

Quality eLearning issues

In 2008 we introduced our third LMS – Moodle. Teaching staff had been using Colts LMS and Blackboard for just over five years. During this time, an ongoing issue from an eLearning Advisor and Developer’s perspective has been the lack of internal quality control procedures. There is no formal moderation processes of online resources centred around best practice models. We have yet to develop a relevant and checklist for design and delivery of an online paper at UCOL.

How the guidelines may address the issues

I would hope that the introduction of a student evaluation by Online Tutors, with the support and guidance of the eLearning Advisor and Developer would assist in the ongoing improvement and effectiveness of online courses.

The criteria for evaluating the quality of online courses would be developed based on research and collaboration.

---

Milne, J. & Dimock, E. (2006). eLearning guidelines - guidelines for the support of elearning in New Zealand tertiary institutions. (version 0.8). Massey University.

Wright, C. R. (n.d.). Criteria for evaluating the quality of online courses. Alberta.

Week 2 - Quality & Evaluation

1. Why is evaluation important to you and how do you define it?
   
   Evaluation is important as it can provide a range of different and useful information to assist in future course design, planning and implementation.
   
   Evaluation is the process through which we examine the learning opportunities and experiences we offer our students and make judgements about their effectiveness and educational value of techniques and resources, as well as the costs. As Heather stated in her Week 2 posting "evaluation is a way of determining what is needed". How? By looking at a broad range of evidence in order the gauge the effectiveness of the elearning – for example, including a course assessment as a method of evaluation data collection as a way of keeping quality issues clearly in focus.
   
   "Evaluation is a continuous ongoing process (Gunn, 1999) that is "fundamentally about asking questions, and then designing ways to try to find useful answers" (Manwaring and Calverley, 1998). It is an expensive and time-consuming process and it is essential that it is worthwhile:
   
   

   "If the answer to the question 'why evaluate?' is that the results will lead to action to improve the teaching and learning within the course or institution,
   then all the effort will be worthwhile." (Shaw, 1998). "

   (Higgison, C. 2001).
   
   
2. What sort of evaluations mentioned on the presentation are familiar to you already and why?
   
   • Observation – in some assessments, observation is required to identify whether the student is able to demonstrate competency in performing a task – this can be observed by audio, video recording or F2F.
   
   • Questionnaires – I have used online surveys, hard-copy evaluation handouts and verbal questions to evaluate teaching and learning.
   
   • Focus groups – regular meeting with our stakeholders –relevant Industry Advisory Board representatives.
   
   • Expert review – internal (pre- and post-) within a School by peers and/or Senior Lecturers, by Subject Matter Experts, by Curriculum & Academic Services at UCOL and across campuses, and external moderation with NZQA.
   
   • Checklists – depending on the paper or unit standard being assessed – often combined with Observation.
   
   • Feedback in a discussion forum – by quality participation, text chat feedback and the use voting buttons (in Elluminate).
   
   
3. Why is quality important in eLearning?
   
   Learning and teaching that incorporates the use of learning technologies is complex and evaluation is seen as the key to developing an understanding of the factors that influence its success. That success will depend on the quality of its instructional design and the academic and technical supported provided to learners and online tutors.
   
   I agree with Joy's comment: "evaluation is based on asking questions, collecting data or information that answer the questions, through analysis then make the decision and take action to ensure the whole project / system fulfil planned outcome. It can be varied depends on what questions we are asking and what result we would like to get."
   
   Quality assurance standards are the responsibility of the institution and therefore it is essential they evaluate their practices. It plays an important role is satisfying the demands of external (and internal) scrutiny. This ensures that innovations are subject to the same institutional scrutiny and evaluation processes as traditional F2F delivery.
   
   However, as Brownyn has suggested in her slide presentation – “we have got away it so far, and have not really bothered to do much evaluation apart from checking about what students think of the course at the end of the course …..” therefore examples of sound pedagogy and integrity of best practice in elearning are at risk. As Michelle states "it is imperative that the course material and activities are of a high quality."
   
   Specific indicators for measuring quality include:
   
   • assessment of student learning,
   • feedback from students,
   • peers and external reviewers and
   • institutional accreditation procedures.

---

Gunn, C. (1999). They love it but do they learn from it? Evaluating the educational impact of innovations. Higher Education Research and Development 18 (2): 185-199.

Higgison, C. (2001). Online tutoring e-book. Chapter 5 Evaluation. Institute for computer-based learning. Edinburgh.