Evaluation

Video, video and more video!

noreply@blogger.com (Unknown) — Wed, 14 Jul 2010 10:29:00 +0000

I have spent the last two weeks video interviewing tutors and learners and editing the footage. This is proving to be very time consuming, especially where the learners are concerned. On Monday I waited around for an hour to talk to learners from the Creative Media class. They were meant to meet me at 3.30, but the tutor decided to let them watch a film as it was the end of term, so I finally got to see them at 4.20, by which time they wanted to leave. I managed to get 10 minutes of their time.

A lot of the tutors did not want me to talk to learners until they had finished their exams, at which point many of them are no longer attending College, so this has proved to be particularly problematic.

The tutor interviews have been more successful and I feel that I have more to work with in the final report. I have decided to edit them into smaller bite size pieces, which will focus on specific uses of the technology (e.g. using Assignments, Quizzes) rather than an amorphous mass of everything they did. This will be easier for me to analyse and will make for easier viewing for others.

Mahara ePortfolios

noreply@blogger.com (Unknown) — Mon, 12 Jul 2010 10:03:00 +0000

This morning I worked with the Creative Media students to help them to create views of their Mahara portfolios, in order to submit for assessment and moderation. Each student has created six folders for each area of the qualification in their portfolio, and the tutor and I decided to get them all to create the same view (as a sort of 'template' view) so that when he marks the work it will all be in the same format. We used the 'Display a folder' option for the views so that it generally fits on one screen and then if a student uploads any additional content, they do not need to change the view as it will automatically update, even if it has been submitted to the tutor.

This is an example of one of the views. The students will now generate secret URLS in Mahara, so that when the tutor is ready to submit the views to the awarding body as evidence, he will simply provide a list of URLs and the examiners will be able to view all of the portfolios.

Changes to interview plans

noreply@blogger.com (Unknown) — Wed, 30 Jun 2010 21:23:00 +0000

I had originally planned to get our multimedia designer to film my interviews, but on reflection I am going to do them myself using the flip camera. It is a more flexible arrangement as any last minute changes to plans can are easier to arrange and perhaps more importantly tutors and learners being interviewed are more comfortable with just me and a flip. Once the flip is set up and running people tend to forget that it is there and will therefore be more relaxed in front of the camera. Although I thought that the sound in some of my previous interviews was not too good, I think that was due to the rooms used and this time I have booked time in our small training room, which is nice and quiet.

I am also going to speak to learners and tutors from other curriculum areas who have used other tools to get a wider range of feedback and not the IT Diploma learners as I had previously intended to do.

14th June 2010

noreply@blogger.com (Unknown) — Mon, 14 Jun 2010 15:35:00 +0000

Today I worked with the the learners from Addey and Stanhope school who are studying on the Creative and Media Diploma. They are planning to upload all of their coursework into their ePortfolios and then submit the work to Edexcel. This is the first time we have tried anything like this and the tutor thinks that it will make a vast difference to the time it will take in submitting all the work, as he will only need to supply web links to the views of their work. I will need to spend another session with them showing them how to create the views, but we are very excited about how this will work. We are also encouraging them to use the CV feature and to use the blog, as they are required to keep a journal in this course. Due to the nature of the course they are creating videos of their work, so this is an ideal vehicle to display what they have done.

I am looking forward to being able to get some positive feedback on the use of Mahara with this group of learners - watch this space!

Preparing for final interviews

noreply@blogger.com (Unknown) — Thu, 03 Jun 2010 11:14:00 +0000

I am currently preparing my questions for the interviews with learners and tutors for the final phase of the project, which is proving to be a logistical nightmare. As well as negotiating times to interview the tutors I need to book a room to record them and also to liaise with Henry, our multimedia designer, who is going to record them for me. Although my previous interviews were recorded using a flip camera, and the quality of the video was good, the sound quality was not as good and it was sometimes difficult to hear what was said.

I have arranged to go into both the Higher and Advanced IT Diploma group classes in the next couple of weeks to get volunteers for interviews. I hope that they will volunteer to be interviewed, but if not, the plan is to offer them a store card (value £20) as an incentive, but this is really a last resort. Following the last JISC event, I had received suggestions of incentives that I could offer. One suggestion was to offer printer credits, which with another group would be useful, but these groups of learners do not need to print out their work for either assessment by tutors or final submissions for the course, so not practical in this instance.

May 2010

noreply@blogger.com (Unknown) — Fri, 28 May 2010 10:31:00 +0000

Not much activity this month. The IT diploma students have finished the controlled assessment and are getting ready to go on their work placements. They have been using their Mahara ePortfolios to create their CVs which will be submitted to employers prior to their placements. I hope to get some feedback from some of the employers about the content and usefulness of CVs presented in this way.

Following the meeting with the creative media diploma tutors and our school contact, I arranged for two members of staff from the school to have access to the learners PDPs as agreed and provided instructions for logging in. I have been checking to see if anybody has looked at the PDPs, and to date they have not even logged in to the course. This is very disappointing as the school liaison was keen to participate and had expressed a special interest in being able to see the learners attendance and punctuality information, rather than waiting for register information to be sent to the school.

They had also agreed that the learners would be able to access a chat room, in the course, on the Learner portal during their IT sessions when at school. This has not happened and Nik, the course tutor, has spent several Thursday afternoons waiting for somebody to join him in the chat room! I spoke to some of the learners to find out why they were not accessing the chat room and they said that their school teachers needed them to spend the time on their Functional Skills. We have a meeting arranged in early June which will be an opportunity to ask why the school contacts did not access the Learner Portal.

New collaborations

noreply@blogger.com (Unknown) — Mon, 19 Apr 2010 11:03:00 +0000

Had a meeting today with the Creative Media tutor and a representitive from the secondary school who the College is collaborating with on the diploma delivery. Nik, the tutor, has agreed to run an online chat on a Thursday afternoon when the diploma students are in an IT lesson at the school, so that they can communicate in between their lessons at the College. We are also enabling the school tutors to access the learners' PDPs so that they can be fully informed of the learner progress and crucially, any attendance or punctuality issues will be immediately highlighted. I hope that this will provide more data for the final evaluation, especially around the collaboration of stakeholders.

Interim evaluation report

noreply@blogger.com (Unknown) — Fri, 26 Mar 2010 10:13:00 +0000

I have now finished the interim evaluation report. It has been hard to write, as instead of having too much data to analyse (which I had feared would be the case) I haven't had a lot to work with. Feedback from students has mostly come in the form of questionnaires and discussion posts and I have been unable to interview as many tutors as I would have liked. Because the Diploma in IT is delivered solely by the College, there have been no opportunities for collaboration between the different stakeholders, which in turn has limited what I could evaluate.

I hope that in the Spring term things will be different. The learners will be going on work experience and the Mahara ePortfolio will provide opportunities for more sharing of information and feedback between employers, tutors and learners.

Also, next term I will be extending the evaluation to include a group of students, who are studying on the Higher Diploma in Creative Media. These students are 14 and 15 year olds who are based at a local school and they only attend the College once a week, so this presents an ideal opportunity to look at the effects of the support that can be offered by different features of the Learner Portal whilst they are away from the College.

February 2010

noreply@blogger.com (Unknown) — Wed, 10 Feb 2010 15:50:00 +0000

Rushing around trying to get data for the interim report and make some sense of the data I have already collected. Trying to find time to speak to tutors and learners has been difficult - everybody is so pushed for time, especially as they are currently in the controlled assessment phase of the delivery. Trying to find a quiet room to film videos, that is free when the tutors are free, is proving difficult and I have had to redo one video because the sound was too poor.

Using the PDP

noreply@blogger.com (Unknown) — Mon, 25 Jan 2010 11:55:00 +0000

I do not think that the learners are using the PDP to its full potential and I need to try another tactic to engage them. We have arranged another event for the whole Diploma team and this time I am going to bring my colleague, Angela, who is the eLearning link person for the school of CIT in the hope that she will be able to work with tutors and support them with their use of the Learner Portal.

We have also discussed ways that we can engage the staff and learners and to get them to see the value of its' use, and she has suggested that I post a question on the discussion boards in both courses in video form. As the learners studying for the Higher Diploma have been given devices to record audio for their media projects, they will be able to record their responses to the discussion and upload. Provided I frame the question and also provide a bit of background to the use of the PDP, I hope to get some more valuable feedback. Although they have already contributed to the discussion board, it feels very much like "I've been told I have to do this, so I will do as I am told"!

I do not think they can see the value that the PDP could add to their learning experience and I hope to be able to change this. I am going to provide an example of a PDP with which to inspire them and to allow them to see how it could really help them.

Second group session

noreply@blogger.com (Unknown) — Thu, 14 Jan 2010 11:44:00 +0000

I decided to try to talk to the whole group of Advanced Diploma students and this time use the Turning Point voting buttons to engage them. This was more successful than the last group - they were more animated (in fact a bit too lively) and they seemed to enjoy using the voting buttons. Jo came with me to video the session, but they did not want to be filmed. Therefore Jo tried to record their feedback by pointing the camera at the floor and simply recording audio.

After a few warm up questions, I began to ask them my questions about the use of the Learner Portal. They were happy to vote but when I asked them to expand on the answers they had given, I had difficulty in getting them to answer sensibly. They were very loud and although Jo recorded their responses it was hard to transcribe. There are a few nuggets that will be in my report.

I am not sure how much I have got out of this and whether the results are any better than if I had just asked them to complete a questionnaire. It would have been much better to do this with a small group, but they are unwilling to put themselves forward for this. Perhaps now that they have met me this might be a possibility.

Meeting the students

noreply@blogger.com (Unknown) — Tue, 12 Jan 2010 09:52:00 +0000

I have been trying to arrange focus groups with both ICT Diploma groups without much success. In both groups the students are very reluctant to put themselves forward, so much so that in the Higher Diploma class only one student is currently willing to attend - not sure that they can be a group of one! Added to this they are beginning to enter the controlled assessment phase of the course and time is getting very tight.

I therefore thought I would go into the class this morning to meet the whole class and ask them my questions whilst their tutor set up their PCs for the assessment - my thinking here was that if I met them all together they would be less reticent. Hmm - how wrong could I be. They were unwilling to be videoed and getting them to respond to my questions was like pulling teeth. One or two of the group had a few things to say, but for the most part, responses were monosyllabic and not particularly illuminating. I was hoping that following this morning session, I could encourage a couple more of the group to join in a focus group, but they are still unwilling.

I have since spoken to the tutor, who informs me that they were quiet because they were nervous about starting the controlled assessment so maybe there is hope yet. I still have to get them to complete a questionnaire, so hopefully this will provide better feedback.

I have a similar session with the Advanced Diploma group on Thursday - this time I am going to try to engage the group by using the voting buttons. Watch this space.

Cluster meeting

noreply@blogger.com (Unknown) — Thu, 17 Dec 2009 15:58:00 +0000

We held a cluster meeting for Cluster group B yesterday and it was an interesting day.

My evaluation work to date has been firmly centred around the learner experience and based on our project objectives; following the session on evaluation yesterday I think I may have to include some evaluation of the impact that the Learner Portal has on the College as a whole and I need to discuss this with Patricia and Jo. We will need to agree priorities because otherwise there will be too much data to collect and analyse. I would love to do a case study around the use of the PDP, but I suspect that there won't be enough time to include this as well.

In the meantime I'm looking forward to the Christmas break now and hope to come back with fresh eyes in the New Year!

Ho ho ho - Merry Christmas!

Not much blogging

noreply@blogger.com (Unknown) — Tue, 15 Dec 2009 15:51:00 +0000

Although the tutors have access to a staff blog, there is not much activity to date. There was a problem with accessing the blog from within the College but I think this has now been resolved.

I am concerned that I am not capturing their experiences as they happen and that even if I ask them in the new year they might not be able to recall significant events. Hopefully we can have a catch up session immediately after the Christmas break and this will prompt them to voice their thoughts about the delivery of the Diploma using the Learner Portal.

Discussion boards and staff blogs

noreply@blogger.com (Unknown) — Wed, 18 Nov 2009 14:42:00 +0000

At our project meeting yesterday Jo Burbidge and I asked the tutors to try to blog at least once a month about their experiences of using the Learner Portal to deliver the Diploma. I know that it may be time consuming but I also know from personal experience that it is invaluable when trying to remember how something feels when looking back some months later. Things that are fresh in the mind this week, will be a distant memory next summer. To help with this, Jane Teh has set up a team blog and is encouraging all tutors to contribute to this, but some tutors are finding that they have to use their Internet Use Quota time to do this, so this will need to be investigated and resolved.

We have also asked tutors to contribute to the discussion boards, so watch this space!

Contributions gratefully received

noreply@blogger.com (Unknown) — Fri, 06 Nov 2009 13:28:00 +0000

At our last project meeting we talked about getting the students to add their comments to the discussion boards that I set up. At last - the students have begun to post their thoughts about using the new features available to them in the Learner Portal (a big thank you to Jane Teh, their wonderful course tutor, for helping with this) and there is lots for me to read. I hope by replying to their posts we can open up a dialogue which might encourage them to continue to talk about their experiences.

Personal Development Plan refresher training

noreply@blogger.com (Unknown) — Tue, 03 Nov 2009 13:43:00 +0000

Held a session yesterday with the tutors who are teaching on the Diploma course in order to refresh their knowledge of using the electronic Personal Development Plan (PDP). It has been sometime since the initial training session and only now are they actively using the PDP with their students. As this initial session took place before the summer holidays and our tutors have also had to learn how to use a new Student tracking system as well as all the other aspects of the Learner Portal, it is unsurprising that they felt that a refresher was needed.

During the session some tutors had an issue with the updating of progress reviews as this is something that is, in the School of Computing, currently completed by the learner in the paper based version and then agreed by the tutor. In the electronic version, only the tutor could complete this area. We agreed to ask ULCC to amend the functionality to allow for update by tutors and learners, which they did immediately (fortunately this was a minor amendment) and tutors seem to be happier about this. I am conscious that sometimes we are adding to their already heavy workload and so I will try to do anything that I can to facilitate the use of the learner portal features with their students.

Evaluation feedback

noreply@blogger.com (Unknown) — Wed, 28 Oct 2009 10:48:00 +0000

I have now received feedback about the evaluation in the interim report which has given me some food for thought. Students who would be willing to keep a Blog were identified by the course tutors, but as yet there is no progress in this area. Unfortunately I was unable to attend the last project team meeting, due to illness, so I was unable to follow up on the progress of this.

Electronic Discussion forums

noreply@blogger.com (Unknown) — Tue, 20 Oct 2009 09:32:00 +0000

I have set up discussion forums for both the Higher and Advanced Diploma courses.

Although I had already set up a forum for the Higher Diploma last term, only one tutor had contributed to it and when I checked, the topic was not available to students, so it was not suprising that there were no posts! I have now made sure that it is available.

In each course there is a discussion area for staff only (which is hidden from students) and one for students and staff. I want staff to be able to comment freely on different aspects of using the learner portal to deliver the Diploma, hence the staff only discussion. To date there have been no posts added in either forum on either course! Hmmm! - maybe the next step is to go into the classroom in order to encourage the students to add a post. I will use the upcoming training session on 2nd November to encourage the staff to do the same.

A lull in the proceedings

noreply@blogger.com (Unknown) — Thu, 08 Oct 2009 09:09:00 +0000

I have not posted to my Blog for some weeks now. Summer holidays came (and are now a distant memory) and we have been very busy implementing the new Learner Portal across the College. I have been heavily involved in training for the wider college and this has taken a large proportion of my time.

A meeting took place at the beginning of October with the course leaders of the two IT diploma courses and this provided an opportunity for me to outline what I need from them. As a result of this meeting we are planning to do some more training in the use of the PDP and other aspects of using Moodle tools such as submitting assignments electronically. I need to ensure that all staff are using these tools effectively in order to properly evaluate the effects of their use.

Writing up my findings

noreply@blogger.com (Unknown) — Thu, 13 Aug 2009 14:27:00 +0000

The time has come to make some sense of the baseline data that I have gathered to date. Although I had to present findings to the curriculum teams at the event in July, I was able to generalise to a degree - now I must collate the information and present it in a report format for contribution to the Project Manager's report. I have been editing videos that I shot back in June and it is a useful experience (I'm very glad I videoed the interviews now - I would never have remembered everything). Reviewing these interviews has brought back lots of small nuggets of information that will help to summarise the way the teams worked before the intervention of the various technologies (Moodle, including electronic ILP's, Mahara etc.)

Evaluation feedback to Curriculum team

noreply@blogger.com (Unknown) — Fri, 17 Jul 2009 10:52:00 +0000

July saw an event which aimed to showcase what the curriculum teams have achieved to date and I was able to feedback to the teams the results of initial baselining. The video below is my presentation to the teams

This event also provided an opportunity for the teams to think about how they will use the different technologies in their delivery, which in turn ensures that I can evaluate as per my plan

Learner interviews

noreply@blogger.com (Unknown) — Wed, 08 Jul 2009 08:45:00 +0000

My attempts to interview the learners were not very successful. Unfortunately, by the time I was ready to interview them, they were no longer attending classes at the college, so my plan to hold a focus group with them and to use the voting buttons came to nothing. Although I did manage to speak to a couple of them, they were reluctant to be videoed. I used a voice recorder to record our conversations, but the quality was very poor and the only voice that stands out is mine! Fortunately, this was not crucial to the evaluation.

I am going to plan my interviews and focus groups further in advance for the coming academic year and make sure that the tutors are aware of these dates, so this does not happen again.

On the move

noreply@blogger.com (Unknown) — Thu, 25 Jun 2009 08:32:00 +0000

I'm on the train to Birmingham - again! Whilst the breakfast is very welcome - it would be nice if an event could be held in London. Today's event is all about looking at the learner experience. I am getting ready to do a group interview with the Diploma students in the next week or so, so I hope I will be able to use what I learn today to help me ask the right questions. I am going to use voting buttons, which I hope will lead to a fun and engaging session.

Questionnaires and interviews

noreply@blogger.com (Unknown) — Fri, 08 May 2009 13:27:00 +0000

The learners have now completed the online questionnaire and I now need to save the results electronically. I have begun to interview key staff who are working on the Diploma and my attempts at interviewing are highlighting some issues.

I am using a a very funky orange Flip camera to record my interviews. In the first interview, conducted with a member of the Entitlement and Support team, I held the camera and frankly it looks like we did the interview sitting on a bus. Second time around, I used the tripod that comes with the Flip. Although the film does not look as jerky, the angle of the tripod meant that the subject was looking down and in her own words it made her look miserable! Third time lucky then...I have just conducted my third interview, this time with a professional tripod and this enabled me to sit behind the camera and the interviewee was able to look at me as she answered the questions. An altogether better result and I will use this tripod in the future.

My questions have been OK and I have managed to stick closely to the questions as written and this has been fairly successful. I intended to interview a number of other staff from the school of Computing, but I am going to ask them to submit their answers on paper now as we are close to going live with the Diploma course on Moodle and I want their feedback before this date (12th May).