JISC Programme meeting Birmingham 17-18th Oct.

Holly and I attended this meeting and our video was presented http://youtu.be/VSaGbPoXPh0 and we gave a poster presentation too which attracted lots of interest.

Programme and logitudinal assessment were mentioned several times by JISC and the HEA and so the Assessment Career theme is very topical. Also 2 other projects from the OU and Dundee have developed feedback analytical tools, although different from ours, and there was interest in unpacking feedback and being more transparent over the nature of feedback and what is appropriate in differnet contexts. e.g. does the very last piece of work for a programme need to have feedback? Also would it be helpful to share feedback with colleagues perhaps anonymously?

Some other points raised that we might consider were:

How do we know where the start and end of an assessment career is in a modularised programme?

What about increasing use of part-time staff who do not have an overview of the programme? Is it the programme leader’s role to provide this overview?

The term dialogue may imply lots of conversations on top of feedback process and implies work intensification.

More use could be made of peer feedback.

The concept of assessment careers may not be obvious to stakeholders other than lecturers e.g. QA or admin staff so do we need a different project slogan?

PASS programme assessment strategies event 24-25th July

This was the final conference for a NTFS funded project on programme-focused assessment or PFA. This is already practiced in tteh US e.g. Alverno College and there are examples from the project in the UK too.
This means assessment of programme learning outcomes rather than module learning outcomes and it means integratative assessment from more than one module.

The PFA approach compliments our Assessment Career approach in that it encourages linking of modules and cumulative work rather than the fragmentation that is often found under modularisation.

Other benefits inlcude the possibility of synoptic assessments that can draw on skills from the whole programme. The EdD thesis would be a good example of this.

We discussed a few issues about implementation of PFA including the possible need for flexible regulations and better working in programme teams.

I think that combining PFA and the Assessment Career approach to feedback would work well and enhance assessment on a large scale.

HEA event on self-assessment

I presented a paper on ipsative assessment at the HEA event “Self-assessment: strategies and software to stimulate learning” held at the Open University.

The concept of ipsative assessment attracted much interest but was out of context from the rest of the day which focussed on online or computer-based self-testing. I would query whether self-testing is self-assessment since it is the academic who designed the test who is doing the assessment via computer software. There was however discussion of certaintly -based marking which does require the student to self-assess on how confident they are about an answer.

Software from NZ “Peerwise ” was mentioned as being innovative in that it allows learners to create their own questions and comment on each other’s questions as well as answer them. Again this was for multi-choice type questions – but the idea could be extended to longer answer questions. I liked the idea of a community of learners discussing assessment questions.

The replay of the event is now available at: http://stadium.open.ac.uk/stadia/preview.php?whichevent=1955&s=1

Copies of the presentations are available on the HEA’s website at: http://www.heacademy.ac.uk/events/detail/2012/seminars/themes/tw037_ou

Presentation of Pilot reports to Teaching Committee

I presented the pilot plans and the pilot methodology to the IOE Teaching Committee on May 23rd. The plans were very well received with words like ‘fantastic’ used and Mary commented that it was good to hear such enthusiasm. There was a question about whether the asssignment forms would apply to formative or summative assessment or both and Ian’s pilot as he pointed out has both with 2 slightly differnet forms.

Notes from a one-day event on changing assessment across an institution

I recently attended a one-day event organised by the TeAL project at Middlesex University. This provided an interesting and frank account of the challenges as well as the successes they faced. It was particularly good to hear from senior leaders and teachers, as well as the project team. There were many issues that are relevant to this project’s work, and that of the Assessment Working Group. I brought back and shared some of the documents from the event – particularly the flow chart talking through the change process, and the table that maps things like values, roles and processes against stages and questions.

JISC webinar on organisational change

Today’s webinar consisted of accounts from the Curriculum JISC projects which are finished/finishing on how they managed large scale organisational change. Contributors gave tips on what worked at differnent stages. Some key issues I identified for the AC project at 3 stages are:

  • Start of project-resisting pressure to deliver too soon which I don’t think we have a problem with. Also stakeholder engagement which maybe we need to address including IT, registry and QA.
  •  Running the project-there was a discussion about formal reports being very dull and alternative such as use of social media, video, stories of problems and how they were solved and use of process mapping. But commitees (and JISC) want formal reports so not sure how we can be more creative. Another approach suggested was the submarine approach where not much is publicised during the project until there are results which are disseminated.
  •  Finishing the project-a long way off for us-there was agreement that it is not the system that is the product but the environment and what people do differently. We have discussed this in our meetings that enabliing more discussion across the IOE about the role and purpose of formative feedback would be a useful outcome.

When and where to have more conversations about the project at the IOE will be important for us.

See recording at http://bit.ly/jiscdslschange

IOE Learning and Teaching Conference session

Gwyneth gave a presentation about the project at the IOE’s learning and teaching conference. The presentation also involved an update from the institution’s Assessment Working Group. Gwyneth used the opportunity to invite discussion and feedback from staff, which raised several issues for us to think about, including:

  • The issues raised seemed representative of the experiences of most – but not all – of the people in the room. If anything, people expected greater use of essays than was reported in relation to formative feedback.
  • It was felt that people may feel more able to experiment and innovate with formative feedback than with summative feedback.
  • Geography uses a bridging module between PGCE and other Masters-level courses, with a formative presentation and a written summative report.
  • Some areas are already considering feedback on earlier work. In this case, feedback sheets are online; markers are encouraged to read prior feedback before marking new work. However, in this programme there’s no face-to-face feedback, which is why the written feedback needs to be careful and detailed.
  • The same project also involved a standardisation meeting that generated an exemplar that markers use as a point of reference to guide current practice.
  • In some areas, in the final summative work students are asked to reflect on how they’ve taken feedback on board, and also to self-assess the degree to which they’ve done this.
  • Peer feedback was discussed. It was felt that this had worked well in specific programmes (high quality, specific, supportive feedback was mentioned), although there were issues, including how important it is to help students understand what a good piece of work is, what the limits of trust and confidence could be in this, etc. It was suggested that working towards peer assessment required students to make progress in relation to understanding, self-assessing, etc and so could be a good way of working through some of these issues, helping people understand their own work (and what needs to be done to it) better. It was suggested that this might be easier in a PGCE group because an environment of trust and discussion was built up across the programme. It could be harder to create this online.
  • There were concerns that the flexibility of the current offer – particularly following curriculum review – could raise issues for coherence of feedback and development. Consistent contact with a tutor could help in relation to this, although there may be an issue with tutors supporting students on modules from other programmes where they don’t have expertise. Feedback on structural aspects (e.g. academic literacy) might be possible to support, however.
  • Getting assessment integrated across modules is important but is likely to take years to achieve.
  • Recognising the time needed for marking, assessment and feedback is important – it might be necessary to spend less time on teaching and more on assessment/feedback.
  • It may help to have a conversation about what good feedback looks like at an institutional level.

David Nicol webinar ‘principles as discourse’

David gave a webinar on 20th March which was recorded and is well worth looking into at https://sas.elluminate.com/p.jnlp?psid=2012-03-20.0628.M.0478009A228274AFECAC08039817D0.vcr&sid=2009077

He argues that projects need clear principles which are not just statements but have a compelling narrative and examples and evidence. It was the principles that transformed his institution in the REAP project.The success of the project will be if it enables the institution to acquire a new discourse based on the principles. This discourse wil take place throughout the institution not just between teaching staff but technical, admin and managers too.

We got a mention in the webinar and our principles relate to ‘Assessment Careers’, ipsative assessment or progress, feedforward on what the learner does next, students becoming more self reliant and there are more so please add any more.

I am thinking that an Assessment Careers framework is not the best terminology (this is the project plan and Baseline report etc.) as it sounds very rigid and a set of principles might be preferable. Any thoughts?

Notes from online seminar about Assessment Efficiency

The University of Hertfordshire presented their Strand A iTEAM project about ‘Integrating Technology-Enhanced Assessment Methods’ as well as the ESCAPE project on 15th March 2012.

Educationally effective and resource effective assessment is the gold standard of assessment methods, which often is difficult to achieve. In the postgraduate arena, summative assessment in the form of 5,000 word essays is often the norm, but how can we be sure this is educationally and resource effective?

The ESCAPE project attempts to answer this question, and they provide a toolkit to ‘calculate’ the time spent on different assessment methods (calculator spreadsheet not yet available publicly at the time of the session). This approach is not too dissimilar from the LDSE project‘s attempt to quantify the learning design process (and would work well as another component of the LDSE toolkit), and it suffers from similar difficulties: The data input is quite often based on guesswork and might not reflect reality adequately enough, though the point was made that the toolkit promotes reflection, which might unearth new insights, and more importantly, it can be used for rough (or highly accurate, depending on the baseline data quality) comparisons of different assessment types.

The online discussion quickly pointed out potential misuse of time calculations, in that they might favour efficiency over quality. However, the presenters insisted that assessment improvements should be about making thoughtful choices and not going for the cheapest option, and the tools were developed with this goal in mind. Indeed, the presenters acknowledge that the real picture is more complex than a set of numbers, and one should not forget the actual purposes of assessment, including the timing of assessment.

The ESCAPE project has therefore developed a series of assessment patterns to visualise how various forms of assessment and the associated feedback is used within modules, and how they interlink. The presenters emphasised that a one-size-fits-all approach is not really desirable – ideally, any assessment item would inform subsequent assessment items, either within one or even across multiple modules. Examples of assessment patterns are available from the Effective Assessment in a Digital Age Workshops site (see Session 7).

It is this final part of the presentation that displayed the highest relevance to our Assessment Careers project, as the ESCAPE team recognises the value of feedback that affects learning (and other assessment items) across many modules. ESCAPE’s timeline visualisation approach is something we might want to adopt – apparently the timelines/patterns have been used at several institutions already with great success, even as a part of validation processes. Interestingly, a team at Greenwich University picked up this idea and is currently developing an online tool to visualise assessment in modules in a similar way.