Should students be trusted with quality assurance?

It seems that at the centre of the recent consumer argument in higher education is a fear from certain thought leaders that a more engaged student could lead to the productisation of the sector.  Intrinsic to this position is the paternalistic idea that the student does not always know what is best and therefore cannot be entrusted with pedagogic leadership. The fact is that these supposedly disengaged students through research and peer influence do successfully seek out and identify the best universities and programmes.  This begs the question; do universities become great because of or exclusive of the active engagement and participation of their students?

In the Times Higher Education article published on 12 April, QAA’s new riff on student feedback: positive notes or waves of jargon?  Reporter Jack Grove points out that “Plans to involve students more heavily in quality assurance have been criticised for introducing unnecessary “jargon-encrusted” bureaucracy to universities.

Students are talking – are universities listening?

Institutions are very active in quality assurance efforts through the NSS and many other survey projects.  Yet, one of the biggest challenges universities face is in letting the student in on results especially at course and module level.  If the opinions expressed in the Times Higher article are an indicator, a potential reason for the lack of student engagement may have something to do a patronising view of the student rather than their desire to participate in improving the quality of their education.

Does transparency matter?

James Williams, Associate editor, Quality in Higher Education added this comment to the Times Higher article. “It is to be hoped that the code will encourage institutions to explore some of the ways in which student feedback was collected before the advent of the NSS. The most successful used instruments that were designed in consultation with the students themselves and that reflected their concerns rather than those of senior managers or statisticians. The resulting data were triangulated with other important sources of intelligence to inform (not dominate) management decision-making.”

Could it be possible that students can be trusted to actively direct their academic future? There is evidence of their abilities in this area through their use of informal channels and social media in making informed decisions as well as voicing their opinions.  Informal peer networks are a reality and simply monitoring Twitter and Facebook will not protect the reputation of academics and institutions. With this in mind it makes sense that universities take a lead role in cultivating partnership and consultation and consider the student as active stakeholder in their education.  Improving survey administration and quickly turning around survey results back to students thereby closing the feedback loop based on statistically valid evidence is a prudent way to cultivate trust, transparency, and engagement.

Should social media sites drive a Universities reputation?

In the 8 March 2012 edition of the Times Higher Education there was an article Leader: Exit the comfort zone 8 March 2012 which indicates that “Until relatively recently, students who wanted to let their lecturer or university know what they thought of them had limited options. They could tell the tutor to their face, perhaps make a formal representation to the university, then go to the complaints watchdog.”

University vs. Student Engagement

In order to improve student engagement, institutions should strive to consider the student any time they create a survey.  They should ask: Is this question relevant? Does the answer to this question result in something we can quantify or improve? Is this question something we are able to change? How are we going to report the findings and resolution back to the students?

Even though the point made in this article seems to contradict the reality of the multitude of surveys that students are being asked to complete, it does reflect the fact that there is a disconnect in what happens once a student completes a survey.  The risk is that this disconnect leads many students to turn to their peers or as the article indicates “…or scrawl something rude on the lavatory wall. Today, through websites such as RateMyProfessors.com (the digital equivalent of the aforementioned wall) and regular student surveys, their opinions are far more assiduously sought.”

Protecting the reputation of the institution

Sites such as rateyourprofessors.com are symptomatic of a breakdown in the engagement with students at the most core level, the course. It’s only natural that students want to know what their peers think about an academic or the module they are about to sign up for. In the increasingly transparent world driven by social media, institutions need to take an active role in driving the feedback process in order to protect the reputation of the institution as well as the reputation of their academics. By quickly turning around results from module evaluations and publishing results and a summary of the module review on the VLE, institutions can facilitate and control information available on their courses thus engaging with students who are looking to register for a course and close the feedback loop by including the student as a stakeholder.

Should ratemyprofessors.com become the voice of the student?

Should ratemyprofessors.com become the voice of the student?

In the 9 February 2012 edition of the Times Higher Education there was an article ’If student feedback stings, go looking for more’ that highlights recent research by two American academics who happen to be highly rated on the social media site ratemyprofessors.com. The academics argue that “rather than dismissing the site, a way to turn the frown upside down is by encouraging more students to use it.”
The article goes on to point out that “the authors of the new study claim that, in addition to potentially improving lecturers’ scores on the site, obtaining a more representative sample of responses will stop academics from falling into a ‘vicious cycle of negative experiences and comments’.”

So far, social media sites such as ratemyprofessors.com have not caught on here in the UK. However, this could change. In fact, HEFCE have recently funded some research into similar social media initiatives.
Does it make sense to push students to ‘rateyourprofessor.com’ to create a representative sample?

In my view, the answer is a resounding no. Students are verging on overload when it comes to surveys. In fact, speaking recently to a quality manager at a major Scottish University – he indicated that some students are being asked to complete 20 separate surveys a semester. This institution is not unique as most universities run the NSS, PTES, PRES, ISB, DLHE student satisfaction initiatives, as well as module surveys. Now, according to this research academics are supposed to also push students towards ratemyprofessors.com. I think not. This approach equates to creating more noise as various groups in the institution are looking for more and more feedback without taking into account the impact on the student as a stakeholder in the process. More ominously, both the institution and their academics could be open to very public harsh criticism which may have long term ramifications to reputations and careers with limited recourse.

Do institutions need to make it easier for students to complete surveys?

A logical reaction to the rise of social media is to try and follow students online to increase engagement. It is important to look at how students use social media sites and monitor technology trends in order to increase engagement. The focus however on making it easier to give feedback misses the point. Institutions are actually in an enviable position as they have the ability to ask students in class on the day to complete surveys and gain a representative sample. Additionally an ‘in class’ approach minimises the risk in over marketing another online survey that could have a negative impact on response rates for other survey projects like the NSS.

Student as a stakeholder

The bigger issue is that institutions need to understand what and how often they are asking students to give feedback and actively report or make available results back to the students for their efforts. By rewarding the students through making available valid reliable results will help students make decisions as to what course or academic is right for them, Institutions can protect their reputations and the careers of their academics all the while engaging more with their students.

Students are suffering from survey fatigue

As we enter into the 2012 academic year – the reality of the NSS and more programme level surveys is upon HE institutions. As Sally Feldman points out in her valuable commentary on Times Higher Education

What I find telling is the first comment to the story from a student who states “Unfortunately my University seems to spend more time on PR promoting the survey than actually making the improvements that would lead to improved scores.”

Survey Abuse: There is no such thing as survey fatigue – rather – the proliferation of multiple external benchmarking surveys can lead to what we refer to as ‘survey abuse.’ Survey abuse occurs from badly administered surveys and is amplified by the fact that students are very often left out of the loop and never see the results of their efforts to provide feedback. Institutions should have an understanding of all the surveys being run across the institution and make a concerted effort at transparency by including students as stakeholders and by clarifying what actions will be taken based upon the results of feedback.

Baby’s relationship to bath water: Universities need to run surveys.  Not only due to the number of services they offer – but also because culturally, they are institutions of research. While survey abuse is bad, it is however not a reason to stop running surveys.  Institutions struggle with engagement with their students and require frank and timely feedback in order to monitor and improve the quality of their courses and services. It is important to focus on improving the survey process and consider the impact of any survey project on the designated population and specifically plan on closing the feedback loop with those being asked to participate.

Improving quality and standards in Higher Education – addressing the needs of course evaluation

Improving quality and standards in Higher Education – addressing the needs of course evaluation

“Students at the heart of the system” insisted the HE White Paper released earlier this year.  Sounds like a simple concept doesn’t it? But much like “eat five fruit and veg a day” or “love thy neighbour” – turning the concept into reality involves a lot of hard work and discipline.

How can we start to place students at the heart of the system?  The white paper outlined that universities will be more accountable to students on matters of teaching quality, and many in the academy had embraced this concept and are forging new two-way communication with their students to allow them to become stakeholders in the future of teaching. However this best practice is by no means uniform across the sector, and if we are being honest with ourselves some are still struggling with the nuances of closing the feedback loop.

In 2011 Electric Paper gathered the opinions of senior academics and student representatives on improving course evaluation practices in universities. This report offered unique insight into “the needs” of the sector in order to improve course evaluation.

The needs

The need to improve response rates:

Many universities seeking feedback on courses and lecturers via surveys are still struggling to achieve a meaningful response from students in order to evaluate teaching quality and make improvements while avoiding “survey fatigue”.

Coventry University has increased its response rates by using paper and moving to mid-module surveys. Professor Ian Marshall, the University’s Deputy Vice-Chancellor (Academic) said “We moved to online surveys, but the response was dreadful, so last year we introduced mid-module surveys and went back to paper. The response was super, and we are now able to turn around feedback in two weeks maximum.”

The need to improve student feedback:

Alex Bols, Head of Education and Quality at the National Union of Students, told us that all too often students who participate in course evaluation surveys are then not told what happens as a result of the process. Alex said: “It’s important for universities to close the loop and tell students what has happened – or hasn’t happened – as a result of the feedback provided and why. This should not be an autopsy at the end of a course, but a process embedded through the learning experience so that it is of benefit to the student giving the feedback and their experience.”

To make evaluation meaningful explore in-module evaluation. It takes time in terms of implementation and analysis, but the benefits are obvious.

The need to improve turnaround time:

Turnaround time is vital, but is hindered by process in many cases or open to human error. As such, feedback may come back when it is too late for the staff to do anything about it as they are, by that time, already committed to a teaching pattern.

Exploiting innovative new technologies could support the requirement to improve turnaround time, according to Professor Huw Morris, Pro-Vice-Chancellor (Academic) at the University of Salford, who in his previous role as Dean of Manchester Metropolitan University Business School led the trial of course surveys via mobile phones. “Going forward I anticipate that the Higher Education sector will need to utilise online devices to capture student feedback, but at the same time ensure that this is not done in an intrusive manner. Some element of compulsion for students in providing feedback will also be helpful in ensuring that the results are representative of underlying views.”

The need to improve survey administration:

Universities need to have a more consistent approach to survey administration, as the management of surveys sits centrally in one university, and departmentally in another. This results in a lack of core information across the institution, mixed responses and no consistent use of data.

To address this, City University London has introduced a centralised modular evaluation system with a standard set of questions for surveys, managed centrally, which individual schools can add to if they wish, and the results of these are now part of staff appraisals. Professor Alan Speight, Pro-Vice-Chancellor (Student Experience and Academic Quality Enhancement) at the University of Swansea, said: “There should be a unified approach that includes core questions and specifies the way the feedback is processed – which allows benchmarking and consistency. A common set of core questions should be owned institutionally, with subject areas able to select from a bank of optional additional questions.”

The need to improve the student experience

The increase in student tuition fees and the focus on student-led decision-making means that universities are under pressure to be more accountable and transparent on issues of quality.

Alex Nutt, Academic Affairs Officer at the University of Leicester Students’ Union, said “I think students will want to know that institutions take their concerns seriously, and that education is seen as a collaborative partnership between the University and the students – not just a business transaction.”

City University London’s Director of Learning Development, Professor Susannah Quinsee, added: “Universities do need to get students more involved in programme design, and evaluation and feedback is all part of that, but students also need to work with universities to tell us what data they find useful, what they expect, and above all what they find meaningful.”

Reflections on the Westminster Forum event ‘The Next Steps for Student Experience and Quality Assurance in Universities’

The Times Higher Education article today ‘Concentrate on overall experience, not jobs’ covers some of the discussions at the Westminster Forum session ‘The Next Steps for Student Experience and Quality Assurance in Universities’ held 13 October 2011.

http://www.timeshighereducation.co.uk/story.asp?storycode=417853#.TqFGkWeAcg4.twitter

Here are some of my additional comments on the event:

Other than the unfortunate comparisons between deciding on a university and the purchase of double glazed windows or a gym membership – the Westminster Forum debate and discussion around ‘The Next Steps for Student Experience and Quality Assurance in Universities’ was useful.

National Student Survey/Key Information Sets as the panacea…

The good – During the session it was clarified that there will be better online access to comparative data for families and students to make decisions. Much of this data will be made available from existing resources such as the National Student Survey and Key Information Sets provided by the university. Vice Chancellor Janet Beer rightly pointed out that the National Student Survey (NSS )is a useful resource in comparison to the alternatives such as the National Survey of Student Engagement (NSSE) in North America. Professor Beer’s strongest defence of the NSS centred on the fact that the NSS is compulsory for institutions, therefore representative and valid.

The bad – Unfortunately, neither the National Student Survey nor the Key Information Sets go deep enough which will lead institutions to search for more in an effort to identify excellence in pedagogic practice. As the NSS is only a year 3 survey administered at programme level – Universities will need to go further and focus on improving best practice in the way they implement module level feedback in order to identify, report and show improvement in the standards of excellence in teaching quality.

NUS as the arbitrar of quality…

In a recent live chat on the Guardian Higher Education Network, Matthew Barrow, vice-president education, Hull University Student Union (@wheelybarrow) asks a valid and relevant questions: Do you all agree that the Students’ Union is the best mechanism for providing the institution with high levels and good quality student engagement??? http://www.guardian.co.uk/higher-education-network/blog/2011/oct/11/student-engagement-live-chat#start-of-comments

My quick answer – No…and Yes. In order to respond properly – I move back to my commentary on the Forum:

At the Forum, Terry Hoad, president of the University and College Union expressed grave concerns about the publishing of student feedback as suggested in the HE whitepaper and now being implemented by HEFCE. I share Mr. Hoard’s concerns regarding open display of student feedback results at course level. Module evaluation feedback is inherently personal and useful to the academic being evaluated and ownership needs to reside with the institution and monitored by the executive. Institutions must however, quickly provide summary reports on feedback at module level including actions for improvement where applicable or as Her Majesty’s Minister David Willets pointed out at the Universities UK conference on Efficiency and Modernisation “Show students the data – or others will do it for you.”

Back to Mr Barrows question: Institutions should rightly look to their student union for direct engagement in creating policy and procedure around module feedback reporting. However the executive needs to retain ownership and control over module level feedback that is made public as this data has implications regarding staff development and the monitoring overall quality of learning and teaching for the institution. Executive control of this process cannot however act as an excuse for delay or the withholding of information. Institutions need to quickly turn around and publish ‘you said-we did’ reporting at module level that will assist students in decision making while they are still at the institution.

ratemyprofessor.com….

Social media is a reality that cannot be ignored. Institutions need to act as the ultimate trustworthy source in regards to their quality of teaching and learning. The best defence against unreasonable and damaging accusations in cyberspace is to put in place policies regarding transparency that include students as empowered interested stakeholders. Institutions need to show that they actively monitor and report on the quality of every module they provide and that they have systems in place to respond where there are issues in order to meet the fair and relevant expectations of their students.

Forecast for rain…

Sian Jones-Davies: As a Solicitor with Eversheds legal firm, Mrs. Jones-Davies brought forth a sobering legalistic view on the ‘student as consumer’ debate. Her point was that in legal terms – As the student is paying money, the relationship between a higher education institution (HEI) and its students is a contractual one (what is commonly called “the student contract”); and the second is that the student contract is a special type of contract, namely a consumer contract..” She goes on to state, again in legal terms, this means that institutions must deliver or “they could find themselves facing legal challenge from students for breach of contract or misrepresentation.” Of course the primary product a university offers is its courses. She goes on to say “What HEIs need to do in practice, therefore, is ensure they provide prospective students with accurate descriptions of what life will be like at that HEI, on that particular course.”

“HEIs will need to ensure that their student contracts comply with consumer legislation and that the provisions (the terms) of their student contracts are fair and reasonable from the student’s point of view.” When looking at the consumer argument from this vantage, it becomes even more clear that the executive must take an active role in monitoring quality of their product ‘the module’ and implementing policies around transparency and act as the primary source in providing accurate timely reporting to students.

The ‘student’ as an empowered highly interested stakeholder

The student as a consumer? The next steps for student experience and quality assurance in universities

Many lecturers – quite rightly proud of the standard of teaching offered by their institution – may baulk at the idea that obtaining a degree can be reduced to a mere business transaction. The label of ‘consumer’ is uncomfortable close to that of ‘customer’, and no-one wants higher education denigrated to the level of booking a holiday or choosing a phone contract. However, the inescapable truth is that – with the introduction and upcoming hike in tuition fees – students themselves are more conscious than ever before about the value of their degree.

To take the consumer analogy further, if you have decided to buy a new car and three are offered to you priced at £9,000, it is only reasonable that you are going to do as much as possible to find out every detail you can about the vehicles. Even the smallest negative report could sway the decision one way or the other. With the majority of universities electing to charge the maximum fee amount for each year of education, not to expect the same behaviour from prospective students would be ludicrous. Students see themselves as consumers even if some educators are not prepared to, as they are being asked to shoulder the cost of their degree.

However, there is a more palatable and practical solution. Rather than viewing the student as a consumer, universities should start to see each student as a stakeholder, helping to provide the vital feedback that will carry that institution forward and ensure that every module adds value to the degree they are investing their future in. A consumer can be forgotten about once the transaction is complete, but a stakeholder will be an advocate for life, working to enhance the reputation of the university as they feel they have had involvement in its development.

An important approach for building the student/university partnership is to ensure the student experience is interactive and immediate. Prompt ‘you said – we did’ feedback reporting that gathers opinions and ideas from students at regular intervals on a modular course is a simple and cost effective method. This can provide details on individual teaching on a module, but has the added advantage that, by capturing a core set of questions across the institution, the university gains visibility as to the quality of the teaching across departments, identifying excellence or areas of concern. The students that have provided that detail can then be kept informed of the actions that will be taken as a result – closing the feedback loop and enhancing the feeling of involvement. By bringing in student representatives as stakeholders in governance and quality, a degree of involvement that can lead to a mutually beneficial environment for education is nurtured.

Universities need to look at ways to benchmark internal processes in order to modernise and implement process improvements. There is much confusion and debate as to how this can be implemented, however there are real opportunities to bring about change and increase efficiency if institutions focus on transparency and promoting change from within, rather than obsessing about league table results and what other institutions are doing.

There are now more sources than ever before to build an image of an institution, and as universities begin to implement changes around Key Information Sets and the recommendations outlined in the recent White Paper, it is important that the executive retain access to course level data to communicate excellence of teaching in the marketplace.

In conclusion, a stakeholder can be defined on a continuum comparing their level of power and interest. Viewing higher education as a partnership and treating students as powerful highly interested stakeholders is a more productive, beneficial, and agreeable alternative for all parties, and relatively simple changes to current practice can begin to nurture this change in paradigm.

Why online ‘only’ course and module surveys don’t work

In 2011 Electric Paper conducted an extensive survey of institutions in Higher and Further education regarding Survey Best Practice. The survey results clearly indicate that many institutions are heavily invested in collecting student feedback for a diverse range of survey projects, yet there is serious concern among respondents regarding survey fatigue and declining response rates.  Interestingly, 3 out of 4 respondents indicated that both paper and online survey methodologies are a requirement at their institution.

Higher and Further education institutions have rightly targeted course and module survey programmes for process improvement as legacy paper based technologies contain many hidden costs and manual processes.  The logical assumption is that moving these paper based processes online will reduce the overall cost of ownership as well as make survey administration easier. Our research shows that the problem is not as easy to solve as it initially appears and that many institutions are still searching for a way to better automate course and module survey projects whilst maintaining valid and meaningful results. Based upon best practice survey results and feedback from our active customer base of over 600 education institutions worldwide, we propose that a hybrid approach is a requirement and that the best methodology for capturing student feedback has little to do with the paper vs. online debate – rather has everything to do with ‘in class’ vs. ‘out of class’ deployment of course surveys. As the real debate has been distorted by the advent of online only survey providers – I will identify the common misconceptions and further explain why the ‘online only’ approach has failed to live up to its promise.

Online only argumentation:

Online module surveys cost less
There are thousands of free online survey tools and many institutions have attempted to cut costs by subscribing to these free, or low cost online services. Setting data protection issues aside, the reality is that where savings are made in the software license, they are increased on the labour intensive administration created in trying to adapt these generic tools to fit the complexity of running module surveys across a diverse institution. Administrators must either create a unique survey project for each course or ask the participants to select the course they are on, making it more difficult for the student to complete. It is important to note that Online module surveys are conducted ‘out of class’ meaning that students are being asked to complete the survey on their own time, making it very difficult to capture valid response rates other than from students who are highly motivated based upon strong opinions that may, or may not, represent the views of the average student.

Online module surveys are easier
If the goal is to make survey administration easier, this is a valid claim on the surface. Running ‘in class’ paper based course and module feedback projects requires up front administrative effort in printing, distributing and scanning. If however, the goal is to gain statistically valid meaningful results as part of a strategic quality enhancement programme, the significant drop in ‘out of class’ response rates (+ 30%), does not justify the reduced administration effort. A more defensible approach would be to end the use of surveys as a quality enhancement tool for module surveys, rather than to waste administration effort in collecting and reporting on useless data.

Online module surveys are faster
This is an area of serious misconception. While it may be easier to deploy an online survey  – even using our product EvaSys, the administrator will need to leave the survey open for up to 5 weeks and set at least 2 reminders, in order to gain a reasonable response rate. However the same administrator, using paper surveys with EvaSys, can capture 100% of the student feedback from students ‘in class’ on the day, scan and deliver instant reports back to the academics within a matter of hours. Because many institutions have struggled with paper surveys in the past, with some still even keying in survey responses manually, a perception has evolved that paper is slow, when in fact just the opposite is true, with the right technology.

Online module surveys fit neatly in with environmental policy
Our experience is that there is a place for online surveys, such as for student satisfaction, alumni, non taught post grad students etc. These are all areas where either there is no way to administer a survey ‘in class’, or where a low response rate is still valid and representative of the overall student population. However, if we agree that ‘in class’ is the best way to gain valid and reliable feedback in order to monitor quality of learning and teaching, then there only two options: Firstly, handing out paper surveys. Secondly, make computers available ‘in class’ for every student to complete the module survey. Our contention is that with a hybrid approach, institutions can use online where it makes sense, thereby reducing the reliance on paper and deploy paper surveys only where necessary. We recommend working with suppliers that use recycled paper, or paper made from sustainable forests, as well as recycling the surveys after completion. Of course ‘in class’ might become a reality in the future with tablet and smart phone technology, which is why we continue to innovate in this area.

Online module surveys can identify how a specific individual has responded
While we do have customers who have crafted data protection statements indicating that the module survey is ‘not anonymous however treated as confidential’, this is not the norm. There is evidence from student focus groups that students are concerned that honest feedback might have a negative effect on their markings, if they were to be identified. Our recommendation is to limit the use of the ‘tie back’ approach to specific institutional research projects and build trust through the use of technology that protects the student’s anonymity. To that end, paper surveys should be deployed and collected by student reps in sealed envelopes, without the involvement of the academics.

Online module surveys are location independent
Again, this is why we recommend a hybrid approach. With the advent of online only technology, many institutions have lost the ability to monitor how many surveys students are being asked to complete. With our hybrid approach, a single system is used for paper based ‘in class’ and for online ‘out of class’ surveys such as offshore campuses, grad students, students not on campus, alumni and online-LMS courses.

Online surveys produce richer comments
We are in agreement that where a high response rate is not required in order to make educated assumptions from the results, such as student satisfaction or facilities surveys, this statement is valid. However, when looking specifically at the course or module, the goal is to capture not only feedback from students with strong opinions, but rather a voice of the average student. It is not possible to draw conclusions as to the quality of the teaching and the course from only 2 students from a class of 20, regardless of the richness of their comments. There are better ways to gain qualitative feedback such as running focus groups on areas identified for low performance through the module survey process.

Online is just better
Electric Paper has over 15 years assisted organisations in applying the right technology, in the right setting, in order to automate paper intensive data capture processes. In this time we have in certain scenarios recommended paper as a capture device over an online approach because of the ease of use as a capture device. Based upon our specific experience in working with course and module surveys, we recommend hybrid vs. online only in order to facilitate the ‘in class’ capture of student feedback. The hybrid paper and online approach gains the highest possible response rates in real-time, making it possible to turn around results faster, in order to close the loop and involve students as stakeholders in improving the quality of the course. It is our opinion based upon feedback from over 600 educational institutions and the responses to our 2011 Best Practice Survey, that institutions who have implemented online course feedback systems have experienced an unacceptable drop in response rates, which renders much of the effort in administration a waste of time and resources. We however, do see online surveys as a valuable tool in certain applications such as institutional research or where it is not possible to capture feedback ‘in class’ which is why we continue to innovate in this area and lead the market in hybrid paper and online solutions.

ELECTRIC PAPER

Electric Paper works with universities to capture student feedback that will help to improve the future design and delivery of their courses. We work with more than 600 education institutions worldwide through web-based data capture and student survey management solutions to drive efficiency in capturing the student voice at course level. Our flagship product EvaSys Education Suite™ works to automate course and module evaluation and reporting, thereby saving staff time and costs, as well as generating cross-institution common best practice in assessing student feedback about their teaching and learning experience. In light of the need to place greater emphasis on the “student experience” in UK higher education, our role is increasingly significant. We also deliver surveys of alumni, employees and other members of the education community. For more information: www.electricpaper.co.uk

4 Ways to Avoid Survey Fatigue in Higher Education

4 Ways to Avoid Survey Fatigue in Higher Education

Students are asked to complete a lot of surveys. This has led to concern in most HE institutions regarding the validity and necessity of the many survey projects originating from different parts of the organisation throughout the academic year. It is not uncommon in Higher Education for students to be asked to complete between 6-10 module surveys along with other programme and organisational surveys, all in addition to the NSS. From our experience it’s not necessarily that there are too many surveys, rather that there are too many poorly organised surveys that lead to survey fatigue. For this reason, it is important to develop a strategy to manage this problem in order to improve the effectiveness of strategic survey initiatives. For organisations managing surveys, I have highlighted some key points to consider below.

1 – Know what surveys are being run at all times.

With the advent of free online surveys it is far too easy for anyone who has the desire to simply log on and create a survey. As surveys are driven by many different stakeholders within an organisation, these survey projects can originate from central administration and quality, institutional research and planning, marketing, department heads as well as the academics themselves. Without some centralised visibility to account for what surveys are being generated, there is a real possibility that students are being asked to complete far too many surveys leading to what has been described as survey fatigue. To solve this problem it makes sense to protect your students as a valuable knowledge resource and implement a common survey platform with controlled administrative access to manage and track all survey activity. With free online survey tools, it is easy to deploy a survey. Just because it is easy does not necessarily mean it is a good idea. In fact, it may negatively impact other key initiatives that matter more such as feedback on teaching and learning at course level.

2- Include students as stakeholders.

Student organisations have made it clear that there needs to be mechanisms in place to capture the student voice regarding teaching and learning and overall satisfaction on every course and module. This is intensified by the increase in fees and governmental focus on empowering student choice as a key driver for quality and accountability. These factors need to be taken into consideration when communicating the importance of feedback initiatives. As many organisations have found, this can be achieved by highlighting and marketing the efforts being made by the executive to address the student experience. This process can be made even more transparent by making a concerted effort to include students as stakeholders in the survey reporting process with the expectation that they too actively participate in directing the future of the institution through their feedback.

3- Implement real-time feedback.

One of the biggest factors of survey fatigue is the fact that students are constantly being asked to complete surveys and they never see the results or evidence of their feedback. The main reason for this breakdown is the complexity in capturing and reporting on survey feedback as many organisations are using antiquated technology or relying on manual processes for capturing and reporting. By implementing advanced real-time technology, institutions are able to gain high response rates and turn the results around quickly. The key is to complete the feedback loop and implement a programme of communicating the outcomes back to the students in an agreed upon strategy for posting results.

4- Communicating the results.

A good ‘best practice’ strategy many organisations have implemented is “You said – we did” communication. This is a vital step in the quality assurance process that leads to higher response rates and reduces survey fatigue. This is because students then understand the importance of completing the surveys they are being asked to participate in. Students are used to completing assignments they are given as long as they understand why and how it will be of benefit to them. While it is important to communicate up front the important role they have in shaping their student experience, it is vital to also provide evidence of follow up action planning based upon the results to help students better understand that they are active stakeholders in the process. By completing the feedback loop students are therefore more likely to actively participate adding validity to the entire survey process.

ELECTRIC PAPER

Electric Paper works with universities to capture student feedback that will help to improve the future design and delivery of their courses. We work with more than 600 education institutions worldwide through web-based data capture and student survey management solutions to drive efficiency in capturing the student voice at course level. Our flagship product EvaSys Education Suite™ works to automate course and module evaluation and reporting, thereby saving staff time and costs, as well as generating cross-institution common best practice in assessing student feedback about their teaching and learning experience. In light of the need to place greater emphasis on the “student experience” in UK higher education, our role is increasingly significant. We also deliver surveys of alumni, employees and other members of the education community. For more information: www.electricpaper.co.uk

5 Key Claims to Watch Out For When Selecting a Survey Product for Automating Course Evaluations

5 Key Claims to Watch Out For When Selecting a Survey Product for Automating Course Evaluations

Many organisations struggle in choosing the best methodology for running course evaluations and other survey projects necessary in driving quality enhancement and strategic planning efforts. In fact there is no easy answer or quick fix. Many organisations choose to go with paper or online methodologies based upon the recommendation of another institution or influential claims by a vendor.  In fact there is no one size fits all method; rather the decision should be based upon the specific circumstances unique to each institution. Claims by manufacturers can be especially dubious as they are heavily influenced by their particular product focus and market experience.  Vendor claims to watch out for can be broken into five main categories.

Claim # 1 – Generic hosted online survey tools can handle course evaluations.

Hosted survey services offer generic survey authoring and publishing tools with basic .csv or statistical output.  A typical response rate from online survey is between 15-30% and extensive additional work is required in order to create aggregate reporting for curriculum managers and senior department heads and executive stakeholders. The biggest challenge however with these offerings is in providing instructor level feedback as individual instructor reports need to be created one at a time.

Claim #2 – Paper only data capture is easy to use and reduces the workload.

Again these products are generic applications designed to be run centrally by a designated administrator. Paper data collection offerings claim to reduce data entry by up to 90% however, these products are characterised by highly complex user interfaces and the claims for reduced data entry do not address the many other manual constraints inherent in distributing surveys. For example, to individual instructors that may teach against multiple courses or participate in lecturing teams with other instructors.  These paper data capture products typically output simple .csv or ascii data and do not attempt to address the highly complex reporting requirements within higher and further education.

Claim #3 – Hybrid paper and online survey offerings make survey administration easy.

Several software manufactures have tried to address the need for paper and online requirements of organisations by attempting to add-on functions after the fact or through business alliance partnership.  Key phrases to be aware of from these vendors are “seamless integration” or “connect agents” The point here is that paper and online surveys are run as separate processes with separate underlying data structures and then combined at some point after the fact.  The end result is a highly complex process for the administration staff that must be managed centrally.

Claim #4 – Add on VLE (Virtual Learning Environment) surveys makes course evaluation easier.

One only needs to look at the marketplace to see that it is not a huge undertaking to develop an online survey tool.  Generic online survey tools are introduced to the market seemingly on a daily basis  and typically named after a tropical animal or fruit.  Clearly these tools are not designed to address the challenges of course evaluation automation or module level analysis and reporting in higher and further education.  Confusion arises as to the viability of an online survey tool when offered as an upgrade module from a VLE or LMS vendor or possibly from the creator of the core student information system. Surely these hosted survey tools are less generic and better designed to meet the real requirements of higher and further education?  Unfortunately the issue with the add-on approach is that the survey plug in is not the core innovation focus of these products. The effort that goes into the development of a (VLE) virtual learning environment or a (SIS) student information system is extensive and the team or developer designated to create the add-on survey module will play a smaller part in the overall product release effort.  This is concerning for organisations that need  or are waiting on additional functionality or are experiencing technical issues as these vendors will not have the ability or focus in order to respond effectively. Lastly, there is a high likelihood that add-on VLE/SIS survey tools were added on through acquisition of one of the many online tools popping up in the market and will, like Hybrid offerings, falsely offer “seamless integration” or “connect agents.” These tools will in fact be based upon a separate underlying data structure with limited generic statistical output and reporting.

Claim #5 – Outsourcing course evaluation will save time and money.

As a last resort many organisations give up on administering core survey processes in-house and choose to outsource to an outside provider.  This decision is understandable for certain targeted qualitative market research initiatives such as focus groups as there are vendors that have developed specific talent and specialised skills in the area of market research.  The choice to outsource quantitative course evaluation initiatives to an outside vendor however is both very expensive and problematic.  This survey approach creates significant delays in gaining access to aggregate reporting limiting the viability and effectiveness of the outcomes. In addition, much of the value gained from the survey process is in the administration and choices made during the analysis process.  This knowledge regarding the data is then lost to the organisation and held by the outside vendor.

ELECTRIC PAPER

Electric Paper works with universities to capture student feedback that will help to improve the future design and delivery of their courses. We work with more than 600 education institutions worldwide through web-based data capture and student survey management solutions to drive efficiency in capturing the student voice at course level. Our flagship product EvaSys Education Suite™ works to automate course and module evaluation and reporting, thereby saving staff time and costs, as well as generating cross-institution common best practice in assessing student feedback about their teaching and learning experience. In light of the need to place greater emphasis on the “student experience” in UK higher education, our role is increasingly significant. We also deliver surveys of alumni, employees and other members of the education community. For more information: www.electricpaper.co.uk