Back To Schedule
Friday, November 21 • 3:45pm - 4:15pm
Crowd sourcing of assessments in a developing country

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

The proposal builds upon an exploratory study, conducted in 2010, to determine whether teachers in a developing country were able to create good quality multiple-choice questions for primary school students. The study also developed and evaluated an adoption model to ascertain if the teachers would actually contribute to such a Wiki. 22 teachers from public and private schools situated in different parts of Pakistan (urban and rural) participated in the project. The exercise ran for about a month at the end of which a survey was conducted to get teachers' feedback. The survey was designed to get feedback on five adoption factors: usefulness, ease of use, enjoyment, intention to adopt, and attitude - and five quality factors: correctness, reliability, efficiency, usability and verifiability. The teachers were asked to rate each adoption factor on a Likert 1 to 5 scale, and two expert judges (with M.Ed. degrees and at least 15 years of relevant experience) were asked to rate the quality factors on the same scale. The study found that, given student learning outcomes, content constraints, and a Bloom's assessment level, a reasonable number of teachers were able to formulate quality questions, and that there was a strong intention to use such a system. Teachers with "high intention" to adopt also had a better attitude, enjoyed making questions and found the process easy to use. However, there was no obvious relationship between "intention to use" and "ability to pose good assessments". In addition, there was no obvious predictor of where the good question contributors came from.

The results of the experimental study were used to implement a small-scale prototype project of "crowd-sourcing based assessment collection system", involving approximately 100 teachers from public and private schools from Islamabad and Peshawar. A survey tool was developed to get feedback from participating teachers. Formal interviews were also conducted from two sets of sample teachers - those who actively contributed to the system, and those who did not contribute at all. The interviews were based on motivational factors (enthusiasm and persistence) as well as social parameters (altruism and empathy); the objective was to ascertain the factors responsible for motivating or de-motivating users. The survey found that the success of the system depended on such adoption factors as the availability of infrastructure and the quality of teacher training. Evidence was also found of the common crowd sourcing problems of long tails, since most of the questions contributed to a limited set of topics.

Based on the feedback from stakeholders and adoption pattern of teachers, a follow-up medium scale adoption model was proposed successfully to the Ministry of IT, Government of Pakistan under the ICT R&D fund. The project kicked off in July 2013 and has since developed a crowd-sourcing platform and assessment engine, which generates exams based on different difficulty levels. The next stage in this project is the start of the deployment phase; it will introduce teachers to the concept and train them on how to use the system. The project plans to target approximately 1700 teachers from public and private schools across Pakistan. An adoption period of six months has been set for users to use the system. A mid-term analysis will provide an opportunity for error correction. A final assessment will be conducted at the end of 2014 in order to analyze the findings from the deployment and propose a full-scale adoption model to crowd source assessments. I hope to bring the results to share at the Conference.

avatar for Luna Banuri

Luna Banuri

Executive Director, Utah Muslim Civic League

Friday November 21, 2014 3:45pm - 4:15pm EST
Virginia Ballroom

Attendees (0)