in CSGlobal Perceptions on the Use of

WeBWorK as an Online Tutor forComputer Science

Olly Gotel, Christelle Scharff, Andrew Wildenberg,Mamadou Bousso, Chim Bunthoeurn, Phal Des,

Vidya Kulkarni, Srisupa Palakvangsa Na Ayudhya,Cheikh Sarr, Thanwadee Sunetnanta

Presented by: Olly Gotel (ogotel@pace.edu)Pace University, New York, USA

[FIE 2008]

in CSOutline

• The Problem with CS and TeachingProgramming … A Personal Perspective …

• Systems for Automated Assessment ofProgramming Assignments

• WeBWorK and WeBWorK-JAG• A Global Experiment - PERCEPTIONS• Conclusions

in CS

Systems for Automated Assessment ofProgramming Assignments

• Web-based systems to encourage practice(with feedback), and improve and reinforcestudents’ understanding of concepts

• Types of questions– True / false, short answer, multiple-choice,

programming• Grading programs

– Correctness + quality + authenticity

in CSSome Existing Systems

• BOSS (http://www.dcs.warwick.ac.uk/boss, Open Source, Computer Science)

• CodeLab (http://www.turingscraft.com, Commercial, Computer Science)

• CourseMarker (http://www.cs.nott.ac.uk/coursemarker, Commercial, Computer Science)

• DevSquare (http://www.devsquare.com, Commercial, Computer Science)

• Educosoft (https://www.educosoft.com/ecf, Commercial, Mathematics)

• Gradiance (http://www.gradiance.com, Commercial, Computer Science)

• JavaBat (http://javabat.com, Free, Computer Science)

• MathXL (http://www.mathxl.com, Commercial, Mathematics)• MyCodeMate (http://www.mycodemate.com, Commercial , Computer Science)

• MyMathTest (http://www.mymathtest.com, Commercial, Mathematics)

• OWL (http://owl.course.com, Commercial , Computer Science)

• Quiz PACK (http://www.sis.pitt.edu/~taler/QuizPACK.html, Open Source, Computer Science)

• Viope (http://www.viope.com, Commercial, Computer Science)

• WebAssign (http://www.webassign.net, Commercial, Mathematics)

• WebCAT (http://web-cat.cs.vt.edu, Open Source, Computer Science)

• WeBWorK (http://webwork.rochester.edu and http://atlantis.seidenberg.pace.edu/~scharff/webwork, Open Source,Mathematics and Computer Science)

• WileyPLUS (http://he-cda.wiley.com/WileyCDA/Section.rdr?id=101003, Commercial, Mathematics)

• ASSYST, CFX, CodeWitz, Marmoset, Praktomat, RoboProf, SQL-Tutor, TRAKLA2…

Not an exhaustive list!

in CS

WeBWorK: About

• http://webwork.rochester.edu• Project funded by NSF• Free, open-source and web-based• Automated problem delivery and grading• Initial development and applications in the fields

of mathematics and physics• Currently in use at more than 60 colleges and

universities (for maths)

in CS

WeBWorK: Underneath

• Problems are written in the Problem Generatingmacro language (PG) ‏– Text, HTML, Latex, Perl

• Underlying engine dedicated to dealing withmathematical formulae– x+1 = (x^2-1)/(x-1) = x+sin(x)^2+cos(x)^2

• Individualized and parameterized versions ofproblems

in CS

WeBWorK: for ProgrammingFundamentals

• Extension of WeBWorK for use in programmingfundamentals - Pace and Cornell (since 2005)

• True / false, short answer and multiple choice problems forJava, Python and SML

• Evaluation of Java program fragments in real time byinterfacing WeBWorK with JUnit [www.junit.org]– WeBWorK-JAG = WeBWorK + Java Auto-Grader

• http://atlantis.seidenberg.pace.edu/webwork2

in CS

A Global Experiment• Global Perceptions on the Use of WeBWorK as an Online

Tutor for Computer Science– Students and Professors– Perceived value and use (class size, integration)– Implications for uptake (US-centric)

• 5 Country Experiment– 10 Java/C problems (common questions)– Time-boxed, unlimited attempts

• Questionnaires– Students: Background info, technology access, time spent coding per

week, opinions of use, preference for question types, etc– Professors: Background info, teaching style, class sizes, homework,

time spent on tasks, contact hours, technology use in classroom, valueof question types, etc

in CS10 Cornell College8 University of Delhi8 Mahidol University

10 Royal Universityof Phnom Penh

8 University of Thies

in CS

Problem Sets

in CS

Problems

in CS

Multiple-Choice Questions

in CS

Fill-in-the-Blanks Questions

in CSYou Can Build up Interesting /

Typical Questions

in CS

Randomize / Parameterize

in CS

JAG - For Open Coding

in CS

Typical Feedback - Correct

in CS

Typical Feedback - Incorrect

in CS

How Did They Do?

in CS

… & With Respect to the Others?

in CS

Results

• Not designed to compare abilities!• Very few students got all the questions

correct• Average number of attempts per question

was 1 to 5 (except US where > 20)– Prior exposure– Driven to achieve 100%

• Background data• Logistics - Internet

in CS

Student Perceptions - Qs

• Like best? Instant feedback• Like least? Clarity of questions/language• Suggestions? Live help-desk links• Most valuable type of questions? Open coding

(MC in Senegal)• Easiest type of question to answer are least

valuable• Why use? Practice• Use again? 72.5%

in CS

Student Perceptions - Free-Form• “WebWorK is good but needs the user to have the Internet.

So if we don't have internet we can't do it.” [Cambodia.]• “Though it was my first experience at WeBWorK, I found it

quite simple to use and really encouraging to remind meabout Java, as I haven't did any java practice for the last 3-4months. Its interface was nice.” [India.]

• “I liked the concept of WeBWorK. It is a useful tool fortesting students’ level at programming.” [Senegal.]

• “WeBWorK make it convenient to do the test byautomatically checking the answer and responding back tothe user.” [Thai.]

• “It was a fun experience to go through the questions and findthe answers. I enjoyed reviewing, and verifying that I knowhow to write code.” [US.]

Implications? For uptake and future work?

in CS

Professor Perceptions - Qs

• All seek help/relief from compiling/markinghomework

• Prepared to invest time to learn -- howmuch?

• Like most? Instant feedback to students solearn from mistakes

• Most valuable question type? Open coding

in CS

Professor Perceptions - Free-Form• “I think the problem should be clear - should [be]

basic Java and OOP concept in Java. Internetwas too slow to have all students complete theexercises.” [Cambodia.]

• “I will be happy to see how it will test a largeprogramming assignment with multiple classes.”[India.]

• “Internet use which expands collaborations.”[Senegal.]

• “…limited programming library; expand problemlibrary…” [US.]

Implications? For uptake and future work?

in CS

Conclusions• Supports an overloaded professor and can tide students

over in-between classes• These systems open up opportunities for professors and

students around the globe to transfer and shareknowledge, and extend practice, but some localadaptations are necessary (Intranet, questionformulation, integration)

• Development of a novel pedagogy, encouragingstudents to contribute their own questions to theWeBWorK library, introducing them to crucialpractices of software engineering

• Aim to create a community of contributors to monitorquality, share work and extend the WeBWorK library

in CS

Acknowledgements

• NSF CCLI AI Grants “Collaborative Research: Adaptingand Extending WeBWorK for Use in the Computer ScienceCurriculum” #0511385 and #0511391

• Students:– The many computer science students in the US, Cambodia, India,

Senegal and Thailand– Jacqueline Baldwin, Nathan Baur (JUnit extension)– Sophal Chiv (input of problems and help desk)– Eileen Crupi, Tabitha Estrellado (input of problems)– Allyson Ortiz, Veronica Portas (existing systems)– Yue Ma (user acceptance testing)