A Means A, a bit more fleshed out
By Arnold Kling
Pasted in below the fold. Comments welcome. A Means A
A Reliable Independent Grading Service for College Level Course Work
Version as of August 5, 2011
A Means A solves the problem of credibility and comparability of grades in courses taught at different institutions of higher education. The innovation is to separate the grading process from other aspects of higher education. For any college-level course, A Means A will devise an appropriate exam and use independent professionals to grade the exam, according to transparent, standard criteria.
Today, what does a good grade mean? Did the student really attain the course objectives? Or was the professor a pushover? Or were the institutional standards low? We envision a grading system that is more robust, so that everyone who reads a transcript knows what the grades mean. A Means A. The potential benefits of this are discussed in the section below called Business Opportunity.
One existing service that illustrates the level of credibility and standardization that we seek is the Advanced Placement testing undertaken by the College Board. AP tests include both multiple-choice sections, which are graded by machines, and free-response sections, which are graded by trained teachers under clear guidelines. The College Board is transparent about what sorts of student responses merit the highest score (5) or any of the other scores. A high school student can use an AP test to demonstrate to a college that the student has attained the objective of the course, regardless of the reputation of the student’s high school.
A Means A will extend the reliable, independent grading model of the AP exam to a broad spectrum of college-level courses. However, while the AP program compels instructors to “teach to the test,” A Means A will “test to what you teach.” That is, A Means A will take course objectives as given by instructors. It will design and grade tests that align with the objectives of the course. The reasons for taking this approach are discussed in the section below called Test To What You Teach.
The organizational roles for this business include CEO, head of marketing, and manager of the grading service. I see myself as best suited to manage the grading service while the business is in its experimental and development phase. Once the business gets going, I would probably fade into an advisory role.
A reliable, independent grading service can earn revenue from educational innovators and from existing institutions. For innovators, the service can provide validation and certification. For existing institutions, it can help with performance management and, in some cases, with efficiency.
Educational innovators face barriers to acceptance. With modern communication technology, it should be possible to come up with alternatives to the traditional classroom and lecture approach to education. However, consumers are reluctant to adopt innovations that have yet to establish credibility. We want innovators to be able to say, “Our instruction methods have been shown to work, as demonstrated by student performance on tests administered by A Means A, the reliable, independent grading service.”
For existing colleges, A Means A could help with performance management. As it stands now, college administrators lack data on how well instructors are performing in the classroom. By obtaining results from exams developed and graded by an independent service, administrators can open a window into what is going on in various classrooms.
For some courses that are large and well standardized across universities, A Means A could offer efficiency based on scale economies. The cost of developing and grading an exam for organic chemistry or intermediate microeconomics could be reduced by having a specialize grading service undertake the effort rather than using the resources of every college that offers such courses.
The ability for A Means A to charge a mark-up over its costs will depend on becoming an industry standard. In the best case scenario, employers will demand to see A Means A grades on transcripts, colleges will use A Means A to determine whether or not to accept credit for transfers, graduate and professional schools will use A Means A in admissions decisions, and consumers will use A Means A results to evaluate alternative educational institutions. Educational researchers will frequently say “According to data from A Means A, …” The higher education market is so large that if a business becomes a standard essential component, the profit opportunity should be rather significant.
If A Means A falls far short of becoming an industry standard, then the costs of providing the service are likely to prove too high relative to the prices that it can charge. The College Board charges over $80 per student for an AP exam. A Means A is likely to cost much more, particularly for non-standard courses. Even though outside tests could substitute for internal testing, colleges are likely to see independent testing as an “add-on” cost. They will only pay if there is strong pressure from external constituents.
Of course, attaining the status of an industry standard represents a formidable marketing challenge. The team will have to possess outstanding skills at marketing strategy and execution.
Test to What You Teach
Centralized, standardized tests are not the best approach for a college environment. With lower-level courses, professors value their autonomy to emphasize certain units rather than others. Upper-level courses are often “custom designed.” If anything, college education is likely to become more dynamic going forward, as educators experiment with new technology to deliver content. A Means A will want to have the flexibility to adapt to diversity in content and teaching methods, not become a hindrance to creativity and innovation.
The practicality of “test to what you teach” as a model for independent grading has been demonstrated by the Swarthmore College Honors program. The courses are seminars, taken by juniors and seniors in their major and minor subjects. The professors who teach the courses have control over the curriculum. The college hires outside examiners who write exams, based on syllabus material supplied by the instructors. The college administers the exams, and the outside examiners grade them. In the Swarthmore program, the exams are all free-response, without any machine-graded component. They also include an oral component. For A Means A, the mix of machine-graded,, free-response, and oral examinations is yet to be determined.
With “test to what you teach,” educators will send A Means A course materials, including a syllabus, a list of course objectives, and examples of the types of questions students are expected to answer. A Means A will find a qualified professional to design an exam and to provide input into grading criteria. Some criteria, for example for grammar and organization of written work, will be standard across all A Means A tests. Other criteria will be specific to course content.
One potential drawback to “test to what you teach” is that one professor teaching, say, organic chemistry, might have course objectives that differ dramatically from those of another professor. A Means A will have to develop reporting methods that make it as easy as possible for users to understand these differences. For example, in organic chemistry, a committee might agree on a standard set of objectives. Any given professor will have the option of having students take the test that is based on the standard objectives or of taking a custom test. A Means A can note when a grade is based on a non-standard test and also include an explanation of how the test differed from the standard.