essay tests have low reliability because

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

Validity and Reliability: Issues In the Direct Assessment of Writing

1992, WPA: Writing Program Administration

Related Papers

James McLean

Educational and Psychological Measurement

Jim Montgomery

Issues surrounding the psychometric properties of writing assessments have received ongoing attention. However, the reliability estimates of scores derived from various holistic and analytical scoring strategies reported in the literature have relied on classical test theory (CT), which accounts for only a single source of variance within a given analysis. Generalizability theory (GT) is a more powerful and flexible strategy that allows for the simultaneous estimation of multiple sources of error variance to estimate the reliability of test scores. Using GT, two studies were conducted to investigate the impact of the number of raters and the type of decision (relative vs. absolute) on the reliability of writing scores. The results of both studies indicated that the reliability coefficients for writing scores decline as (a) the number of raters is reduced and (b) when absolute decisions rather than relative decisions are made.

Journal of English Language Studies

elda selja putri

George Engelhard

Early Holistic Scoring of Writing: A Theory, A History, a Reflection

Norbert Elliot

What is the most fair and efficient way to assess the writing performance of students? Although the question gained importance during the US educational accountability movement of the 1980s and 1990s, the issue had preoccupied international language experts and evaluators long before. One answer to the question, the assessment method known as holistic scoring, is central to understanding writing in academic settings. Early Holistic Scoring of Writing addresses the history of holistic essay assessment in the United Kingdom and the United States from the mid-1930s to the mid-1980s—and newly conceptualizes holistic scoring by philosophically and reflectively reinterpreting the genre’s origin, development, and significance. The book chronicles holistic scoring from its initial origin in the United Kingdom to the beginning of its heyday in the United States. Chapters cover little-known history, from the holistic scoring of school certificate examination essays written by Blitz evacuee children in Devon during WWII to teacher adaptations of holistic scoring in California schools during the 1970s. Chapters detail the complications, challenges, and successes of holistic scoring from British high-stakes admissions examinations to foundational pedagogical research by Bay Area Writing Project scholars. The book concludes with lessons learned, providing a guide for continued efforts to assess student writing through evidence models. Exploring the possibility of actionable history, Early Holistic Scoring of Writing reconceptualizes writing assessment. Here is a new history that retells the origins of our present body of knowledge in writing studies.

George Engelhard , Stefanie Wind

Advances in health sciences education : theory and practice

John Boulet

The use of experts to judge performance assessments is desirable because ratings of performances, carried out by experts in the content domain of the examination, are often considered to be the "gold standard." However, one drawback of using experts to rate performances is the high cost involved. A more economic alternative for scoring performance assessments entails using analytic scoring, which typically involves assigning points to individual traits present in the performance, and summing to arrive at a single score. This strategy is less costly, but may lack the richness of holistic scoring. This study investigates the use of regression-based techniques to predict expert judgments on a written performance task from a combination of analytic scores. Potentially, this will result in scores that approximate the richness of holistic ratings while maintaining the cost-effectiveness of analytic scoring. Results show that a substantial proportion of variance in expert judgmen...

Journal of Writing Assessment

Vicki Hester

Early Holistic Scoring of Writing: A Theory, A History, A Reflection

Richard Haswell

What is the most fair and efficient way to assess the writing performance of students? Although the question gained importance during the educational accountability movement of the 1980s and 1990s, the issue had preoccupied language experts and evaluators long before. One answer, holistic scoring, is central to knowledge about writing and the evaluation of writing in academic settings. Indeed, one could have walked into any school or college in the US in 1987, used the term “holistic scoring” to an English teacher, and be pretty sure that the listener would know the meaning of that term—and, perhaps, a great deal more. Early Holistic Scoring of Writing covers holistic essay assessment, in the UK and the US, from the 1930s to the mid 1980s. The volume chronicles this contentious scoring method from its first spread and decline in the UK to the beginning of its heyday in the US. Much of the history will be unfamiliar to readers. For instance chapters cover holistic scoring of 11+ exam...

Journal of writing assessment

RELATED PAPERS

Pollarise GROUPS

European Journal of Anaesthesiology

Alparslan Kuş

Revista De Ciencias Medicas E Biologicas

Neci matos soares

Khasanah Ilmu - Jurnal Pariwisata Dan Budaya

nurul lestari

Eeden, P. van den, Terwel, J., & Mooij, T. (1993). Gruppenzusammensetzung und Interaktion bei kooperativem Lernen in Mathematik. In G.L. Huber (Hrsg.), Neue Perspektiven der Kooperation. Grundlagen der Schulpädagogik, Band 6 (pp. 216-232). Hohengehren: Schneider.

ainun awaliah

Revista de Patologia Tropical

Estevão Figueiredo

albuquerque: revista de história

Murilo Meihy

Kwame Annin

Development

Jose Norberto Lopez Trejo

Advanced Powder Technology

ashraf mubarak

International Journal of Plant & Soil Science

NIRMAL MANDAL

Sustainability

Onur Alper Tekin

International Journal of Recycling of Organic Waste in Agriculture

James k phiri

Journal of Biological Chemistry

Muralidharan Anbalagan

Journal of Neurology, Neurosurgery & Psychiatry

John Pollard

Biomass and Bioenergy

Nehru Chevanan

Journal of Thoracic Oncology

kate burbury

Wondimu Lambamo

Nils Svendsen

Materials Science and Engineering: A

Bernd Baufeld

yyjugf hfgerfd

Frontiers in Rehabilitation Sciences

Imelda Coyne

Alimentary pharmacology & therapeutics

Robert JM Brummer

F1000 - Post-publication peer review of the biomedical literature

Jeff Hardin

See More Documents Like This

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

Models for Uncertainty in Educational Testing pp 17–46 Cite as

Reliability of essay rating

• Nicholas T. Longford 2  

238 Accesses

1 Citations

Part of the book series: Springer Series in Statistics ((SSS))

Standardized educational tests have until recently been associated almost exclusively with multiple-choice items. In such a test, the examinee is presented items comprising a reading passage stating the question or describing the problem to be solved, followed by a set of response options. One of the options is the correct answer; the others are incorrect. The examinee’s task is to identify the correct response. With such an item format, examinees can be given a large number of items in a relatively short time, say, 40 items in a half-hour test. Scoring the test, that is, recording the correctness of each response, can be done reliably by machines at a moderate cost. A serious criticism of this item format is the limited variety of items that can be administered and that certain aspects of skills and abilities cannot be tested by such items. Many items can be solved more effectively by eliminating all the incorrect response options than by deriving the correct response directly. Certainly, problems that can be formulated as multiple-choice items are much rarer in real life; in many respects, it would be preferable to use items with realistic problems that require the examinees to construct their responses.

• Mean Square Error
• Severity Variance
• Advance Placement
• Rater Grade

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, log in via an institution .

Buying options

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Unable to display preview.  Download preview PDF.

Author information

Authors and affiliations.

Research Division, Educational Testing Service, 15-T, Rosedale Road, Princeton, NJ, 08541, USA

Nicholas T. Longford

You can also search for this author in PubMed   Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 1995 Springer-Verlag New York, Inc.

About this chapter

Cite this chapter.

Longford, N.T. (1995). Reliability of essay rating. In: Models for Uncertainty in Educational Testing. Springer Series in Statistics. Springer, New York, NY. https://doi.org/10.1007/978-1-4613-8463-2_2

Download citation

DOI : https://doi.org/10.1007/978-1-4613-8463-2_2

Publisher Name : Springer, New York, NY

Print ISBN : 978-1-4613-8465-6

Online ISBN : 978-1-4613-8463-2

eBook Packages : Springer Book Archive

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Creating and Scoring Essay Tests

FatCamera / Getty Images

• Tips & Strategies
• An Introduction to Teaching
• Policies & Discipline
• Community Involvement
• School Administration
• Technology in the Classroom
• Teaching Adult Learners
• Issues In Education
• Teaching Resources
• Becoming A Teacher
• Assessments & Tests
• Elementary Education
• Secondary Education
• Special Education
• Homeschooling
• M.Ed., Curriculum and Instruction, University of Florida
• B.A., History, University of Florida

Essay tests are useful for teachers when they want students to select, organize, analyze, synthesize, and/or evaluate information. In other words, they rely on the upper levels of Bloom's Taxonomy . There are two types of essay questions: restricted and extended response.

• Restricted Response - These essay questions limit what the student will discuss in the essay based on the wording of the question. For example, "State the main differences between John Adams' and Thomas Jefferson's beliefs about federalism," is a restricted response. What the student is to write about has been expressed to them within the question.
• Extended Response - These allow students to select what they wish to include in order to answer the question. For example, "In Of Mice and Men , was George's killing of Lennie justified? Explain your answer." The student is given the overall topic, but they are free to use their own judgment and integrate outside information to help support their opinion.

Student Skills Required for Essay Tests

Before expecting students to perform well on either type of essay question, we must make sure that they have the required skills to excel. Following are four skills that students should have learned and practiced before taking essay exams:

• The ability to select appropriate material from the information learned in order to best answer the question.
• The ability to organize that material in an effective manner.
• The ability to show how ideas relate and interact in a specific context.
• The ability to write effectively in both sentences and paragraphs.

Constructing an Effective Essay Question

Following are a few tips to help in the construction of effective essay questions:

• Begin with the lesson objectives in mind. Make sure to know what you wish the student to show by answering the essay question.
• Decide if your goal requires a restricted or extended response. In general, if you wish to see if the student can synthesize and organize the information that they learned, then restricted response is the way to go. However, if you wish them to judge or evaluate something using the information taught during class, then you will want to use the extended response.
• If you are including more than one essay, be cognizant of time constraints. You do not want to punish students because they ran out of time on the test.
• Write the question in a novel or interesting manner to help motivate the student.
• State the number of points that the essay is worth. You can also provide them with a time guideline to help them as they work through the exam.
• If your essay item is part of a larger objective test, make sure that it is the last item on the exam.

Scoring the Essay Item

One of the downfalls of essay tests is that they lack in reliability. Even when teachers grade essays with a well-constructed rubric, subjective decisions are made. Therefore, it is important to try and be as reliable as possible when scoring your essay items. Here are a few tips to help improve reliability in grading:

• Determine whether you will use a holistic or analytic scoring system before you write your rubric . With the holistic grading system, you evaluate the answer as a whole, rating papers against each other. With the analytic system, you list specific pieces of information and award points for their inclusion.
• Prepare the essay rubric in advance. Determine what you are looking for and how many points you will be assigning for each aspect of the question.
• Avoid looking at names. Some teachers have students put numbers on their essays to try and help with this.
• Score one item at a time. This helps ensure that you use the same thinking and standards for all students.
• Avoid interruptions when scoring a specific question. Again, consistency will be increased if you grade the same item on all the papers in one sitting.
• If an important decision like an award or scholarship is based on the score for the essay, obtain two or more independent readers.
• Beware of negative influences that can affect essay scoring. These include handwriting and writing style bias, the length of the response, and the inclusion of irrelevant material.
• Review papers that are on the borderline a second time before assigning a final grade.
• Study for an Essay Test
• How to Create a Rubric in 6 Steps
• 10 Common Test Mistakes
• Self Assessment and Writing a Graduate Admissions Essay
• UC Personal Statement Prompt #1
• Holistic Grading (Composition)
• Ideal College Application Essay Length
• How To Study for Biology Exams
• Tips to Cut Writing Assignment Grading Time
• T.E.S.T. Season for Grades 7-12
• What You Need to Know About the Executive Assessment
• Best Practices for Subjective Test Questions
• 2019–2020 SAT Score Release Dates
• Top 10 GRE Test Tips
• What Would You Do Differently? Interview Question Tips
• A Simple Guide to Grading Elementary Students

Your Article Library

Essay test: types, advantages and limitations | statistics.

ADVERTISEMENTS:

After reading this article you will learn about:- 1. Introduction to Essay Test 2. Types of Essay Test 3. Advantages 4. Limitations 5. Suggestions.

Introduction to Essay Test:

The essay tests are still commonly used tools of evaluation, despite the increasingly wider applicability of the short answer and objective type questions.

There are certain outcomes of learning (e.g., organising, summarising, integrating ideas and expressing in one’s own way) which cannot be satisfactorily measured through objective type tests. The importance of essay tests lies in the measurement of such instructional outcomes.

An essay test may give full freedom to the students to write any number of pages. The required response may vary in length. An essay type question requires the pupil to plan his own answer and to explain it in his own words. The pupil exercises considerable freedom to select, organise and present his ideas. Essay type tests provide a better indication of pupil’s real achievement in learning. The answers provide a clue to nature and quality of the pupil’s thought process.

That is, we can assess how the pupil presents his ideas (whether his manner of presentation is coherent, logical and systematic) and how he concludes. In other words, the answer of the pupil reveals the structure, dynamics and functioning of pupil’s mental life.

The essay questions are generally thought to be the traditional type of questions which demand lengthy answers. They are not amenable to objective scoring as they give scope for halo-effect, inter-examiner variability and intra-examiner variability in scoring.

Types of Essay Test:

There can be many types of essay tests:

Some of these are given below with examples from different subjects:

1. Selective Recall.

e.g. What was the religious policy of Akbar?

2. Evaluative Recall.

e.g. Why did the First War of Independence in 1857 fail?

3. Comparison of two things—on a single designated basis.

e.g. Compare the contributions made by Dalton and Bohr to Atomic theory.

4. Comparison of two things—in general.

e.g. Compare Early Vedic Age with the Later Vedic Age.

5. Decision—for or against.

e.g. Which type of examination do you think is more reliable? Oral or Written. Why?

6. Causes or effects.

e.g. Discuss the effects of environmental pollution on our lives.

7. Explanation of the use or exact meaning of some phrase in a passage or a sentence.

e.g., Joint Stock Company is an artificial person. Explain ‘artificial person’ bringing out the concepts of Joint Stock Company.

8. Summary of some unit of the text or of some article.

9. Analysis

e.g. What was the role played by Mahatma Gandhi in India’s freedom struggle?

10. Statement of relationship.

e.g. Why is knowledge of Botany helpful in studying agriculture?

11. Illustration or examples (your own) of principles in science, language, etc.

e.g. Illustrate the correct use of subject-verb position in an interrogative sentence.

12. Classification.

e.g. Classify the following into Physical change and Chemical change with explanation. Water changes to vapour; Sulphuric Acid and Sodium Hydroxide react to produce Sodium Sulphate and Water; Rusting of Iron; Melting of Ice.

13. Application of rules or principles in given situations.

e.g. If you sat halfway between the middle and one end of a sea-saw, would a person sitting on the other end have to be heavier or lighter than you in order to make the sea-saw balance in the middle. Why?

14. Discussion.

e.g. Partnership is a relationship between persons who have agreed to share the profits of a business carried on by all or any of them acting for all. Discuss the essentials of partnership on the basis of this partnership.

15. Criticism—as to the adequacy, correctness, or relevance—of a printed statement or a classmate’s answer to a question on the lesson.

e.g. What is the wrong with the following statement?

The Prime Minister is the sovereign Head of State in India.

16. Outline.

e.g. Outline the steps required in computing the compound interest if the principal amount, rate of interest and time period are given as P, R and T respectively.

17. Reorganization of facts.

e.g. The student is asked to interview some persons and find out their opinion on the role of UN in world peace. In the light of data thus collected he/she can reorganise what is given in the text book.

18. Formulation of questions-problems and questions raised.

e.g. After reading a lesson the pupils are asked to raise related problems- questions.

19. New methods of procedure

e.g. Can you solve this mathematical problem by using another method?

Advantages of the Essay Tests:

1. It is relatively easier to prepare and administer a six-question extended- response essay test than to prepare and administer a comparable 60-item multiple-choice test items.

2. It is the only means that can assess an examinee’s ability to organise and present his ideas in a logical and coherent fashion.

3. It can be successfully employed for practically all the school subjects.

4. Some of the objectives such as ability to organise idea effectively, ability to criticise or justify a statement, ability to interpret, etc., can be best measured by this type of test.

5. Logical thinking and critical reasoning, systematic presentation, etc. can be best developed by this type of test.

6. It helps to induce good study habits such as making outlines and summaries, organising the arguments for and against, etc.

7. The students can show their initiative, the originality of their thought and the fertility of their imagination as they are permitted freedom of response.

8. The responses of the students need not be completely right or wrong. All degrees of comprehensiveness and accuracy are possible.

9. It largely eliminates guessing.

10. They are valuable in testing the functional knowledge and power of expression of the pupil.

Limitations of Essay Tests:

1. One of the serious limitations of the essay tests is that these tests do not give scope for larger sampling of the content. You cannot sample the course content so well with six lengthy essay questions as you can with 60 multiple-choice test items.

2. Such tests encourage selective reading and emphasise cramming.

3. Moreover, scoring may be affected by spelling, good handwriting, coloured ink, neatness, grammar, length of the answer, etc.

4. The long-answer type questions are less valid and less reliable, and as such they have little predictive value.

5. It requires an excessive time on the part of students to write; while assessing, reading essays is very time-consuming and laborious.

6. It can be assessed only by a teacher or competent professionals.

7. Improper and ambiguous wording handicaps both the students and valuers.

8. Mood of the examiner affects the scoring of answer scripts.

9. There is halo effect-biased judgement by previous impressions.

10. The scores may be affected by his personal bias or partiality for a particular point of view, his way of understanding the question, his weightage to different aspect of the answer, favouritism and nepotism, etc.

Thus, the potential disadvantages of essay type questions are :

(i) Poor predictive validity,

(ii) Limited content sampling,

(iii) Scores unreliability, and

(iv) Scoring constraints.

Suggestions for Improving Essay Tests:

The teacher can sometimes, through essay tests, gain improved insight into a student’s abilities, difficulties and ways of thinking and thus have a basis for guiding his/her learning.

(A) White Framing Questions:

1. Give adequate time and thought to the preparation of essay questions, so that they can be re-examined, revised and edited before they are used. This would increase the validity of the test.

2. The item should be so written that it will elicit the type of behaviour the teacher wants to measure. If one is interested in measuring understanding, he should not ask a question that will elicit an opinion; e.g.,

“What do you think of Buddhism in comparison to Jainism?”

3. Use words which themselves give directions e.g. define, illustrate, outline, select, classify, summarise, etc., instead of discuss, comment, explain, etc.

4. Give specific directions to students to elicit the desired response.

5. Indicate clearly the value of the question and the time suggested for answering it.

6. Do not provide optional questions in an essay test because—

(i) It is difficult to construct questions of equal difficulty;

(ii) Students do not have the ability to select those questions which they will answer best;

(iii) A good student may be penalised because he is challenged by the more difficult and complex questions.

7. Prepare and use a relatively large number of questions requiring short answers rather than just a few questions involving long answers.

8. Do not start essay questions with such words as list, who, what, whether. If we begin the questions with such words, they are likely to be short-answer question and not essay questions, as we have defined the term.

9. Adapt the length of the response and complexity of the question and answer to the maturity level of the students.

10. The wording of the questions should be clear and unambiguous.

11. It should be a power test rather than a speed test. Allow a liberal time limit so that the essay test does not become a test of speed in writing.

12. Supply the necessary training to the students in writing essay tests.

13. Questions should be graded from simple to complex so that all the testees can answer atleast a few questions.

14. Essay questions should provide value points and marking schemes.

(B) While Scoring Questions:

1. Prepare a marking scheme, suggesting the best possible answer and the weightage given to the various points of this model answer. Decide in advance which factors will be considered in evaluating an essay response.

2. While assessing the essay response, one must:

a. Use appropriate methods to minimise bias;

b. Pay attention only to the significant and relevant aspects of the answer;

c. Be careful not to let personal idiosyncrasies affect assessment;

d. Apply a uniform standard to all the papers.

3. The examinee’s identity should be concealed from the scorer. By this we can avoid the “halo effect” or “biasness” which may affect the scoring.

4. Check your marking scheme against actual responses.

5. Once the assessment has begun, the standard should not be changed, nor should it vary from paper to paper or reader to reader. Be consistent in your assessment.

6. Grade only one question at a time for all papers. This will help you in minimising the halo effect in becoming thoroughly familiar with just one set of scoring criteria and in concentrating completely on them.

7. The mechanics of expression (legibility, spelling, punctuation, grammar) should be judged separately from what the student writes, i.e. the subject matter content.

8. If possible, have two independent readings of the test and use the average as the final score.

Related Articles:

• Merits and Demerits of Objective Type Test
• Types of Recall Type Test: Simple and Completion | Objective Test

Educational Statistics , Evaluation Tools , Essay Test

Comments are closed.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• The 4 Types of Reliability in Research | Definitions & Examples

The 4 Types of Reliability in Research | Definitions & Examples

Published on August 8, 2019 by Fiona Middleton . Revised on June 22, 2023.

Reliability tells you how consistently a method measures something. When you apply the same method to the same sample under the same conditions, you should get the same results. If not, the method of measurement may be unreliable or bias may have crept into your research.

There are four main types of reliability. Each can be estimated by comparing different sets of results produced by the same method.

Table of contents

Test-retest reliability, interrater reliability, parallel forms reliability, internal consistency, which type of reliability applies to my research, other interesting articles, frequently asked questions about types of reliability.

Test-retest reliability measures the consistency of results when you repeat the same test on the same sample at a different point in time. You use it when you are measuring something that you expect to stay constant in your sample.

Why it’s important

Many factors can influence your results at different points in time: for example, respondents might experience different moods, or external conditions might affect their ability to respond accurately.

Test-retest reliability can be used to assess how well a method resists these factors over time. The smaller the difference between the two sets of results, the higher the test-retest reliability.

How to measure it

To measure test-retest reliability, you conduct the same test on the same group of people at two different points in time. Then you calculate the correlation between the two sets of results.

Test-retest reliability example

You devise a questionnaire to measure the IQ of a group of participants (a property that is unlikely to change significantly over time).You administer the test two months apart to the same group of people, but the results are significantly different, so the test-retest reliability of the IQ questionnaire is low.

Improving test-retest reliability

• When designing tests or questionnaires , try to formulate questions, statements, and tasks in a way that won’t be influenced by the mood or concentration of participants.
• When planning your methods of data collection , try to minimize the influence of external factors, and make sure all samples are tested under the same conditions.
• Remember that changes or recall bias can be expected to occur in the participants over time, and take these into account.

Prevent plagiarism. Run a free check.

Interrater reliability (also called interobserver reliability) measures the degree of agreement between different people observing or assessing the same thing. You use it when data is collected by researchers assigning ratings, scores or categories to one or more variables , and it can help mitigate observer bias .

People are subjective, so different observers’ perceptions of situations and phenomena naturally differ. Reliable research aims to minimize subjectivity as much as possible so that a different researcher could replicate the same results.

When designing the scale and criteria for data collection, it’s important to make sure that different people will rate the same variable consistently with minimal bias . This is especially important when there are multiple researchers involved in data collection or analysis.

To measure interrater reliability, different researchers conduct the same measurement or observation on the same sample. Then you calculate the correlation between their different sets of results. If all the researchers give similar ratings, the test has high interrater reliability.

Interrater reliability example

A team of researchers observe the progress of wound healing in patients. To record the stages of healing, rating scales are used, with a set of criteria to assess various aspects of wounds. The results of different researchers assessing the same set of patients are compared, and there is a strong correlation between all sets of results, so the test has high interrater reliability.

Improving interrater reliability

• Clearly define your variables and the methods that will be used to measure them.
• Develop detailed, objective criteria for how the variables will be rated, counted or categorized.
• If multiple researchers are involved, ensure that they all have exactly the same information and training.

Parallel forms reliability measures the correlation between two equivalent versions of a test. You use it when you have two different assessment tools or sets of questions designed to measure the same thing.

If you want to use multiple different versions of a test (for example, to avoid respondents repeating the same answers from memory), you first need to make sure that all the sets of questions or measurements give reliable results.

The most common way to measure parallel forms reliability is to produce a large set of questions to evaluate the same thing, then divide these randomly into two question sets.

The same group of respondents answers both sets, and you calculate the correlation between the results. High correlation between the two indicates high parallel forms reliability.

Parallel forms reliability example

A set of questions is formulated to measure financial risk aversion in a group of respondents. The questions are randomly divided into two sets, and the respondents are randomly divided into two groups. Both groups take both tests: group A takes test A first, and group B takes test B first. The results of the two tests are compared, and the results are almost identical, indicating high parallel forms reliability.

Improving parallel forms reliability

• Ensure that all questions or test items are based on the same theory and formulated to measure the same thing.

Internal consistency assesses the correlation between multiple items in a test that are intended to measure the same construct.

You can calculate internal consistency without repeating the test or involving other researchers, so it’s a good way of assessing reliability when you only have one data set.

When you devise a set of questions or ratings that will be combined into an overall score, you have to make sure that all of the items really do reflect the same thing. If responses to different items contradict one another, the test might be unreliable.

Two common methods are used to measure internal consistency.

• Average inter-item correlation : For a set of measures designed to assess the same construct, you calculate the correlation between the results of all possible pairs of items and then calculate the average.
• Split-half reliability : You randomly split a set of measures into two sets. After testing the entire set on the respondents, you calculate the correlation between the two sets of responses.

Internal consistency example

A group of respondents are presented with a set of statements designed to measure optimistic and pessimistic mindsets. They must rate their agreement with each statement on a scale from 1 to 5. If the test is internally consistent, an optimistic respondent should generally give high ratings to optimism indicators and low ratings to pessimism indicators. The correlation is calculated between all the responses to the “optimistic” statements, but the correlation is very weak. This suggests that the test has low internal consistency.

Improving internal consistency

• Take care when devising questions or measures: those intended to reflect the same concept should be based on the same theory and carefully formulated.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

It’s important to consider reliability when planning your research design , collecting and analyzing your data, and writing up your research. The type of reliability you should calculate depends on the type of research  and your  methodology .

If possible and relevant, you should statistically calculate reliability and state this alongside your results .

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Normal distribution
• Degrees of freedom
• Null hypothesis
• Discourse analysis
• Control groups
• Mixed methods research
• Non-probability sampling
• Quantitative research
• Ecological validity

Research bias

• Rosenthal effect
• Implicit bias
• Cognitive bias
• Selection bias
• Negativity bias
• Status quo bias

Reliability and validity are both about how well a method measures something:

• Reliability refers to the  consistency of a measure (whether the results can be reproduced under the same conditions).
• Validity   refers to the  accuracy of a measure (whether the results really do represent what they are supposed to measure).

If you are doing experimental research, you also have to consider the internal and external validity of your experiment.

You can use several tactics to minimize observer bias .

• Use masking (blinding) to hide the purpose of your study from all observers.
• Triangulate your data with different data collection methods or sources.
• Use multiple observers and ensure interrater reliability.
• Train your observers to make sure data is consistently recorded between them.
• Standardize your observation procedures to make sure they are structured and clear.

Reproducibility and replicability are related terms.

• A successful reproduction shows that the data analyses were conducted in a fair and honest manner.
• A successful replication shows that the reliability of the results is high.

Research bias affects the validity and reliability of your research findings , leading to false conclusions and a misinterpretation of the truth. This can have serious implications in areas like medical research where, for example, a new form of treatment may be evaluated.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Middleton, F. (2023, June 22). The 4 Types of Reliability in Research | Definitions & Examples. Scribbr. Retrieved March 28, 2024, from https://www.scribbr.com/methodology/types-of-reliability/

Is this article helpful?

Fiona Middleton

Other students also liked, reliability vs. validity in research | difference, types and examples, what is quantitative research | definition, uses & methods, data collection | definition, methods & examples, what is your plagiarism score.

DigitalCommons@University of Nebraska - Lincoln

• < Previous Article
• Next Article >

Home > College of Business > Faculty Publications > 34

Business, College of

College of business: faculty publications, achievement differences on multiple-choice and essay tests in economics.

WILLIAM WALSTAD , University of Nebraska-Lincoln Follow William E. Becker , Indiana University - Bloomington Follow

Date of this Version

Published in American Economic Review 84:2 (May 1994), pp. 193-196. Copyright © 1994 American Economic Association. Used by permission.

Multiple-choice and essay tests are the typical test formats used to measure student understanding of economics in college courses. Each type has its features. A multiple- choice (or fixed-response) format allows for a wider sampling of the content because more questions can be given in a testing period. Multiple-choice tests also offer greater efficiency and reliability in scoring than an essay. The major disadvantage of a multiple-choice item is that the fixed responses tend to emphasize recall and encourage guessing. In an essay (or constructed- response) test, students generate responses that have the potential to show originality and a greater depth of understanding of the topic. The essay also provides a written record for assessing the thought processes of the student. Despite the claimed differences for each format, little empirical work exists to support the suppositions. If a multiple-choice and an essay test that cover the same material measure the same economic understanding, then the multiple-choice test would be the preferred method for assessment because it is less costly to score and is a more reliable measure of achievement in a limited testing period. If, however, an essay test measures unique aspects of economic understanding, then the extra examinee time and substantial scoring costs may be justified.

Since August 08, 2008

Included in

Business Commons

Advanced Search

Search Help

• Notify me via email or RSS
• Administrator Resources
• How to Cite Items From This Repository
• Copyright Information
• Collections
• Disciplines

Author Corner

• Guide to Submitting
• College of Business Website

Home | About | FAQ | My Account | Accessibility Statement

Privacy Copyright