Posts Tagged ‘tests’

Can You Be Data Driven Without Statistics?

Sunday, September 16th, 2012

Have you ever seen a margin of error reported on a state test result or an error bar on a state test graph? Has anyone ever reported a p value, an R squared, a standard deviation, median, or any other statistical measure along with a test result? Frankly I can’t recall even seeing an average (mean) when state tests are discussed. If we are truly trying to be data driven in our decisions as educators and institutions, I believe we need to do some basic data analysis to understand this data or else we end up in a state of DRIP (Data Rich, Information Poor).

Scientists/statisticians will tell you that the result of a test is not a single, true number, but a number with an error margin (or confidence interval) around that number. So in political polls you will see 55%  +/- 3%, because we understand if we polled multiple times on the same day using the same polling method, we would get a variation in the end result. This is true for students taking tests as well, however the information about the amount of variance is unpublished. Why care? Because important decisions are based around whether or not test scores rise or fall. So a department in a school might be put under increased pressure if their scores fell by 5%. However, if the variance of the test is +/- 8% then a 5% change is insignificant. That is, it is not possible to say that the decrease in test scores is due to students actually knowing less or whether it is due to random chance and natural variation.

Error bars will increase with a smaller population size and with a wider range of results. So it is more difficult to make solid claims of change on a single class/department than an entire district and it is more difficult to make solid claims of change with a diverse group of abilities than on a group of students that have similar abilities.

Let’s look at an example of actual data, first without statistics, then with statistics. Here is a table and graph of test results:

Year 1 Year 2 Change
Far Below Basic 12 11 -1
Below Basic 21 27 6
Basic 62 48 -14
Proficient 36 52 16
Advanced 11 14 3



This data suggests some improvement in test scores based on the “squint” analysis technique, i.e. getting a general impression based on the amount of green. In fact if you average the test scores on a 0=Far Below Basic to 4=Advanced scale you do see an improvement from 2.09 to 2.20 which is a 5% improvement.

However, to truly state the facts we have to include some information about the variance. Based on the standard deviation and population size, the 90% confidence interval for these averages is 14%, which means that the numbers actually have to be 14% greater in year 2 to show a significant increase. Another way of thinking about confidence intervals is that we are 90% certain the test results are +/- 14% from the reported value. When running a t-test to see if the two averages are significantly different,  the p value is .35, with .1 or less being considered significant in psychology. Here is a graph showing the average test scores with 90% confidence intervals for error bars.

The large amount of overlap shows that there is no measurable difference between the two years, yet this sort of analysis is not regularly done. Based on the few tests I have looked at for my school’s variance and sample size, it seems that around a 15% difference is a significant difference. While there are certainly nuances of statistics I don’t understand, I could easily accomplish this analysis based on a single college stats course and a spreadsheet program (LibreOffice calc or Microsoft Excel). Instead of spending ten of thousands of dollars on data warehouse systems, schools should run their data through basic analysis and then only do further investigation on significant changes. Now that I know a 15% difference is the threshold for a real difference I can easily dismiss smaller variations as likely due to random noise. I would hope that administrators and board members would ask for this confidence interval for their schools and then use it as a filter before jumping to conclusions. I think it would surprise people at my school that even a 10% change from year to year is statistically insignificant. But that is the power of math, it reveals truths which are often counter intuitive. If we want to be data driven, we need to stop analyzing numbers with our gut and do the math.

PS- This example data does not show significant improvement in student learning, even given the assumption that a single 60 question multiple choice test is an accurate measure of student learning of a complex subject. If we start questioning the correlation between the results of the test and the actual student learning outcomes (success in college, career readiness, application of material learned in the real world), then we are even farther from being able to prove anything with these numbers.

In Defense of Facts

Thursday, November 5th, 2009

It is easy to fall into false dichotomies: mind vs body, teacher vs administrator, Coke vs Pepsi (maybe not). One of the most prevalent false dichotomies in education is facts vs concepts, often meant to mean multiple choice test vs performance based assessment, rote memorization versus understanding, book learning versus real world learning. (an important side note that could be a whole discussion in and of itself is that these groups of dichotomies are not synonymous)  Painting these ideas as in opposition goes against how knowledge works in practice, the current understanding of the brain,  and is an impediment for people who want to reform schools.

How knowledge works: One of the loudest arguments against learning facts these days is “You can look up anything on the internet.” This statment has value when it is used to encourage teaching kids how to do good research and how to synthesize multiple sources into a cohesive understanding. When used to argue that facts themselves have no value and do not assist in the process of analysis, however, that argument is misleading.

Hypothetical situation: Pick 2 topics, one you know a lot about and one you know very little about. I will pick botany and accounting. Now you have to perform some research to answer a fairly complex question on each of these topics (Compare the different groups of gymnosperms and compare absorption costing to asset turnover) . You have 10 minutes for each question. Go.

I don’t have the aspects of the different gymnosperms memorized. However, I know what a gymnosperm is, I remember that there are different groups, and I understand the basics of plant structure so that when I read articles about the topic I can more easily filter the information.

I couldn’t even think of a question to ask in accounting, so I first had to search for a glossary of terms and then look around until I found two terms that seemed comparable (I don’t know if they are actually comparable). I don’t even know what I don’t know. As I try to learn more about accounting, I keep having to stop and look up basic ideas used in articles because I lack the background information. The words themselves seem to “stick less” in my head than the botany terms.

I am the same person, with the same research and synthesis skills, the same ability to pick up new ideas, and yet presented with a similar problem in two different factual areas I yield two different results. Why? Because facts matter. My prior knowledge about plants made it easier for me to understand new knowledge about plants. I had mental structures in place for categorizing information about plants, that were built when I was first asked to learn botany. I know how roots, stems, and leaves work ( I know that all plant structures can be classified as a root, stem or leaf), and so I can more easily compare the roots of the groups of gymnosperms because my brain groups that information together for me.

These mental structures are not helping me remember the things I read about accounting. I find myself wanting someone to explain the basics of accounting from an expert perspective so that I can frame my mind around ideas. Then I might be able to sift through all of the terms. I want someone to teach me the facts about accounting, so that I can approach the sea of information with some guidance.

Obviously facts matter or we wouldn’t have specialists. Specialists were not created merely due to the fact that only a few people had access to pieces of information. Specialists are still needed in an open information world, because information is really hard to understand without a lot of background.

Modern understanding of the brain? Well we don’t have a perfect understanding of memory, but we do know a few things that help break this dichotomy. There are no isolated facts in the brain. There are only connections. Your brain is a network of billions of neurons and every time you experience anything new, your brain changes those connections. That is learning. The more your brain is exposed to a certain language as a child the more the connections in the brain match the patterns of the language and the better the brain is at that language. When you are trying to learn specific details (your mother’s face, a phone number, the capital of France), your brain must change to emphasize a certain pathway and then be able to activate that pathway at some other time (recall). Your brain seems to store similar ideas together (apparently when you put an electrode in a brain and stimulate one area ideas of cars and predatory animals are activated) The more an idea is connected to other ideas in the brain (due to similarity) the easier it is to reactivate. This is why progressive educators are right in moving teaching towards contextualizing knowledge. No matter how hard you try to cram an idea into a person’s head as an isolated fact, the brain will store that idea as a connection to other ideas.

So, what we call facts are really just parts of concepts, and what we call concepts are just a bunch of facts. No fact can exist on its own in the brain, and no concept can exist on its own in the brain independent of the specific facts that gave rise to the concept.

OK, so hopefully you are willing to consider that facts and concepts aren’t antithetical, and maybe content knowledge is useful, but why is the dichotomy bad for reform? If we put the choice to administrators, teachers, and the public: the children will learn EITHER facts or concepts, facts will win. We shouldn’t be saying that the only way to teach a fact is to be as boring as a test. Teach the children the content through engaging problem based activities that activate their brains and allows them to create multiple connections. If you give the kid only one connection to an idea and isolate from the world because it is a hated fact, then the kid won’t learn that fact. We should teach facts the same way we teach concepts, because they are no different.

Teaching Example: The parts of the cell are a collection of facts that are often on the standardized tests. I have used some form of the following lessons at some time in my time as a teacher.

Bad Lesson: Give the kids a sheet with the list of cell parts and definitions and tell them they need to learn them.

OK Lesson: Have kids make flashcards for each cell part by looking up the definition and spend 10 minutes a day quizzing themselves and others on the definition of the cell part.

Good Lesson: Have the kids make a poster of the cell parts with the definitions, pictures, examples of the cell part doing its job, and a metaphor for each cell part (as a factory or town)

Great Lesson: Have the kids debate which cell part is the best. Each group defends two cell parts in a single elimination tournament. Each side presents an opening statement with facts about what the cell part does and why that is important to life (and the audience). Then there is a cross examination/rebuttal period for the two teams to interact. The winner is decided by secret ballot (the teacher gets extra votes) and moves on until a “Most Important Cell Part” is crowned (during the debate students realize that every part is important for life and the parts of the cell are interconnected).

Theoretically  Awesome Lessons (never actually tried these): What about having students contact researchers who spend their whole lives studying one cell part to reveal how a single vocab word has a huge existence in the real world? What about a game/simulation where the students manage a cell to reveal the dynamic nature of these parts?

In all of these lessons, the students are learning factual content that will help them perform well on multiple choice tests. However the more the students connect those facts to other ideas (the metaphors, the social memory of the debate) the better they will learn. Also the standardized test is a pretty good measure of how well the students learned about cell parts and how engaging the class was.