One variable at a time?

I’ve been reflecting on the concept of “Fair Tests” and what the implications are for how we teach Scientific Enquiry and the thoughts are a little unsettling.

As a science teacher I teach the “identify your variables” and “control everything except the one  you want to measure the effect of” – the well taught “one variable at a time” approach.  I am fairly certain that most if not all science teachers reading this will teach the idea of fairness and variables in a similar way.

Indeed, this approach is embedded into the National Curriculum (Wales):

• KS2: When carrying out a fair test, the key variables that need to be controlled and how to change the independent variable whilst keeping other key variables the same.
• KS3: When carrying out a fair test, control variables appropriately and identify any variables that cannot readily be controlled
• Levels
  • Level 4: In a fair test enquiry, they recognise, with support, the variables to change and measure and those to be kept the same. They decide upon some basic success criteria.
  • Level 5: When planning a fair test, they identify key variables and distinguish between independent and dependent variables and those that they will keep the same.
  • Level 6: They make predictions using abstract scientific ideas. In a fair test enquiry, they plan how to control the variables that they need to keep the same and make decisions about the range and values of the independent variable.
  • Level 7: They identify key variables that may not be readily controlled explaining why this is the case.
  • EP: Plan to track changes in more than one dependent variable.
• KS4:HSW defines a fair test as: one in which only the independent variable affects the dependent variable.  All other variables are controlled, keeping them constant if possible

Interestingly the sophistication here goes from “keeping other variables the same” (the aforementioned one variable at a time) through to KS3 accepting that this might not be possible “…and identifying any variables that cannot readily be controlled” to KS4, where we seem to go back a step to “ All other variables are controlled, keeping them constant if possible”.  Nowhere in this are we directed to explain how to deal with variables that might change — indeed, if like me, you will explain this as a source of error or to explain “anomalous” results.

I am perturbed by this approach for 2 reasons:

1. It does not seem to get appreciably more sophisticated as the key stages progress.  In fact, this level of controlling everything and using non control as a source of error persists at KS5 and I’ve lost track of the number of A-level students who write “to keep it a fair test” when describing controlling variables.
2. This is not how the real world works. Science rarely has the luxury of controlling ALL variables and just changing the one that we are interested in.  Yes, in an ideal world that’s what we’d do (may be), but certainly not how research science approach a problem.

As someone who worked as a research scientist prior to coming back to teaching, it is this latter point that is eating away at my inner scientist and here’s why:

The real world is about processes, with inputs and outputs.
Some we can control, some we can measure.  As a scientist researching, we try to control what we can (in a vain attempt at one variable at a time) and only change what we are trying to investigate.  Oftentimes, this is not practical, but more importantly, sometimes our attempts at controlling things actually affects the results we measure.  Take measuring the bounce height of a tennis ball.  We would normally determine that keeping the surface that we bounce onto constant in order to make a “fair test” was a sensible thing.  So, we bounce the ball onto a laminate floor and can conclude that “the higher we drop the ball, he higher it bounces” – great stuff.  But….. this is only true for the special case of bouncing the ball onto a laminate floor.  We forget to say, “The higher we drop a ball onto a laminate floor, the higher it bounces”.  The learner is left with the (potential) misconception that dropping a ball higher will always lead to a higher bounce.  (As an aside, dropping a tennis ball onto the surface of a trampoline produces a different result).

A Process “Black Box” Model

Suppose that we have a process with two inputs and one output (Y), as shown below. We do not know what mechanism is inside the process ‘box’ so we call it a ‘black box.’ This could be our ball experiment, with height of drop being X1, and surface being X2, with the output being the bounce height.

Now, we consider an experiment to determine the bounce height (Y) as a function of the input variables. One such experiment could be: Hold all inputs but one fixed, and see the best result when the one free input is varied. Fix that input at that ‘best’ value. Then vary one other input. See where the best output is now. Fix the second input at that ‘best’ value, and so on until we run out of input factors. This is called a ‘One variable at a time’ (OVAT) experiment, and is practiced widely. It used to be thought that this was the only ‘scientific’ approach.

OVAT experiments will work if the true model inside the black box looks generally as follows:

This model is ‘flat’ in all dimensions, and is called a ‘main effects’ model. No matter at which point on the surface one begins, increasing an input always has the same effect on the output. There are no ‘interactions’ between inputs.

In this model, X₁ at its highest setting will always give best Y, and similarly X₂ at its highest setting will always give best Y. (We are assuming highest Y is ‘best.’)
OVAT experiments will not work if the true model inside the black box looks something like: 

An interactions model of this type has a ‘twist’ to the response surface. This means that the response Y₁ to, say, X₁ will be different depending on the setting of X₂(and vice-versa).  Interactions can also be plotted in two dimensions as in the following two examples.

In this model, if one started experimenting with X₂ set at its lowest value, X₁ would have to be moved toward its lowest value to get a high Y. On the other hand, if one started out with X₂ fixed at its highest value, X₁would have to be moved up to get a high Y. We do not know if X₁ and X₂ both set at low will give a better Y that X₁ and X₂ both set high.

In a model with many inputs, the two-factor interactions such as X₁*X₂are usually of interest, as they might point the way to a better product with minimal additional expense. OVAT experimentation leaves us in the dark about factor interactions. 

In this model, if one started experimenting with X₂ set at its lowest value, X₁ would have to be moved toward its lowest value to get a high Y. On the other hand, if one started out with X₂ fixed at its highest value, X₁would have to be moved up to get a high Y. We do not know if X₁ and X₂ both set at low will give a better Y that X₁ and X₂ both set high. 

In a model with many inputs, the two-factor interactions such as X₁*X₂are usually of interest, as they might point the way to a better product with minimal additional expense. OVAT experimentation leaves us in the dark about factor interactions.

So what and why am I commenting?

There is an increasing debate about the practice of “real science” in the classroom – where we get students to undertake real research and present on real findings.  If we are going to push the agenda of real science, we also need to use real maths.

An engineer called Genichi Taguchi formalised this approach to handling many variables and using maths to sort out the main effects.  Oddly though, unless you did an engineering degree you are unlikely to have heard of him.  If you did a biology degree, you might remember running designed experiments to use statistics to analyse many variables at once — but have you ever mentioned this in the class room?

So, do you teach “One variable at a time?” – If so, have you considered the alternatives?

Substancial portions of this explaination came from: here