Don’t always believe what you hear. Lisa was watching TV and saw a commercial that claimed that kids were 20% more attentive when they eat this breakfast as opposed to no breakfast. This prompted her to ask me "What if they had a donut? How does that score?"

She had a point, as she often does. We joked about it for a few minutes and forgot about it. Recently I saw a Shredded MiniWheat commercial that said "kids that had a wholesome Shredded MiniWheats breakfast were more focused." They changed it. They used to say that "in a recent clinical study children who ate Frosted Mini Wheats had 23% better memory than those who went without breakfast." Now all they focus on is how your kid will be full and focused. So I decided to look a bit more into it.

Kelloggs went to the trouble of funding a "clinical study." This theory behind the study was that children who eat Frosted Mini Wheats are better prepared for School than alternative breakfasts. Not a bad idea. So let’s think about the best way to setup this experiment:

1. Get a group of children all about the same age, background, socioeconomic status and grade. This way we eliminate the variables these would introduce to the equation.
2. Inform the parents of the rules of the test: no feeding the kids for x number of hours before the test, bring them to the testing center at the same time, etc.
3. Setup testing at breakfast time for all these children. Using the School cafeteria would be a great place.
4. Feed them all the same thing.
5. Feed them all a different thing.
6. Don’t feed them anything.
7. Give them a memory test at specified periods after the meal (1 hour, 2 hours, 3 hours, etc.).
8. Repeat this experiment enough times to be statistically relevant.
9. Publish methods and results.
10. Experts and testers then draw donclusions.

It’s all pretty basic really. Anyone who’s spent any time in college should know how to set this up. So let’s go to the study and see what they did. Aaand… um… what? Really? So, in case you missed it, they didn’t do most of our suggestions above. Let us run through them again:

1. “tests were conducted in 8 – 12 yr old children from various backgrounds” How do we know if they gorged themselves the night before or that their mother wakes them up with a cup of hot chocolate or that those going through puberty aren’t going to score differently than those who are not. Too many variables. This almost invalidates the test. Let’s move on.
2. Doesn’t say, so let’s give them the benefit of the doubt.
3. They used a testing center. Good.
4. Of course.
5. Nope. They completely ignored my Wife’s initial question, which is probably what most people asked when they saw the commercial. So they cannot answer "Compared to what?" Which is really the important question. Even if the answer was "doughnuts are far better than any other breakfast for cognitive ability", the data would still be important.
6. They provided the kids water which somehow equals nothing.
7. I assume they did this because they say the children "demonstrated better attentiveness and quality of memory throughout the morning." Good on ya mate.
8. They did this, but they only considered the results of one of the measurements. I’m assuming the rest were thrown out. Yay missing data.
9. This was the only thing published. The data was not allowed to be scrutinized by independent sources.
10. The conclusion was reached by the testers only.

Only when someone complained to the Federal Trade Commission was anyone allowed to really scrutinize the study. They didn’t try to control variables, they didn’t use other testing parameters (including Keloggs other cereals) and the worst bit is that the data wasn’t written up and published for the rest of the world to see.

This is an example of corporations doing what we ususally see the political machine doing. It’s a simple formula:

1. Come up with an answer you want.
2. Find a way to test to see if it’s true.
3. Engineer the test to give the answer you want.
4. Make it sound scientific and legitimate.

I love lists, they’re so organized. Anyways, question your commercials, you might learn something.