Taking research with a grain of salt

All of it is anecdotal

By Allen Williams

The British statistician George E. P. Box stated that “All models are wrong, but some are useful.”

This has become an oft-quoted statement in scientific circles. Box was referring to the fact that in science there is a growing trend to develop theoretical models with the purpose of predicting some type of behavior or outcome based on data assumptions used in the model.

While no model can predict the exact outcome of any singular event, models can be useful if the assumptions are good and the output is close enough.

Having been a scientist and a farmer for more than 30 years now, I often hear people talk about “anecdotal” research or data. Their point is that if the research was not peer-reviewed and published, it has no value. This is particularly insinuated with observational data.

There is even sharp disagreement among various scientific disciplines and communities about what constitutes a legitimate peer-review journal, and what does not. In other words, even scientists cannot agree on which peer-review is good, and which is not so good.

Scientific opinions differ
In today’s scientific community there are many different opinions relative to what research is deemed legitimate, and what research is deemed anecdotal.

This is largely because several decades ago the scientific community coalesced around something called the “reductionist model”. The basis of this model is that we must control as many variables as possible in order to examine the effects of one or two specific variables.

The reductionist model has significant applications in certain circumstances. If you need to determine the exact dose to use of a particular pharmaceutical, then a reductionist model is beneficial.

However, in other applications it can actually serve as a hindrance. For instance, if you need to determine the impact of regenerative practices on soil health, the reductionist model gets in your way.

Observation is better
Drug research is best served through reductionist science, while soil health is better served through what we observe. This is observational science.

For several thousand years we made significant scientific advancements through observational science. There was no need for a reductionist model to discover gravity or many of our laws of physics.

If Joe in Africa, Robert in Europe, Jim in the U.S., and Colin in Australia all throw a rock in the air, it will come down. If they all experience a collision, the laws of physics will apply and are readily observed.

So let’s address how much of the data from reductionist research trials can be anecdotal in nature. To do this we will use a fictional research trial that is typical of much of today’s research. Since this is a publication that focuses heavily on grazing, we will use a fictional grazing trial.

A research example
Let’s say the researchers want to examine grassfed beef production. To do so, they set up a trial on a specific research station. They determine the variables they want to measure and the variables they want to control.

They decide to measure animal gain performance, carcass quality and sensory evaluation of the end product. To do so, they decide on a specific stock density, rotation frequency and end point for harvest.

The livestock are steer calves sourced from the university system. The research is conducted for a specific time period each year for three consecutive years. The control group is cattle finished in a traditional feedlot situation. Replicates would be added for greater validity.

There would be much more detail beneath this, but you can understand the premise. This trial would fit an experimental statistical model quite well, and thus be eligible for peer review.

After the trial is completed, the data analyzed and the results peer-reviewed and published, how are the results of this accepted, reductionist research both non-anecdotal and anecdotal?

They are non-anecdotal when the results are presented without any extrapolation beyond the specific trial itself.

The results have validity as long as the presenter indicates they pertain to the exact conditions, location, years, breed type, phenotype, soil type, soil health status, climatic conditions, end point at harvest, researcher knowledge of what they were doing, plant species in the pastures, inputs used and other factors affecting the particular trial.

However, that is not how we treat peer-reviewed research. The majority of scientists immediately extrapolate the results to broader applications. At that very moment, they have just become anecdotal.

Let’s assume the results from the trial show that the mean ADG was 1.5, the mean quality grade was Mid-Select, the mean liveweight at harvest was 1,100 lbs., the mean hanging carcass weight was 594 lbs., and the sensory evaluation showed issues with tenderness, juiciness and flavor.

Do these results accurately represent the grassfed beef sector and indicate that they would be typical?

The definitive answer is “no”. However, I have been present at meetings and conferences where similar peer-reviewed data was presented and represented as gospel by the presenting scientist. They extrapolated their specific data to a broader application, and thus made it anecdotal.

Why it’s anecdotal
Why is such research anecdotal when extrapolated to the larger world?

There are too many reasons to fully list, but I will ask some questions to give you a flavor for why this is so:

• What knowledge and experience did the researchers themselves have in grassfed beef production? Just because you have a Ph.D. behind your name does not make you a competent practitioner. (I am allowed to say that since I do have that Ph.D. behind my name). Do they know how to graze properly? Do they understand when an animal is fully finished?

• What was the specific location of the research? Climate? Environment? Annual precipitation? Temperatures?

• What was the soil type(s)?

• What were the forages growing in the paddocks? Biomass production? Diversity?

• What was the grazing strategy?

• How did they determine the harvest end point for the animals on trial?

• What breed or breed types were used? What was the phenotype?

• How were the cattle managed from conception to harvest?

• Where were they harvested? What skill level did the processor have? Did they dry age? How long? Wet age? How were the carcasses fabricated?

• Who actually did the day-to-day management and data collection? The researcher themselves (i.e., the Ph.D.), or graduate and undergraduate students?

• What variables were they trying to control?

• What was actually measured and taken into account? Did they quantify and measure soil health parameters (biology, water infiltration, aggregate, etc.)? Did they account for all plant species growing in each field, even the forbs or “weeds” the cattle may have eaten?

• What inputs were applied? Chemical, synthetic, supplements, amendments, mechanical?

When I hear scientists present their peer-reviewed data, they usually do so by telling the producers in the meeting that this is what the producer should expect or anticipate if they try to do grassfed production (or any other practice).

Instead, what the scientist should say is that this is what we did, how we did it, and these were our results under these specific circumstances and constraints. You, the producer, may experience very different results on your own farm.

Don’t extrapolate
What I am not saying is that peer-reviewed research is useless and has no place in science. That would be grossly untrue.

What I am saying is that all research can be made anecdotal through extrapolation beyond the specifics of the trial itself.

So examine all research carefully and with a grain of salt. Many non-scientists gain their information about research through the popular press, rarely reading the actual peer-reviewed article. Be especially careful with those articles. The author(s) of the popular press articles usually do their own version of extrapolation trying to prove whatever their point may be.

Do your own trials
At Understanding Ag, we encourage our farmers to do their own on-farm trials to see what really works best under their context. Read, study, consider, get advice and consultation, but do your own research. That is how you will make the most progress in the shortest period of time.

I am reminded of what is stated in Acts 17:11: “Now the Bereans were of more noble character than the Thessalonicans, for they received the message with great eagerness and examined the Scriptures daily to see if what Paul said was true.”

If the Bible says to do our own research, that’s good enough for me.

Dr. Allen Williams is president of Livestock Management Consultants, LLC, based in Starkville, Mississippi.