Bruce G. Marcot
Research Wildlife Ecologist
USDA Forest Service
Pacific Northwest Research Station
Portland Forestry Sciences Lab
 

EXTRACT FROM:

Marcot, B. G., M. A. Castellano, J. A. Christy, L. K. Croft, J. F. Lehmkuhl, R. H. Naney, R. E.
Rosentreter, R. E. Sandquist, and E. Zieroth.  1997.  Terrestrial ecology assessment.  Pp.
1497-1713 in:  T. M. Quigley and S. J. Arbelbide, ed.  An assessment of ecosystem
components in the interior Columbia Basin and portions of the Klamath and Great Basins.
Volume III.  USDA Forest Service General Technical Report PNW-GTR-405.  USDA Forest
Service Pacific Northwest Research Station, Portland, OR.  1713 pp.
 

METHODS

COLLECTING EXISTING INFORMATION

The general approach to meeting the study elements entailed gathering existing
information on ecology of individual species, species assemblages, and ecological
communities in the assessment area, and on developing computer models that describe
species-environment relations.  Two interesting dilemmas arose in this general approach.
First, limited information was available from a wide variety of sources, including experts
in panels and individual contract reports, scientific publications, agency management
plans, and other sources; how could all this be meaningfully combined?  Second, much of
the existing information has not been formally published and resides in the files and heads
of experts; how could this knowledge be gathered?  To meet the first challenge we turned
to various methods of combining information.  To meet the second, we contracted for
technical reports from species experts, and gathered information in panels of experts.  It is
imperative to note, however, that formalizing expert knowledge from panels of experts is
not a substitute for open publication which has undergone peer review.  Rather,
knowledge gathered in this assessment could be used to create tentative working
hypotheses that need explicit empirical testing and validation (for example, Doak and Mills
1994).

Collectively, the information we gathered on species-environment relations and the
interpretations we made on possible trends of species and their ecological roles in the
ecosystem constitute an ecological risk analysis (for example, Calabrese and Baldwin 1993,
Norton and others 1988).  We evaluated trends of potentially suitable environments of
species from historic to current times, given the changes in vegetation cover types and
structural stages and the occurrence of stochastic events such as wildfire.  From this, we
estimated the resultant likelihoods that populations of plants and animals have increased,
declined, or not changed over that time, and how such changes might influence the
ecosystems in which the species reside.

Of importance in combining information from disparate sources is depicting its quality.
The most reliable species and community data for this assessment came from well-designed
empirical field studies with ample sample sizes and time periods.  Such studies are rare.
Rather, most of the information we used in this assessment was gathered from
unpublished (in scientific journals) or ongoing studies, or from expert judgment.

One kind of error that is inescapable when dealing with biological systems, particularly at
the geographic extent and time durations addressed in this project, is that of natural
variation of conditions.  Although much of the data collection for this project seems
deterministic (unvarying), it is collected and interpreted with full understanding that
ecosystems and populations are inherently inconstant.

Other kinds of errors are those of human judgment and understanding: uncertainty and
ambiguity (see Cleaves 1994).  Uncertainty refers to  incompleteness of knowledge.  To help
us determine how well non-plant species' ecologies are understood, we asked each expert
panelist (see methods section on expert panels) to denote on a scale of 1 to 5 how well they
thought the species is known scientifically.  We assigned the median certainty level to each
species, then tallied the number of species under each of the five certainty levels for each
taxonomic group.  We also explored the variation in certainty levels among panelists for a
given species to measure within-species (among panelist) certainty variation.  Ambiguity
refers to differences in interpretation of direction and information among experts.  We
produced what we hoped were clear direction in contracts, and held standard pre-panel
briefings so that all experts began from the same knowledge level of the project.

A final potential source of error is motivational bias.  This occurs when someone is asked to
provide information on value-laden categories or terms, rather than on more neutrally
phrased terms.  For example, a reply is more apt to highlight problems when the category
is "threats causing extinction" than when it is "likely events that may affect populations."
To help avoid errors associated with motivational bias, we strived to ensure a clinical,
neutral tone in panel proceedings and other contacts with experts.

Problems and Opportunities of "Combining Information"

A new area of statistics has recently emerged that deals with the problem of making
meaningful inference from disparate sources of information.  This area is dubbed
"combining information" or CI (Draper and others 1992).  We used several CI techniques.
CI helped us to think hierarchically, such as by first denoting ecological attributes of
individual species, and then search for similar ecological functions among groups of
species.  When combining separate sources of judgmental information, process is important
to allay bias and to ensure consistency in interpretation.  Building tentative working
hypotheses, such as on species fitness or viability or community productivity, is a plausible
and useful approach even if the base information is taken from widely disparate sources.
We used this approach.

Another arena that helped us develop our approach to gathering existing information is
that of expert system programming.  In a sense, much of the information we gathered is in
the form of expert system rules.  In our context, these "rules" are information bases that
describe conditions suitable to meet the needs of individual species or species groups.

Using Expert Panels

To collect and combine expertise, we convened a total of 26 panels of leading species
experts.  Panelists were chosen individually for their expertise in specific taxonomic
groups and geographic areas and collectively for their coverage of all groups and areas.
Panelists chosen were recognized as the leading experts, or among the leading experts, in
their fields.  Often, as with invertebrates and plants and allies, the leading experts were
essentially the only scientists conducting primary field research on the organisms of
interest.  The process we used generally followed that of two previous scientific
assessments of biodiversity and species viability (FEMAT 1993; also see Kangas and others
1993, Thomas and others 1993).  This was a modified Delphi process as follows.  Panelists
followed a standard protocol consisting of:  individually providing information on species
ecologies on standardized forms; discussing initial results to uncover overlooked or new
information; and refining the final submitted information.  This is a modification of the
standard Delphi process in that discussions and final recording of information was not
meant to achieve group consensus, due to the mandates of the Federal Advisory Committee
Act as interpreted by SIT management and legal counsel.  Also, retaining individual
information allowed us to pool the diversity of expertise among panelists and to poll their
opinions of confidence levels of scientific knowledge of each species.  In the end, each
panelist provided separate information, and we merged the panelists' contributions.

The panelists were asked to record key environmental correlates -- those requirements that
contribute to high realized fitness, and by inference, viability, of each species.  The
panelists also recorded key ecological functions of each species and the degree of confidence
in current scientific understanding of correlates and functions of each species.  We deviated
from the approaches of previous assessments (namely, FEMAT 1993, Thomas and others
1993) by having the panels provide information on basic ecology and species-environment
relations, rather than having the panels provide a final judgment on species viability
effects of management scenarios and planning alternatives.  This change was made because
(1) the EIS alternatives had not yet been formulated, (2) one purpose of the current
exercise was to expose the ecological relations to help us interpret and project viability
effects, (3) another purpose was to build an explicit information base on species-
environment relations, which is lacking for the assessment area, and (4) we wished to
avoid having to reconvene expert panels for every change in EIS alternatives.  Information
from the panelists and contractors helped us to build basic and explicit ecological
databases, including range and distribution maps, from which we developed repeatable
and explicit ecological risk analyses. cross check with intro chapter

An advantage of this approach is that it helped reduce the potential problem of
motivational bias.  On the other hand, one disadvantage of the "modified" paneling
approach is that it is more difficult -- but not impossible -- for many technical experts to
explain how they think than it is for them to provide only a summary judgment on a
condition (such as on potential species viability effects).  Because of this, we peer-reviewed
and revised the information.

Environmental "dependencies" were not listed in the resulting species information
database per se.  Rather, the aim of collecting expert knowledge was to list key factors
influencing realized fitness of each species.  For some panelists or experts this meant listing
all environmental factors whether critical to maintaining populations or only incidentally
used; for others, it meant listing only those that contributed to "core" populations (those
with births exceeding deaths).  This was quite difficult to quality-control during the panel
and information-gathering process, as it meant different things to different panelists.  This
is one reason that the resulting information base must be viewed as a "first approximation"
of species-environment relations and must be interpreted in light of local biological
information and knowledge.

Confidence in Knowledge of Species

As part of the species data collection process (modified Delphi panel process), we asked
experts to rank their confidence in how complete scientific knowledge is for each
individual species on a scale of 1 (low confidence) to 5 (high confidence).  We clarified that
scores of low confidence should correspond to few or no empirical scientific studies of the
species within the Basin assessment area, whereas scores of high confidence should
correspond to adequate studies for predicting population response to environmental
conditions.

Different experts may have interpreted this scale in slightly different ways.  Some ranked
their own empirical experience, whereas others ranked the collective knowledge of the
scientific community.  However, in the final pass these two interpretations should be the
same, as the best experts available were engaged on the panels.

Some general trends emerged.  The median confidence levels (among panelists ranking the
same species) on the whole were greater with plants (51 percent of species rated a median
confidence rank of 4 or 5) than with vertebrates (37 percent), and greater with
vertebrates than with invertebrates (23 percent) (fig. 2).  The plants and plant groups
included in this tally are the rarer species with federal or proposed listing status, and may
be those more intensely studied.  This does not represent knowledge levels of all plant
species.  Also significant is how poorly known the invertebrates are; 22 percent of the
example species addressed were ranked as confidence level 1, as compared with 8 percent
of the vertebrates and 7 percent of the plants.   Overall, these median confidence ranks
should be interpreted as denoting that few species taxonomic groups have been completely
studied, that further basic biological research is needed on many groups, and that
conclusions drawn on species historic trend and current response to environmental
conditions should be treated as management hypotheses to be tested.

Where more than one panelist provided information on a species, we calculated median
confidence scores for each species among panelists.  Where only one panelist provided
information on individual species (often the case with lesser-known taxa including fungi,
lichens, bryophytes, many of the rare plants, most invertebrates, and some amphibians and
reptiles), we used their one confidence score.  To compare overall confidence in knowledge
among taxa, we plotted the confidence mean among species (as proportions) for each
confidence level for each taxonomic group (plants and allies, invertebrates, and
vertebrates).

The following sections describe general methods used to evaluate habitat trends and
functions of plants, invertebrates, and vertebrates.  Detailed accounts of methods and
results are available in the contract reports (Appendix B), and in on-file material.

LITERATURE CITED [FROM THE ABOVE EXTRACT]

Calabrese, E. J., and L. A. Baldwin, ed.  1993.  Performing ecological risk assessments.
Lewis Pub., Boca Raton, FL.  250 pp.

Cleaves, D. A.  1994.  Assessing uncertainty in expert judgments about natural resources.
Southern Forest Experiment Station Gen. Tech. Rpt. SO-110.   USDA Forest Service, New
Orleans LA.   17 pp.

Cleaves, D. A.  1995.  Assessing and communicating uncertainty in decision support
systems: lessons from an ecosystem policy analysis.  AI Applications 9(3):87-102.

Doak, D. F., and L. S. Mills.  1994.  A useful role for theory in conservation.  Ecology
75(3):615-626.

Draper, D., D. P. Gaver, Jr, P. K. Goel, J. B. Greenhouse, L. V. Hedges, C. N. Morris, J. R.
Tucker, and C. M. Waternaux.  1992.  Combining information.  Statistical issues and
opportunities for research.  Contemporary statistics No. 1.   National Academy Press,
Washington D.C.   217 pp.

FEMAT.  1993.  Forest ecosystem management:  an ecological, economic, and social
assessment.  Report of the Forest Ecosystem Management Assessment Team.  U.S.
Government Printing Office, Washington, D.C.  (chapters numbered separately) pp.

Kangas, J., J. Karsikko, L. Laasonen, and T. Pukkala.  1993.  A method for estimating the
suitability function of wildlife habitat for forest planning on the basis of expertise.  Silva
Fennica 27(4):259-268.

Norton, S., M. McVey, J. Colt, J. Durda, and R. Hegner.  1988.  Review of ecological risk
assessment methods.   US Environmental Protection Agency, Washington, D.C.   (sections
numbered separately) pp.

Thomas, J. W., M. G. Raphael, R. G. Anthony, E. D. Forsman, A. G. Gunderson, R. S.
Holthausen, B. G. Marcot, G. H. Reeves, J. R. Sedell, and D. M. Solis.  1993.  Viability
assessments and management considerations for species associated with late-successional
and old-growth forests of the Pacific Northwest.   USDA Forest Service, U.S. Govt. Print.
Office, Washington, D.C.   530 pp.

==================================
 

SELECTED REFERENCES ON EXPERT PANELS -- METHODS AND EXAMPLES

Bruce G. Marcot
Research Wildlife Ecologist
USDA Forest Service
Pacific Northwest Research Station
Portland Forestry Sciences Lab

11 March 1998
 

Richey, J. S., R. R. Horner, and B. W. Mar.  1985.  The Delphi technique in environmental
assessment.  II.  Consensus on critical issues in environmental monitoring program design.
J. Env. Managem. 21:147-159.

Richey, J. S., B. W. Mar, and R. R. Horner.  1985.  The Delphi technique in environmental
assessment.  I.  Implementation and effectiveness.  J. Env. Managem. 21:135-146.

Richter, B. D., D. P. Braun, M. A. Mendelson, and L. L. Master.  1997.  Threats to imperiled
freshwater fauna.  Cons. Biol. 11(5):1081-1093.

Schuster, E. G., S. S. Frissell, E. E. Baker, and R. S. Loveless.  1985.  The Delphi method:
application to elk habitat quality.  USDA Fores Service Research Paper INT-353.   32 pp.

Swanston, D. N., C. G. Shaw, III, W. P. Smith, K. R. Julin, G. A. Cellier, and F. H. Everest.
1996.  Scientific information and the Tongass Land Management Plan: key findings derived
from the scientific literature, species assessments, resource analyses, workshops, and risk
assessment panels.  USDA Forest Service General Technical Report PNW-GTR-386.
Portland OR.   30 pp.

Zuboy, J. R.  1981.  A new tool for fishery managers: the Delphi technique.  No. Amer. J.
Fish. Manage. 1:55-59.