Quantitative measures of science give the impression ob being more objective,
more precise and more reliable than information of a qualitative nature.
However, validity, reliability and relevance should be cheked
when trying to quantify science.
Reliability
A measure is said reliable if repeated measurements of a phenomenon
yield a similiar outcome.
Any bibliometric measurement incorporates a certain amount of unsystematic "random"
error (such as misspellings of author names)
but systematic errors should be avoided (for example, when not all important
researchers or journals in a field are considered)
The concept of reliability may be defined as the extent to which the results
are independent of their technical calculation.
Validity
Do measurements really measure what they intend to measure?
The validity of bibliometric variables as indicators of the quality of the scientific
research performance is especially questioned.
Relevance
Relevance is concerned with limitations or extensions in the range
of application.
Bibliometrics is founded upon the assumption that scientific publications represent
the total outcome of scientific research activity.
Relevance can be questioned in those subfields of scientific activity that are
characterized by local orientation of research activities, non-journal publishing
practices and insufficient reliance on references to acknowledge intellectual
debts.
Tijssen stresses that any bibliometric measure will only remain a proxy for mapping and assessing the complex system of knowledge production.
Four major issues are related to the validity, reliability
and relevance of bibliometric analyses:
1) completeness of bibliometric data
Especially if the units of analysis are quite small, even small errors or a
small number of omissions can lead to dramatic differences in results and interpretations
of bibliometric indicators.
2) coverage of scientific literatur databases
Problems of inadequate coverage arise particularly in scientific fields in which
researchers do not publish primarily in a limited number of internationally
oriented, and consequently mainly English-language, top journals, but rather
in a wide rangeof journals with a national scope, which may be nonetheless of
high quality.
3) limitations to the use of citations
Problems concerning the use of citation data:
a) The problem of multiple authorship
There are three ways to deal with this problem: 1) straight count (only the
first author receives all the credit), 2) normal count (full credit to all contributors)
and 3) adjusted or fractional count (every coauthor is assigned a fraction of
the authorship)
b) The problem of self-citations
c) The problem of unique identification of authors
d) implicit citations (eponyms)
e) Fluctuations in time
f) Variations across (sub)fields
g) Errors: Citation analyses can be no more accurate than the raw material used
4) the problem of statistics
This problem deals with the question when a given difference between two bibliometric
analyses should be considered as significant with a certain probability and
when it should be ascribed to mere chance. Small differences shoulld be interpreted
with great care.
R. J. W. TIJSSEN, Cartography of Science: Scientometric Mapping with Multidimensional
Scaling. Methods. Thesis, Leiden 1992