(2020-2022)
Acronym ExSciVir
Funding European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement Nº 801509 TÜBİTAK Co-Funded Brain Circulation Scheme 2, Project No: 120C064
Investigator Duygu Uygun-Tunç
Project Mentors Murat Baç; Duncan Pritchard
Host Institutions METU; UC, Irvine
Summary
Several scientific fields are currently facing a replicability and credibility crisis that concerns the reproducibility of research findings, reliability of research methods, integrity of research practices, responsible dissemination of scientific outcomes and the role of scientific institutions. This crisis has led to an extensive and multifaceted meta-scientific debate on scientific normativity. Technologically enhanced research, verification and evaluation processes and large-scale scientific collaboration characterize the newly emerging scientific values and norms. The project Extended Scientific Virtue addresses this debate from a novel epistemological perspective that will equally benefit theorizing in several areas of epistemology. The project will develop an account of scientific virtue that extends beyond the qualities of the individual scientist to include those of the social and technological environment in which scientific inquiry takes place. In particular, it proposes a novel, anti-individualistic account of epistemic virtue that will be applicable on individual, group and system levels. This account will enable characterization of technologically and socially extended scientific expertise and formulation of an account of virtuous epistemic systems. By connecting work on extended cognition and virtue epistemology with current meta-scientific debates and concerns deriving from the replicability and credibility crisis, the proposal will not only do philosophically important work at the forefront of epistemology and philosophy of science, but also address broader issues concerning science policy and the role of scientific institutions.
Project Description
Introduction and the state-of-the-art
The values and norms of scientific inquiry are being revisited and partially reformulated in the contemporary context, where the growth of scientific knowledge depends ever increasingly on technologically enhanced research skills and large-scale scientific communication and collaboration. The latter shed new light on the reliability of the processes and credibility of the outcomes of research, as evidenced by the augmenting volume of self-reflexive discussions addressing a reproducibility[1][2] or a credibility crisis.[3] The widespread nature of questionable research practices,[4][5] underpowered studies[6] and scientific misconduct[7] raise questions regarding the adequacy of the internal quality-control system of science. Technological integration and large-scale collaboration also open possibilities for novel solutions, as addressed by emerging fields such as meta-science[8] and new initiatives such as Open Science (OS).[9][10] This heightened interest in the normative aspect of science brings into focus novel values such as transparency and openness, and novel standards such as pre-registration.[11]
As scientific normativity has moved to the very center of the debate on the scientific front, epistemic normativity has become a major topic in the relatively new field of virtue epistemology (VE). VE maintains that intellectual virtues play a significant, even primary role in a normative account of knowledge and other epistemic states. Although scientific normativity has traditionally been conceived as a specific case of epistemic normativity, currently there is hardly any connection between these two debates.[12] The notion of epistemic virtue, on the other hand, offers a promising perspective that can contribute to establishing a substantial and mutually beneficial interaction between epistemology and empirical research with regard to the current challenges and opportunities facing the scientific community.
There is, however, a theoretical impediment to building such a bridge. While epistemology has traditionally been individualistic, scientific inquiry is inherently social[13] and often technologically assisted.[14] Epistemological individualism typically implies that knowledge or other epistemic states are attributed exclusively to individuals. VE also shares this commitment in that intellectual virtues are conceived as cognitive traits and abilities constitutive of valuable epistemic states like knowledge or justified belief.[15] Epistemological individualism prevents VE’s expansion to the field of scientific practice, where collective knowledge is already a meaningful notion and social factors such epistemic authority, incentive structures, or distribution of scientific effort are of crucial and increasing importance. In particular, the OS framework places the focus on system level values and policies as much as it does on the virtues of the individual scientist. Science is thoroughly social, moreover, in the more basic sense that scientists, research teams and institutions act primarily in their capacity as informants and scientific knowledge is, to a significant extent, a public good. Epistemic individualism is even more problematic with regards to the integration of processes of inquiry with technological tools. While epistemology centers around unaided agents, contemporary scientific research features technological elements such as learning algorithms, data repositories, statistical analysis tools or large computational networks to an even greater extent than agential ones. Epistemology of science thus needs to pay special attention to instrumentally and mathematically extended scientific methods employed to acquire and evaluate knowledge.
ExSciVir will develop a virtue epistemology that can contribute to the identification, understanding and realization of the contemporary and emerging norms of scientific inquiry. To this aim, it undertakes to investigate technologically extended and socially distributed epistemic states. This will provide us with a novel account of technologically and socially extended scientific knowledge in terms of two key concepts: extended epistemic competence and distributed epistemic responsibility. Extended competence will shed light to what the integration of agential and technological elements implies for scientific expertise and methodological reliability. Distributed responsibility will help conceive how credibility and accountability of agents, groups and institutional structures are interrelated in the context of socio-technical extension. These concepts will be used in turn to develop an account of collective knowledge and virtuous epistemic systems. This account will enable an analysis of system level, specifically scientific virtues such as transparency, openness and collaboration which begin to arise out of meta-scientific studies. Last but not least, ExSciVir will make a philosophical contribution to OS debates on promoting such virtues.
Virtue epistemology
Intellectual virtue is one of the oldest philosophical notions. It broadly denotes the intellectual capacities that enable one to arrive at truth. Although virtue-oriented theories enjoyed a more continuous existence in ethics, virtue was hardly a theme in epistemology until a few decades ago. Beginning with the works of Sosa,[16][17][18] Montmarquet[19] and Zagzebski[20] it has once more moved to the very center of contemporary epistemology. VE, across all orientations, stands out as a distinctively normative approach to epistemology, which concentrates its efforts on understanding the values and norms pertaining to intellectual practices as such. In so doing, it argues that the normativity of knowledge should be grounded in the virtuous intellectual qualities of epistemic agents. In other terms, epistemic success or failure should be explained in terms of intellectual virtue or lack thereof. Beyond this basic consensus, two central notions characterize theorizing in the field: reliability and responsibility. Reliability is the truth-conduciveness of an epistemic process, which is often associated with an externalist approach to justification; that is, with the view that justificatory grounds can be external to epistemic agents. Virtue reliabilists conceive epistemic virtues as natural or deeply entrenched, reliable cognitive faculties and abilities such as good memory, acute perception or capacity for rational inference.[21][22][23] Epistemic responsibility, on the other hand, is linked to wider normative notions such as epistemic duty and obligation,[24] or trustworthiness.[25] It is generally related more intimately with an internalist approach; that is, with the view that consciously available grounds are required for justification. From a virtue responsibilist perspective, intellectual virtues comprise epistemically praiseworthy character traits such as open-mindedness, perseverance or intellectual courage.[26][27][28]
The notions of reliability and responsibility are indeed of utmost relevance to the virtues and vices of scientific practice. Although VE construes reliability and responsibility along individualistic lines, scientific misconduct such as data fabrication or questionable research practices such as data dredging and selective reporting of results or studies may arise due to a lack of reliability and/or irresponsibility not only on the individual, but also on group and system levels, for instance through publication bias.[29] Moreover, scientific inquiry is epistemically dependent on the social environment and technological instruments to such an extent that individual intellectual virtue cannot be the sole explanation for scientific success or failure. A most pertinent example is the prominent psychologist Daryl Bem’s research on extrasensory perception (ESP), the ability to sense future events, which was one of the most influential cases to set off the debate on a credibility crisis in psychology in 2011.[30][31] Bem’s numerous experiments and statistical analyses were well justified on the mainstream standards of methodological rigor at the time, he was an established expert and manifested exemplary character virtue though his insistence on replication and transparency, and his results were accepted for publication by the JPSP, the top-ranking journal of the American Psychological Association. The extensive debate on justification the ESP research inaugurated has eventually gave rise to mainstream recognition of p-hacking and selective reporting as unreliable research practices and made a strong case for pre-registration of research hypotheses. The case demonstrates how individual virtue can be insufficient for epistemic success and how epistemic responsibility is distributed, especially when epistemic competence is extended. Thus, we need an anti-individualistic virtue epistemology that cuts across the reliabilist/responsibilist and externalist/internalist distinctions.
Extended and distributed virtue
VE recently undertook a first step in an anti-individualistic direction by expanding epistemology towards highly pertinent debates in cognitive science and philosophy of mind on the question of extended and distributed cognition.[32][33][34] In particular, two recent volumes, Extended Epistemology and Socially Extended Epistemology, constitute a significant move in this regard.[35] A parallel development is the case also on the philosophy of science front.[36][37] The central idea uniting the research on various forms of extended and distributed cognition is that cognitive processes are embedded in their socio-technical environments and cognitive agents characteristically integrate or couple their organismic cognitive operations with those of other individuals, informants and environmental resources like cultural and technological artifacts.[38][39][40] With the increasing dependence of scientific research on socio-technical networks, the case for extended cognition will arguably only become stronger.
Extended VE is on the right track to investigate the epistemological implications of technologically enhanced and collaboratively realized cognitive processes. Not only the constituents of an agent’s cognitive character, but certain properties of the material and social environment sufficiently integrated with the former can contribute to virtue. While extended VE can incorporate cognitive extension and distribution, however, the primarily social and institutional aspects of the production and dissemination of scientific knowledge is beyond its current purview. This latter dimension has been traditionally investigated by disciplines such as sociology of knowledge and sociology of science, which have nonetheless but limited bearing on strictly epistemological questions.[41] The insights of extended VE, consequently, need to be re-evaluated and further developed within the context of a social epistemology.
Social epistemology
Social epistemology (SE), under this particular name, is a relatively new field in epistemology that studies the social dimension of doxastic processes, epistemic justification and dissemination of information. Although these topics have been addressed, among others, from sociological and political perspectives, SE is a normative, rather than descriptive, field in connection with the topics of traditional epistemology.[42][43] It can thus promise a distinctively epistemic perspective on scientific practices through keeping the focus on the normative evaluation of doxastic processes along traditional dimensions such as justifiedness or rationality, while re-conceiving the former as significantly social phenomena. As scientific knowledge is becoming increasingly social through the transition towards larger and more heterogeneous research teams and institutionalized reward structures, epistemology has to keep up by incorporating the social environment into its account of knowledge and other epistemic standings. Extended VE can establish the required theoretical bridge between traditional epistemology and social epistemology of science, if the epistemic environment within which cognitive processes are embedded is conceived in a way that comprises the interpersonal, institutional and systemic factors that influence the quality of epistemic outcomes. Moreover, a social epistemology of science formulated in extended VE terms can account for socially extended virtues (and vices) and virtuous (or vicious) epistemic systems.
Objectives and overview of the action
The overarching objective of the project is to formulate an account of virtuous scientific practice on the basis of extended competence and distributed responsibility, and to apply this account to the current debates on scientific normativity. This main objective has both a theoretical and an applied dimension. The theoretical dimension aims to formulate an account of extended competence and distributed responsibility, which will in turn allow for a characterization of scientific expertise, collective knowledge and virtuous epistemic systems. Beyond this theoretical investigation, the project will demonstrate how these concepts can be applied in evaluating the effect of system level interventions on the epistemic value of individual or group level performance. This applied dimension will contribute to the meta-scientific discussion of the efficiency and responsibility of research practices and institutional structures. These objectives will be realized in three work packages: WP1: Epistemic competence and responsibility, WP2: Collective knowledge and virtuous systems, WP3: Virtuous scientific practice.Three work packages consist of seven research objectives, which are explicated in the approach and methodology section below.
Approach and methodology
ExSciVir will follow mostly the standard methods of analytic epistemology like conceptual analysis and formulating definitions. It will expand the scope of conventional epistemological analysis, however, by examining the social dimension of cognition as it is done in SE. Moreover, in line with the interdisciplinary nature of epistemology of science, it will examine scientific cases as customary in PoS. In particular, it will draw on empirical findings from meta-science and OS literature and epistemologically evaluate the proposed standards and norms of justification. The objectives outlined will be pursued over the course of three project stages:
Stage 1: Extended virtue epistemology The first project stage addresses the extant literature on extended and socially extended VE with a view to defining epistemic competence and responsibility from the proposed perspective. The first major task is to formulate a definition of extended epistemic competence (WP1.i). The definition will refer to the reliability of epistemic performance in relation to a specific field, which is grounded in the reliability of particular abilities, relevant background beliefs, employed methods and tools as well as in a sensitivity to changes in the fit between the demands of the epistemic situation and the abilities, methods and tools. The resulting account of epistemic competence incorporates elements from both virtue reliabilism and responsibilism. It is virtue reliabilist in that it evaluates intellectual virtue in terms of truth-conduciveness, and virtue responsibilist in that it that it recognizes the role of agential factors such as assessing the epistemic environment to evaluate abilities and methods for reliability. The definition of epistemic competence will draw on certain elements of Sosa’s characterization of intellectual virtue in relation to a specific environment, field of propositions and set of conditions.[44] By involving virtuous performance as opposed to mere capacity, however, epistemic competence is a stronger condition of knowledge than most reliabilist accounts of epistemic virtue. In difference to Sosa, it incorporates an element of sensitivity towards the situational and environmental factors as part of the epistemic performance manifesting virtue. Moreover, when so defined, epistemic competence is a combination of several virtues, thus more specific than any single dispositional virtue such as good memory, but more coarsely individuated than particular epistemic processes such as using a specific data analysis method. Subsequently, the advantages of the proposed account over strongly individualist virtue epistemologies will be defended vis-à-vis epistemic situationism[45][46] and the generality problem[47] or the question of how to individuate intellectual virtues.[48][49] The next step is to analyze the concept of extended virtue in terms of necessary conditions for extension. The pertinent research question is what, exactly, is extended: virtuous epistemic processes, intellectual dispositions, agency or credit for epistemic success? The second major task of this project stage is to define epistemic responsibility in relational terms, as informant-reliability, and formulate an account of distributed responsibility (WP1.ii). The account of epistemic responsibility will incorporate virtues relevant to being a reliable information source. Epistemic responsibility conceived as informant-reliability can potentially be explicated in externalist terms, hence can be more relevant to the kind of objective epistemic justification science is primarily concerned with. The general “theme” of the virtues investigated will thus be reliability as truth-conduciveness. The definition of responsibility will draw on certain elements of Craig’s notion of a good informant.[50] I will demonstrate that competence may not be sufficient for responsibility in this sense. With respect to the distribution of responsibility, my research hypothesis is that epistemic responsibility does not extend to include non-agential elements but can be distributed over multiple agents as well as institutions. The pertinent research question is what, exactly, is distributed: accountability or epistemic duties that attach to the formation of justified beliefs? Further, I will demonstrate that extended competence implies distributed responsibility, especially when the reliability of the integrated elements refers to social grounds such as expert authority or testimony.
Stage 2: Social epistemology of science This project stage focuses on SE as it applies to the domain of scientific inquiry. The first aim of this stage is to evaluate the epistemic role of individual scientists and scientific institutions as expert informants, not only vis-à-vis the greater public but also other scientists (WP2.i). The extant literature in SE on testimony, credibility, epistemic trust and authority is pertinent in this context, because any research relies significantly both on previous results and ongoing collaboration without seeking independent, sufficient external justification for all derived information. The hypothesis is that expert testimony in science and research collaboration should be understood in terms of indirect and direct epistemic dependence,[51][52][53] which in turn makes a case for weak and strong versions of distributed knowledge.[54] Moreover, it will be argued that such an understanding is compatible with VE formulated in anti-individualist and externalist terms. The second aim is to analyze the concept of collective knowledge in relation to extended competence and distributed epistemic responsibility, as defined in the previous stages (WP2.ii). The research hypothesis is that we can speak of collective knowledge when there is a network of direct epistemic dependencies in a group, which implies strong distributed competence as well as distributed responsibility. The final aim is to analyze the concept of an epistemic system (WP2.iii). Epistemic systems will be characterized, in line with Goldman, as social processes that affect the epistemic performance of its members and the epistemic value of their doxastic attitudes through their institutions and/or patterns of interpersonal influence.[55][56] I will discuss, in particular, whether we can and should speak of the reliability of entire scientific fields, conceived as social processes, in terms of their overall epistemic success. Meta-scientific research engages in such assessments, for instance through large-scale replication attempts, and makes a case for their normative value. Further, I will propose an epistemological account of system-level distribution of epistemic responsibility, with respect to the influence of institutional factors such as incentivisation on the larger network of indirect epistemic dependencies.
Stage 3: Values and norms of scientific practice This final project stage will apply the hitherto analyzed concepts in a virtue epistemological investigation of meta-scientific results and propositions in five areas; namely methodology, reporting (research communication), reproducibility (research verification), evaluation (quality assessment) and incentives.[57] I will particularly focus on how technological extension of competence and distribution of responsibility can increase reproducibility and facilitate self-correction of the literature. The first task of this stage is to characterize how specifically scientific virtues on the individual and system level are related to the general intellectual virtues as discussed in VE (WP3.i). On the individual level, I will investigate how good scientific conduct and reliable research practices, such as good experimental design, collecting representative samples and disseminating research results responsibly, are related to epistemic virtues. My research hypothesis is that extended epistemic competence and distributed epistemic responsibility are both better suited as epistemic concepts that can be applied to the scientific domain and can function as a theoretical bridge between the general intellectual virtues of epistemology and the specific scientific virtues. The major system level scientific virtues that the project will focus on are transparency, openness and collaboration.[58] The second aim, accordingly, is to investigate how system level scientific virtues can promote good scientific conduct and reliable research practices, and thereby have a bearing on the questions of reproducibility and credibility (WP3.ii). The account of virtuous systems developed in the previous stage will be applied to the current attempts to promote OS. In particular, OS proposals such as pre-registration of research hypotheses and methods and publication of registered reports in journals will be investigated as system-level policies that can affect the reliability of individual and group-level epistemic performance in the first two areas outlined above, conducting and communicating research, by distributing epistemic responsibility. The pertinent research question is whether the organization of the epistemic system can override or minimize demands on individual virtue.
Originality and innovative aspects
The role of epistemic virtues in science, especially in the contemporary context, has so far not been adequately investigated and when done so, with an exclusive focus on the epistemic character of the individual scientist.[59] Moreover, partly due to reasons addressed in the preceding, epistemology has not yet addressed the normativity debate in science inaugurated by meta-scientific studies and the OS movement. By establishing such a theoretical bridge through the concept of extended scientific virtue, ExSciVir will make an original contribution to epistemology and PoS. In particular, it proposes a novel, anti-individualistic account of epistemic virtue that will be applicable on individual, group and system levels. Most importantly, through the concepts of extended competence and distributed responsibility, system level scientific virtues can enter the purview of epistemology and epistemic virtues can gain purport with regards to actual scientific practice, its evaluation and amelioration.
Outcomes
Papers
Uygun Tunç, D. (in press). We Should Redefine Scientific Expertise: An Extended Virtue Account. European Journal for Philosophy of Science. Preprint: http://philsci-archive.pitt.edu/id/eprint/21360
Uygun Tunç, D., Tunç, M. N., & Eper, Z. B. (in press). Is Open Science Neoliberal? Perspectives on Psychological Science. Preprint: https://doi.org/10.31234/osf.io/ft8dc
Uygun Tunç, D. & Tunç, M. N. (in press). A Falsificationist Treatment of Auxiliary Hypotheses in Social and Behavioral Sciences: Systematic Replications Framework. Meta-Psychology. Preprint: https://psyarxiv.com/pdm7y/
Lakens, D., Uygun Tunç, D., & Necip Tunç, M. (2022). There is no generalizability crisis. Behavioral and Brain Sciences, 45, E25. https://doi.org/10.1017/S0140525X21000340
Uygun Tunç, D. The Subject of Knowledge in Collaborative Science. Preprint: http://philsci-archive.pitt.edu/id/eprint/20132 (accepted pending revisions in Synthese)
Uygun Tunç, D., & Pritchard, D. Collective Epistemic Vice in Science: Lessons from the Credibility Crisis. Preprint: http://philsci-archive.pitt.edu/id/eprint/21120 (under review in Synthese)
Uygun Tunç, D., Tunç, M. N., & Lakens, D. The Epistemic and Pragmatic Function of Dichotomous Claims Based on Statistical Hypothesis Tests. Preprint: https://doi.org/10.31234/osf.io/af9by (under review in Theory & Psychology)
Uygun Tunç, D., Tunç, M. N. We Need a Copernican Revolution in Modelling Scientific Expertise. Preprint: https://psyarxiv.com/agk4b (under review in Social Psychology Bulletin)
Talks
Collective Epistemic Vice in Science: Lessons from the Credibility Crisis
Seminar: Scientific Progress – Individual and Collective Amsterdam, 24-26 August 2022
Is Open-Science Neoliberal?
Online Symposium on Values in Medicine, Science, and Technology Dallas, 15-19 August 2022
What is an Informative Replication?
Collective scientific knowledge without a collective subject
Bilkent Social Ontology Workshop Ankara, 17-18 April 2021
Replication under Underdetermination: Introducing Systematic Replications Framework
Models Theories Research Programs (Boğaziçi University) Project Colloquium Istanbul, 20 May 2020
References
[1] Camerer, C. F. et al. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637–644. DOI: https://doi.org/10.1038/s41562-018-0399-z
[2] Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science,349(6251), aac4716.
[3] Ioannidis, J. P. A. (2005) Why most published research findings are false. PLoS Med, 2(8): e124.
[4] John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532.
[5] Fraser, H., Parker, T., Nakagawa, S., Barnett, A., Fidler, F. (2018). Questionable Research Practices in Ecology and Evolution. PLoS ONE, 13(7): e0200303.
[6] Ioannidis, J. P. A., Stanley, T. D., & Doucouliagos, H. (2017). The power of bias in economics research. Econ J, 127, F236-F265.
[7] Fanelli, D. (2009) How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data. PLoS ONE, 4(5): e5738.
[8] Ioannidis, J. P. A., Fanelli, D., Dunne, D. D., & Goodman, S. N. (2015) Meta-research: Evaluation and Improvement of Research Methods and Practices. PLoS Biol, 13(10): e1002264.
[9] Munafò, M. R. et al. (2017). A manifesto for reproducible science. Nature human behaviour, 1(1), 0021. DOI: https://doi.org/10.1038/s41562-016-0021
[10] Nosek, B. A. et al. (2015). Promoting an open research culture. Science, 348(6242), 1422 LP – 1425. DOI: https://doi.org/10.1126/science.aab2374
[11] Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600-2606.
[12] For an exploratory connection between VE and PoS that, however, does not address the problems I indicate below, see A. Fairweather (Ed.). (2014). Virtue Epistemology Naturalized: Bridges Between Virtue Epistemology and Philosophy of Science. Synthese Library, vol 366. Springer, Cham.
[13] See e.g. Longino, H. (1990). Science as Social Knowledge: Value and Objectivity in Scientific Inquiry. Princeton: Princeton UP.
[14] See e.g. Humphreys P. (2004) Scientific Knowledge. In I. Niiniluoto, M. Sintonen, J. Woleński (Eds.), Handbook of Epistemology. Dordrecht: Springer.
[15] Kallestrup, J. And Pritchard, D. (2012). Robust virtue epistemology and epistemic anti‐individualism. Pacific Philosophical Quarterly, 93, 84-103.
[16] Sosa, E. (1980). The raft and the pyramid: Coherence versus foundations in the theory of knowledge. In P. A. French, T. E. Uehling Jr., & H. K. Wettstein (Eds.), Studies in Epistemology. Minneapolis: University of Minnesota Press.
[17] Sosa, E. (1991). Knowledge in Perspective. Cambridge: Cambridge UP.
[18] Sosa, E. (2007). A Virtue Epistemology: Apt Belief and Reflective Knowledge. Oxford UP.
[19] Montmarquet, J. A. (1987). Epistemic Virtue. Mind, 96(384): 482–497.
[20] Zagzebski, L. T. (1996). Virtues of the mind: An inquiry into the nature of virtue and the ethical foundations of knowledge. Cambridge UP.
[21] Sosa, E. (2007). A virtue epistemology.
[22] Greco, J. (1993). Virtues and vices of virtue epistemology. Canadian Journal of Philosophy, 23(3), 413–432.
[23] Pritchard, D. (2012). Anti-luck virtue epistemology. The Journal of Philosophy, 109(3), 247–279.
[24] BonJour, L. (1985). The Structure of Empirical Knowledge. Harvard UP.
[25] Lehrer, K. (2001). The virtue of knowledge. In A. Fairweather & L. Zagzebski (Eds.), Virtue epistemology: essays on epistemic virtue and responsibility. New York: Oxford UP.
[26] Zagzebski, L. T. (1996). Virtues of the Mind. Heather Battaly, “Virtue Epistemology,” Philosophy Compass 3, no. 4 (2008): 639–63.
[27] Baehr, J. (2011). The Inquiring Mind: On Intellectual Virtues and Virtue Epistemology. New York: Oxford UP.
[28] Battaly, H. (2008). Virtue epistemology. Philosophy Compass, 3(4), 639–663.
[29] Sterling, T. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance−or vice versa. Journal of the American Statistical Association, 54(285), 30-34.
[30] Bem, D. J. (2011). Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100(3), 407–25.
[31] For one of the earliest, most influential rebuttals, see Wagenmakers, E. J., Wetzels, R., Borsboom, D. & van der Maas, H. L. J. (2011). Why psychologists must change the way they analyze their data: the case of psi: comment on Bem (2011). Journal of Personality and Social Psychology, 100(3), 426-432.
[32] Pritchard, D. (2010). Cognitive ability and the extended cognition thesis. Synthese, 175(1), 133–151.
[33] Carter, J. A., Kallestrup, J., Palermos, S. O., & Pritchard, D. (2014). Varieties of externalism. Philosophical Issues, 24(1), 63–109.
[34] Pritchard, D. (2018). Extended virtue epistemology. Inquiry, 61(5–6), 632–647.
[35] Carter, J. A., Clark, A., Kallestrup, J., Palermos, S. O., & Pritchard, D. (Eds.). (2018). Extended Epistemology. Oxford UP.
Carter, J. A., Clark, A., Kallestrup, J., Palermos, S. O., & Pritchard, D. (Eds.). (2018). Socially Extended Epistemology. Oxford UP.
[36] Bird, A. (2014). When is there a group that knows? Distributed cognition, scientific knowledge, and the social epistemic subject. In J. Lackey (Ed.), Essays in Collective Epistemology. Oxford: Oxford UP.
[37] Giere, R. (2002). Scientific cognition as distributed cognition. In P. Carruthers, S. Stich, & M. Siegal (Eds.), The Cognitive Basis of Science. Cambridge UP.
[38] Hutchins, E. (1995). Cognition in the Wild. MIT Press.
[39] Clark, A. (1996). Being There: Putting Brain, Body, and World Together Again. MIT Press.
[40] Clark, A., & Chalmers, D. (1998). The extended mind. Analysis, 58(1), 7–19.
[41] See e.g. Merton, R. K. (1973) [1942]. The Sociology of Science: Theoretical and Empirical Investigations. University of Chicago Press.
[42] Goldman, A. I. (1999). Knowledge in a Social World. Oxford UP.
[43] Goldman, A. I. (2010). Why social epistemology is real epistemology. In A. Haddock, A. Millar, & D. Pritchard (Eds.), Social Epistemology. Oxford UP.
[44] Sosa, E. (1991). Knowledge in Perspective, 284.
[45] Alfano, M. (2011). Expanding the situationist challenge to responsibilist virtue epistemology. The Philosophical Quarterly, 62(247), 223-249.
[46] Olin, L., & Doris, J. M. (2014). Vicious minds: Virtue epistemology, cognition, and skepticism. Philosophical Studies, 168(3), 665-92.
[47] Pollock, J. L. (1984). Reliability and justified belief. Canadian Journal of Philosophy, 14(1), 103-114.
[48] Goldman, Alvin (1986). Epistemology and Cognition. Cambridge, MA: Harvard UP.
[49] Zagzebski, L. T. (1996). Virtues of the Mind.
[50] Craig, E. (1990). Knowledge and the State of Nature: An Essay in Conceptual Synthesis. Oxford: Oxford UP.
[51] Goldberg, S. (2011). The division of epistemic labor. Episteme, 8(1), 112–125.
[52] Hardwig, J. (1985). Epistemic dependence. The Journal of Philosophy, 82(7), 335–349.
[53] Pritchard, D. (2015), Epistemic dependence. Philosophical Perspectives, 29, 305-324.
[54] See Pritchard, D. & Palermos, S. O. (2013). Extended Knowledge and Social Epistemology. Social Epistemology Review and Reply Collective, 8, 105-120.
[55] Goldman, A. I. (2011). A guide to social epistemology. InA. I. Goldman & D. Whitcomb (Eds.), Social epistemology: essential readings. Oxford UP.
[56] See also, Koppl, R. (2006). Epistemic systems. Episteme, 2(2), 91-106.
[57] Ioannidis, J. P. A., Fanelli, D., Dunne, D. D., & Goodman, S. N. (2015) Meta-research: Evaluation and Improvement of Research Methods and Practices. PLoS Biol, 13(10): e1002264.
[58] Cf. sociologist Merton’s categorization of science’s institutional norms, “communism, universalism, disinterestedness, and organized skepticism” in The Sociology of Science.
[59] See e.g. Pennock, R. T. (2019). An Instinct for Truth: Curiosity and the Moral Character of Science. Cambridge, MA: MIT Press.
Duygu Uygun-Tunç 