Presentation is loading. Please wait.

Presentation is loading. Please wait.

The Place of Ratings and Reputational Tools in Knowledge Organization

Similar presentations


Presentation on theme: "The Place of Ratings and Reputational Tools in Knowledge Organization"— Presentation transcript:

1 The Place of Ratings and Reputational Tools in Knowledge Organization
On the Epistemic Value of Reputation:  The Place of Ratings and Reputational Tools in Knowledge Organization 11th International ISKO Conference Rome, February Gloria Origgi & Judith Simon Institut Jean Nicod ENS-EHESS-CNRS Paris, France Vorstellung, Name, Interesse, Dank… Erweiterung Epistemisch Transdisziplinär

2 REPUTATION: Overview Background Introduction Reputation as Evaluative Social Information A Rational Model for the Epistemic Use of Reputation Reputational Tools on the Web Problems with the Epistemic Use of Reputation Conclusions

3 Guiding Questions REPUTATION: Overview
How to use reputation for epistemic purpose? What’s the epistemic value of reputation? Is this a good thing or a bad thing?

4 REPUTATION: Background: LiquidPub
Different methods of quantifying, assessing & propagating reputation I don’t present results, this is rather an ongoing theoretical inquiry in the epistemic use of reputation Reputation propagation: upwards & downwards :journal  paper, institution  person  institution Different proxies, how to combine and weight them Further Information on:

5 REPUTATION: Background
Thesis: Types of Epistemic Sociality I don’t present results, this is rather an ongoing theoretical inquiry in the epistemic use of reputation

6 REPUTATION: Overview Background Introduction Reputation as Evaluative Social Information A Rational Model for the Epistemic Use of Reputation Reputational Tools on the Web Problems with the Epistemic Use of Reputation Conclusions

7 REPUTATION: Introduction
Hero Sinner Drunkard Reputation as Heuristic: Reputation as a way to classify social types within the community that will allow its member to manage their relations with others, to make inferences and predictions about their behavior, i.e. to construct a basic "social map" that will help them orient in their society. Source:

8 REPUTATION: Introduction
Hero Sinner Drunkard Morally questionable! CENSORED! Reputation as Heuristic: Reputation as a way to classify social types within the community that will allow its member to manage their relations with others, to make inferences and predictions about their behavior, i.e. to construct a basic "social map" that will help them orient in their society. Morally this may be questionable, but epistemologically it can be useful. Source:

9 REPUTATION: and Knowledge Organization
Reputation as social information about the value of people, systems and processes that release information. Focus: relationship between this special form of social information and the processes of knowledge organization and evaluation. More precisely, we argue not only that (1) we make use of other people's reputations to evaluate information in various ways (2) within systems, like the Web, that make possible the easy and dynamic organization and re-organization of knowledge, we also use our own rankings to determine new content and generate new categories.

10 REPUTATION: Overview Background Introduction Reputation as Evaluative Social Information A Rational Model for the Epistemic Use of Reputation Reputational Tools on the Web Problems with the Epistemic Use of Reputation Conclusions

11 REPUTATION: as Evaluative Social Information
Reputation is the informational track of our past actions, it is the credibility that an agent or an item earn through repeated interactions. Reputational Cues are indicators/proxies of reputation where quality of objects or agents cannot be directly assessed Relying on reputational cues is an efficient way of shaping the too rich informational landscape around us by creating new relevant categories. epistemological prediction is that the higher is the uncertainty on the content of information, the stronger is the weight of the opinions of others in order to establish the quality of this content.

12 REPUTATION: as Evaluative Social Information
In an information-dense environment, where sources are in constant competition to get attention and the option of the direct verification of the information is often simply not available at reasonable costs, evaluation and rankings are epistemic tools and cognitive practices that provide an inevitable shortcut to information Modest Prediction: The higher the uncertainty on the content of information, the stronger is the weight of the opinions of others in order to establish the quality of this content. epistemological prediction is that the higher is the uncertainty on the content of information, the stronger is the weight of the opinions of others in order to establish the quality of this content.

13 REPUTATION: Overview Background Introduction Reputation as Evaluative Social Information A Rational Model for the Epistemic Use of Reputation Reputational Tools on the Web Problems with the Epistemic Use of Reputation Conclusions

14 REPUTATION: A Model for the Epistemic Use of Reputation
Lehrer & Wagner (1981) “Rational Consensus in Science and Society” Proposes a model for rational decision making processes in science, society and the arts that makes epistemic use of reputation It rests upon the employment of consensual probabilities, utilities and weights For decision making processes to be rational, it is central that all evidence or empirical information available for the topic of concern has to be used Experimental + Social Information epistemological prediction is that the higher is the uncertainty on the content of information, the stronger is the weight of the opinions of others in order to establish the quality of this content.

15 REPUTATION: A Model for the Epistemic Use of Reputation
Social information = information about the expertise of other experts on issues at hand = Reputation Example: Expert Dilemma: Do we need to vaccinate large parts of the population to prevent a pandemie? Step 1: each expert gives a weights other experts’ competency Step 2: weights are laid open Step 3: revision of own weights taking the others‘ assessment into account depending on the weights assigned to them Repeat cycle till consensus is achieved… Once these consensual weights are achieved, they can be applied to answering the question of concern by weighting each member’s votes on the issue with their consensual personal weight. Compulsory vaccination for every citizen being threatened by the pig flu?

16 REPUTATION: A Model for the Epistemic Use of Reputation
Lehrer & Wagner propose a model of how to rationally reach consensus that rests upon the epistemic use of reputation This implies that reputational information, i.e. social information about other people that is evaluative, is epistemically useful. Do we need such formal models? Epistemic use of reputational cues does not have to follow such a formal method. But on the Web, models similar to this one are embedded and hidden within different applications. Clearly, not all epistemic usage of reputation cues has to follow such a formal method. Quite on the contrary, ratings and other reputational tools might be used in a variety of different ways on the Web and our everyday life more generally. Nonetheless, Lehrer & Wagner’s model delivers a clear example of the potential that reputation understood as social information from an evaluative stance, can have for epistemic tasks ((Lehrer and Wagner 1981)).

17 REPUTATION: Overview Background Introduction Reputation as Evaluative Social Information A Rational Model for the Epistemic Use of Reputation Reputational Tools on the Web Problems with the Epistemic Use of Reputation Conclusions

18 REPUTATION: Reputational Tools on the Web
Digg. Com Reddit.com

19 REPUTATION: Reputational Tools on the Web
What the Web makes possible today is an algorithmic treatment of methods of gathering social information to extract knowledge. Ratings and rankings on the Web are the result of collective human registered activities with artificial devices. However, the control of the heuristics and techniques that underlie this dynamics of information may be out of sight or incomprehensible for the users who find themselves in the very vulnerable position of relying on external sources of information through a dynamic, machine-based channel of communication whose heuristics and biases are not under their control. Thus, the reputational tools that are proliferating on the Web should be scrutinized by epistemically responsible users who do not want to accept too naïvely the outcome of a process they do not control.

20 REPUTATION: Reputational Tools on the Web

21 REPUTATION: Reputational Tools on the Web

22 REPUTATION: Reputational Tools on the Web

23 REPUTATION: Reputational Tools on the Web
Interestingness! “There are lots of elements that make something 'interesting' (or not) on Flickr. Where the clickthroughs are coming from; who comments on it and when; who marks it as a favorite; its tags and many more things which are constantly changing. Interestingness changes over time, as more and more fantastic content and stories are added to Flickr.” There are elements of how to become interesting: what they all have in common that people assess the merits of an object. On the Web ususally via rating or ranking mechanisms. More formalized jury selections or even dictatorial choice would be different options. One research question within Liquidpub indeed concerns the comparison between different types of peer evaluation (e.g. scientific peer review versus amazon rating and ranking mechanisms. LiquidJournal

24 REPUTATION: Reputational Tools on the Web
Interestingness! “There are lots of elements that make something 'interesting' (or not) on Flickr. Where the clickthroughs are coming from; who comments on it and when; who marks it as a favorite; its tags and many more things which are constantly changing. Interestingness changes over time, as more and more fantastic content and stories are added to Flickr.” Interestingness is a new category based on reputational mechanisms, making use of different proxies whose weight and combination is not obvious! There are elements of how to become interesting: what they all have in common that people assess the merits of an object. On the Web ususally via rating or ranking mechanisms. More formalized jury selections or even dictatorial choice would be different options. One research question within Liquidpub indeed concerns the comparison between different types of peer evaluation (e.g. scientific peer review versus amazon rating and ranking mechanisms. LiquidJournal

25 REPUTATION: Reputational Tools on the Web
Reputational tools get more and more central on the Web Rankings and Ratings provide new arrangements of information Early years of 2000: focus on personalized information (My-Features) Now: trend towards systems of shared preferences, were people can rely on others’ preferences and rankings to construct there own access to and categorization of information Examples: Flickr’s Interestingess Twitter-Logic of Followers and Leaders LiquidJournal: people or groups create their own journals by selecting (existing) content and making it available via their selection

26 REPUTATION: Problems for the Epistemic Use of Reputation
So, all is well, or? Ad 1: According to her “testimonial injustice occurs when prejudice causes a hearer to give a deflated level of credibility to a speaker’s word, whereas hermeneutic injustice “[...] occurs at a prior stage, when a gap in collective interpretative resources puts someone at an unfair disadvantage when it comes to making sense of their social experience” ((Fricker 2007) 1). Clearly, both forms of injustice are easily conceivable when reputational cues and their epistemic usage are not critically reflected upon and kept open for constant scrutiny and revision. Ad 2:. Do you use the person’s academic development, his institutional background, some form of communal evaluation, such as ratings or recommendations that he has received from other people as a cue to assess someone’s reputation? Do you rely on your own experience with her? On some indicator of the quality of his former research? On her track record of different academic achievements? Her H-index or impact factor? Which of these proxies are valid and which are not? The second crucial questions concerns the stability of reputation, resp. the way you deal with evidence that supports or contradicts your view on the reputation of others. When, under which conditions and up to which point of counter-evidence or you warranted in keeping your reputation value for someone or something?

27 REPUTATION: Problems for the Epistemic Use of Reputation
So, all is well, or? Well, not quite… Ad 1: According to her “testimonial injustice occurs when prejudice causes a hearer to give a deflated level of credibility to a speaker’s word, whereas hermeneutic injustice “[...] occurs at a prior stage, when a gap in collective interpretative resources puts someone at an unfair disadvantage when it comes to making sense of their social experience” ((Fricker 2007) 1). Clearly, both forms of injustice are easily conceivable when reputational cues and their epistemic usage are not critically reflected upon and kept open for constant scrutiny and revision. Ad 2:. Do you use the person’s academic development, his institutional background, some form of communal evaluation, such as ratings or recommendations that he has received from other people as a cue to assess someone’s reputation? Do you rely on your own experience with her? On some indicator of the quality of his former research? On her track record of different academic achievements? Her H-index or impact factor? Which of these proxies are valid and which are not? The second crucial questions concerns the stability of reputation, resp. the way you deal with evidence that supports or contradicts your view on the reputation of others. When, under which conditions and up to which point of counter-evidence or you warranted in keeping your reputation value for someone or something?

28 REPUTATION: Overview Background Introduction Reputation as Evaluative Social Information A Rational Model for the Epistemic Use of Reputation Reputational Tools on the Web Problems with the Epistemic Use of Reputation Conclusions

29 REPUTATION: Problems for the Epistemic Use of Reputation
The danger of misuse of reputation: danger of epistemic injustice (Fricker 2007), judging epistemic credibility and social identity (Alcoff 2001) Using proxies that are not valid to assess the reputation and epistemic credibility of epistemic agents (gender, race, nationality, institutional background,…) “testimonial injustice occurs when prejudice causes a hearer to give a deflated level of credibility to a speaker’s word” ((Fricker 2007) 1) Ad 1: According to her “testimonial injustice occurs when prejudice causes a hearer to give a deflated level of credibility to a speaker’s word, whereas hermeneutic injustice “[...] occurs at a prior stage, when a gap in collective interpretative resources puts someone at an unfair disadvantage when it comes to making sense of their social experience” ((Fricker 2007) 1). Clearly, both forms of injustice are easily conceivable when reputational cues and their epistemic usage are not critically reflected upon and kept open for constant scrutiny and revision. Ad 2:. Do you use the person’s academic development, his institutional background, some form of communal evaluation, such as ratings or recommendations that he has received from other people as a cue to assess someone’s reputation? Do you rely on your own experience with her? On some indicator of the quality of his former research? On her track record of different academic achievements? Her H-index or impact factor? Which of these proxies are valid and which are not? The second crucial questions concerns the stability of reputation, resp. the way you deal with evidence that supports or contradicts your view on the reputation of others. When, under which conditions and up to which point of counter-evidence or you warranted in keeping your reputation value for someone or something?

30 REPUTATION: Problems for the Epistemic Use of Reputation
2. Limits of the epistemic usefulness of reputation itself How to calculate reputational values in the first place? What are the pros and cons of different methods: e.g. peer review versus Amazon-type ratings? Which proxies should be used and how should they be combined? Stability of reputation over time? Transferability of reputation over domains? 3. Lack of transparency of reputational algorithms and metrics How should users be responsible knowers if they do not understand the functioning, the strengths and weaknesses of different mechanisms? How to detect biases, if the mechanisms are not laid open? Need to make these mechanisms visible and understandable Ad 1: According to her “testimonial injustice occurs when prejudice causes a hearer to give a deflated level of credibility to a speaker’s word, whereas hermeneutic injustice “[...] occurs at a prior stage, when a gap in collective interpretative resources puts someone at an unfair disadvantage when it comes to making sense of their social experience” ((Fricker 2007) 1). Clearly, both forms of injustice are easily conceivable when reputational cues and their epistemic usage are not critically reflected upon and kept open for constant scrutiny and revision. Ad 2:. Do you use the person’s academic development, his institutional background, some form of communal evaluation, such as ratings or recommendations that he has received from other people as a cue to assess someone’s reputation? Do you rely on your own experience with her? On some indicator of the quality of his former research? On her track record of different academic achievements? Her H-index or impact factor? Which of these proxies are valid and which are not? The second crucial questions concerns the stability of reputation, resp. the way you deal with evidence that supports or contradicts your view on the reputation of others. When, under which conditions and up to which point of counter-evidence or you warranted in keeping your reputation value for someone or something?

31 REPUTATION: Overview Background Introduction Reputation as Evaluative Social Information A Rational Model for the Epistemic Use of Reputation Reputational Tools on the Web Problems with the Epistemic Use of Reputation Conclusions

32 REPUTATION: Conclusions
Ratings and reputational tools in knowledge organization have epistemological, practical as well as ethical implications. Epistemological questions: How epistemically warranted is the use of these tools? Practical questions: How to develop these mechanisms? Which proxies to use, how to combine and weigh them? What’s the status of these new types of classes, such as interestingness? Can ratings and ranking serve as middle-ground categorizations? Ethical and political question: Epistemic injustices & lack of transparency: Once reputation mechanisms become formalized and are embedded within tools, there is a clear danger that epistemic injustices are inscribed in and reinforced by technology. Ad epist.: Is it just based on blind and imperfect heuristics that have a serendipitous effect on our search of information, or is it possible to conceive second o A purely epistemological analysis of using reputation for epistemic purposes will not suffice: the goals and standards for knowledge organization and epistemic practices have to be discussed and decided upon taking political and ethical considerations into account. Reputational tools open up new possibilities for knowledge organization, but they also bring with them their own problems. rder epistemic criteria that allow us to pry apart “good” and “bad” practices of trust and reliance on these reputational metrics?

33 REPUTATION: Conclusions
What is the epistemic values of reputation? Is it useful? Or rather dangerous? Ad epist.: Is it just based on blind and imperfect heuristics that have a serendipitous effect on our search of information, or is it possible to conceive second order epistemic criteria that allow us to pry apart “good” and “bad” practices of trust and reliance on these reputational metrics?

34 REPUTATION: Conclusions
What is the epistemic values of reputation? Is it useful? Or rather dangerous? Both - it is useful and dangerous. But either way, reputational information, different reputational proxies and methods of quantifying and combining them are being used extensively on the Web and elsewhere. Ad epist.: Is it just based on blind and imperfect heuristics that have a serendipitous effect on our search of information, or is it possible to conceive second order epistemic criteria that allow us to pry apart “good” and “bad” practices of trust and reliance on these reputational metrics?

35 REPUTATION: Conclusions
What is the epistemic values of reputation? Is it useful? Or rather dangerous? Both - it is useful and dangerous. But either way, reputational information, different reputational proxies and methods of quantifying and combining them are being used extensively on the Web and elsewhere. An additional problem on the Web concerns the lack of visibility: for the users the metrics and algorithms behind different reputation tools are often unknown. Ad epist.: Is it just based on blind and imperfect heuristics that have a serendipitous effect on our search of information, or is it possible to conceive second order epistemic criteria that allow us to pry apart “good” and “bad” practices of trust and reliance on these reputational metrics?

36 REPUTATION: Conclusions
What is the epistemic values of reputation? Is it useful? Or rather dangerous? Both - it is useful and dangerous. But either way, reputational information, different reputational proxies and methods of quantifying and combining them are being used extensively on the Web and elsewhere. An additional problem on the Web concerns the lack of visibility: for the users the metrics and algorithms behind different reputation tools are often unknown. There is an epistemic duty of epistemologists and knowledge organization scholars to thoroughly analyze these different reputational practices from epistemological, ethical and political perspectives. Ad epist.: Is it just based on blind and imperfect heuristics that have a serendipitous effect on our search of information, or is it possible to conceive second order epistemic criteria that allow us to pry apart “good” and “bad” practices of trust and reliance on these reputational metrics?

37 Thank you for your attention!
REPUTATION Thank you for your attention! Contact Judith Simon Institut Jean Nicod Ecole Normale Supérieure 29, rue d'Ulm F Paris www: tel: +33 (0) fax: +33 (0)

38 REPUTATION: Problems for the Epistemic Use of Reputation
Two major problems of using reputation for epistemic purpose the use of reputation to assess content can be epistemically beneficial while being morally questionable 2) limits of the epistemic usefulness of reputation itself

39 REPUTATION: as Evaluative Social Information
We want to explore the epistemic value of reputation, while being aware of the ethical and political problems that might come with using it for epistemic purpose. Using the judgment on past records to classify an agent or an item can be epistemologically useful in the absence or - as is especially relevant today - overabundance of information. But it has to be and remain open to constant scrutiny and revision to be epistemically useful and ethically just.

40 REPUTATION: Reputational Tools on the Web
Early years of 2000: focus on personalized information (My-Features) Now: trend towards systems of shared preferences, were people can rely on others’ preferences and rankings to construct there own access to and categorization of information Examples: Flickr’s Interestingess Twitter-Logic of Followers and Leaders LiquidJournal: people or groups create their own journals by selecting (existing) content and making it available via their selection LJ: someone‘s preference is a mode of classifying content

41 REPUTATION: Background
Epistemic Use and Value of Reputation as ongoing inquiry by two authors fuelled by different sources I don’t present results, this is rather an ongoing theoretical inquiry in the epistemic use of reputation More general, but I’d be happy to go into details either after the talk or via

42 REPUTATION: Background
Two authors -Two perspectives

43 REPUTATION: Conclusions
What is the epistemic values of reputation? Is it useful? Or rather dangerous? Both - it is useful and dangerous. Ad epist.: Is it just based on blind and imperfect heuristics that have a serendipitous effect on our search of information, or is it possible to conceive second order epistemic criteria that allow us to pry apart “good” and “bad” practices of trust and reliance on these reputational metrics?

44 REPUTATION: Conclusions
What is the epistemic values of reputation? Is it useful? Or rather dangerous? Both - it is useful and dangerous. But either way, reputational information, different methods or reputational cues of assessing it are being used extensively on the Web and elsewhere. Ad epist.: Is it just based on blind and imperfect heuristics that have a serendipitous effect on our search of information, or is it possible to conceive second order epistemic criteria that allow us to pry apart “good” and “bad” practices of trust and reliance on these reputational metrics?


Download ppt "The Place of Ratings and Reputational Tools in Knowledge Organization"

Similar presentations


Ads by Google