What is evaluated and what is not (yet)?

Disclaimer: No claim to completeness is made for any of the statements and assumptions contained herein. This wiki, like the project as a whole, is a "work in progress". We warmly welcome constructive criticism and content-related information or additions in order to continuously improve our work and thus the COMPPRESS project. We are happy to receive your feedback via info@comppress.org or via Github.

The formation of social and political opinion that is relevant for a democracy does not depend solely on media reporting. To a large extent, the media landscape also offers non-political entertainment that cannot be assessed in terms of its contribution to the formation of political opinions. Social media is an important source of information for many people – especially for political content.

Nevertheless, the use of traditional media such as television, radio or newspapers continues to predominate in the political informational behaviour of the German population.¹ The first limitation of the assessment object of the COMPPRESS project therefore relates to the selection of traditional media output. Information products of social media, blog articles, etc. are not included in the current development stage of COMPPRESS.

Newspapers and Magazines

The range of newspapers and magazines in Germany is diverse – both in terms of the breadth of topics and the different demands to distribute politically and socially high-quality articles. Newspapers with high quality standards do not have the highest circulation, but still have a high political influence and are considered opinion leaders.² The most important are, for example, the more liberal Süddeutsche Zeitung with approx. 380,000 copies per day, the more conservative Frankfurter Allgemeine Zeitung (approx. 228,000 copies / day), the Berliner taz and the Handelsblatt as the largest newspaper with an economic focus. The weekly political magazines such as Der Spiegel, Die Zeit and Der Stern are also to be classified in a similar way.²

The second limitation relates to the (current) sole evaluation of newspaper or magazine articles. Broadcasting output cannot be taken into account by COMPPRESS at the current stage of development.

Online Articles

2019, 68% of Germans used the internet as a source of political information. The most popular news sites were Spiegel online, t-online, Focus online, Bild.de and Web.de.² he third limitation of the evaluation is focused on the currently available online offers of the common newspaper and magazine providers.

Which Newspapers and Magazines Are Rated?

The selection of the media outlets whose articles can be rated in the COMPPRESS app was made for the initial phase of the project based on a review of the German and English-speaking international media markets. The aim was to capture the breadth of the existing social and political spectrum as well as to do justice to the reality of the circulation of newspapers and magazines, while depicting the selection of media outlets as fairly as possible. In the future, with more resources available for the project, all newspapers and magazines whose content is within the legal framework of the basic law should be available. The aim of COMPPRESS is not to create an editorial filter when making the offer available – not even when selecting the offer of the articles to be rated. The technical solution is to display the articles by dragging them from the freely accessible and available "feeds" of the media outlets.

The Rating System

With the COMPPRESS rating system, a wide variety of online articles from newspapers and magazines should be able to be rated by all users. This presents an evaluation system with conceptual and theoretical challenges.

Principles

t is essential that the evaluation criteria do not come from a (further) arbitrary, individual worldview and provide a subjective definition of what is "right", but correspond to the (self-imposed) obligations which media producers are already officially subject to. This is guaranteed by the German Press Code (DPK), which represents the content-related basis of the COMPPRESS rating system for the German-speaking countries. COMPPRESS is not an editorial team!

In addition, terms have to be developed as evaluation criteria that can be meaningfully used in a scale evaluation: They must be "capable of improvement" or "semantically quantifiable" (counter example: truth – something cannot be "a little or a lot" true, while something can be "a little" or "very" accurate).

Furthermore, the challenge arises to develop an evaluation system that is suitable for evaluating different types of newspaper and magazine articles, e.g. from different subject categories and articles with different analysis depths. The decisive difference for the evaluability of newspaper or magazine articles lies in their respective content-related complexity. Complex journalistic texts contain a judgmental assessment of what is being told by the author taking a position on what is being told or by conveying a high level of information and value judgments. Complex texts or articles go beyond a pure information announcement, as is the case with sub-complex articles, e.g. program announcements or simple reports. The evaluation of simpler reports can only be limited to a reduced requirement of the content. The COMPPRESS evaluation system counteracts this difficulty, in which only individual evaluation criteria can be applied – in the event that not all evaluation criteria can be sensibly applied to an article.

Ultimately, the terms used should be as comprehensible as possible – even if subjectivity remains unavoidable in the understanding. Pure objectivity cannot be given in the use of terms, which is why the evaluation criteria derived represent an attempt to choose overly complicated or scientific terms. 

It is particularly important that the COMPPRESS evaluation system does not make any intersubjective truth claims when evaluating the quality of an article. In contrast to a research project, the COMPPRESS evaluation system collects a large number of evaluations and shows existing majority relationships. This is like a choice process in which all eligible voters allow their individual understanding or assessment to flow into a collective result. Thus, the subjectivity that goes with every evaluation and every individual understanding of the terms of the evaluation system does not contradict the aim of the evaluations.

Derivation of The Evaluation Criteria – The Press Code

The German Press Code (DPK) used as a basis remains general in its execution of the commandments for good journalism. It was therefore necessary to derive meaningful syntheses from its individual paragraphs for an evaluation of newspaper or magazine articles.

Additional Sources

In order to incorporate general principles for good journalism, the syntheses from the DPK were compared and contrasted with ideas from two other recognised press codes and the requirements immanent in the State Media Treaty (MStV) for good journalism (Table 1).

Diligence and Truthfulness

Within the DPK there are two commandments that can be applied to the evaluation of online articles: "Truthfulness" and "Diligence". The remaining criteria of the DPK relate more to the concrete implementation of the work of journalists. Nevertheless, ideas for products can also be derived or transferred from other criteria of the DPK (Table 1).

Care: The term diligence is relatively easy to "translate" as an evaluation criterion. It is scalable (e.g. something is more or less careful) and it can be assumed that its horizon of meaning is easy to grasp – a common idea with regard to a definition can be found.

Truthfulness: The concept of truthfulness, like that of truth, is much more complex and it is much more difficult to deal with it in the sense of an evaluation criterion. Besides which, it's not scalable, but only applicable in binary form (something is either true or not). Truthfulness is the central requirement of journalistic work, so it is essential for the evaluation at COMPPRESS. In order to "translate" the concept of truthfulness into an evaluation criterion, essential ideas of the term were extracted. This "translation" is described below. It is based on historical methodological principles.

The Concept of Truthfulness

Essentially, an article describes an object (story, event, etc.) by a subject. It is therefore a narrative. Accordingly, there is a "filter" of the narrating subject between what happened and the product of the narrative. Theoretically, it is assumed that no objective image is created, but rather that the subject makes a narrative about an object (event, fact, etc.) in which the perception is shaped by subjective aspects. These include, for example, socialisation, prior knowledge, culture, personality, etc. It follows from this that objects can be represented in different ways by different subjects. A narrative should therefore not be judged solely in terms of the categories "true" or "false", but rather the question must be asked to what extent the subject adequately takes certain aspects into account in its narrative. In Jörn Rüsen's "Basic Features of a History", the truthfulness of a narrative can, among other things, gain validity through empirical and normative validity:

1. "[…] Stories justify their claim to validity by showing that the events they tell actually happened in the past as they tell it. This usually happens in that disclosure on sources are made in the stories […] Historical truth may be characterised as empirical validity of this reasoning regard. Stories are empirically valid if the facts asserted in them are backed up by experience. [...]"³
2. "[...] Stories justify their claim to validity by showing that the events they tell have a meaning for the everyday life of their addressees. The narrator brings norms into play […]. In this regard, truth can be described as normative validity. Stories are normatively valid if the meanings claimed in them are secured by valid norms. [...]"³

The concept of truthfulness was thus interpreted from two points of view: On the one hand, based on the claim that deliberate fraud or deliberate misrepresentation by the journalist is prohibited. On the other hand, through the claim to truthfulness as a narrative – which means that the journalists, despite their subjective view, should try to report events or facts as empirically and normatively as conclusively as possible.

From this understanding of the requirement of truthfulness, together with the requirement of care and taking into account the ideas from other international codes, three pairs of terms were derived that make up the COMPPRESS evaluation criteria. These are:

• Diligent Source Work
• Plausible Argumentation
• Balanced Coverage

Table 1 shows the sources of the derivation and the assignment of the respective meanings. The table provides an overview of the quality criteria taken from the DPK, the other sources that have contributed to extracting the final evaluation criteria as ideas for the DPK and the terms of the evaluation criteria that we have derived from them.

The results – our evaluation criteria

Diligent Source Work: This criterion results both directly from the claims mentioned in the statements on research work in the requirement of "diligence", as well as from the claim of the empirical validity of a narrative in the historical interpretation of the term "truthfulness". Cited statements must be developed on the basis of well-founded research, and personal statements must be substantiated and (if possible) different sources must be taken into account.

Plausible Argumentation: This criterion arises from the claims mentioned in the explanations of the research work in the requirement of "diligence", as well as from the claim of the normative validity of a narrative in the historical interpretation of the term "truthfulness". The (personal) arguments cited within a text must be reflected on and must not contradict each other logically. This is expressed both through the citing of evidence (research work) and through the fact that it is disclosed which value standards are referred to or from which examples standards are derived (normative validity). If, for example, it is stated that a politician's proposal is to be rejected as immoral, it must be demonstrated that the politician has actually made this proposal and what exactly about it, and on the basis of which standard, it is to be considered so. 

Balanced Coverage: This criterion arises from the claim of the normative validity of a narrative in the historical interpretation of the term "truthfulness". In the context of every evaluation of an object (event, fact), which is necessarily contained in complex articles, it is important to consider different points of view in order to present plausibly and fairly why the evaluation was carried out in this way. To ensure this, other points of view and counter-arguments must be reported in a fair and balanced manner. However, this is not to be equated with the claim of neutrality. An article or a comment can also openly contain the opinion of an author. However, this expression of opinion does not absolve them from the fact that they must also consider other arguments in their judgment in order to explain in a fair manner why they represent their opinion. This is illustrated by the example of the politician. If a journalist raises the argument that it is an immoral demand by politician X, then in addition to plausible arguments, it is also important to explain whether the politician should not be understood differently. It should also be demanded that a counter perspective is taken into account.

Final Notes

The evaluation criteria formulated here place high demands on journalistic products. However, the aim of the project is to promote and support good journalism. Certain quality features can be demanded. At the same time, we are aware that not all of our categories are applicable to every article. Our rating system has to manage a balancing act and ensure that both short reports and the years of research are assessable on the basis of a well-founded rating system, without the rating system losing its user-friendliness due to excessive complexity.

Sources:
1. Schüler, Ruth Maria / Niehues, Judith / Diermeier, Matthias (2021): Politisches Informationsverhalten: Gespräche und traditionelle Medien liegen vorn, IW-Report, Nr. 2, Köln. Abgerufen unter: https://www.iwkoeln.de/studien/ruth-maria-schueler-judith-niehues-matthias-diermeier-gespraeche-und-traditionelle-medien-liegen-vorn.html
2. CeMIL 2020: https://www.polsoz.fu-berlin.de/kommwiss/institut/cemil/schule/schule_inhaltselemente/12_mediensystem/a_mediensystem.html
3. Jörn Rüsen: Historische Vernunft. Grundzüge einer Historik I: Die Grundlagen der Geschichtswissenschaft. Göttingen: Vandenhoeck & Ruprecht 1983., abgerufen am 15.07.2021 unter: https://www.academia.edu/36948078/Jorn_Rusen_Grundzuge_einer_Historik_2018