Methodology

Data Quality Assessment is a broad field that describes both qualitative and quantitative methods to evaluate the 'fitness', 'usefulness', or 'utility' towards specific purpose(s). By extension, Documentation Quality Assessment refers to evaluation of the documentation towards specific purposes, such as the role of technical documentation in equipping developers or engineers with a sufficient understanding of the system. Quality assessments consist of selecting 'dimensions', which are broad objectives that we want to assess, and then identifying specific 'metrics', which are granular evaluations, for example as tests or questions, whose answer produces a quantifiable number to represent degree to which the documentation satisfies the criteria. There are various ways in which the metrics can be combined to produce a single score or label for a dimension, and similarly there are various ways in which dimension scores can be consolidated to create an overall or global quality representation.

For our work, the document in question is the template for the public summary, which is not a dataset or a technical documentation, but is in essence a legally relevant document. This means 'quality' not only refers to typical characteristics associated with use of a document, but must also include the specific objectives of why the AI Act requires the public summary to be published in a specific manner and to include specific content. For practicality, we chose two hypothetical use-cases to guide what the assessment is intended to highlight: 'good faith' implementations where we can see that a conscentious effort has been made to provide the summary as intended, and 'bad faith' implementations where the summary has one or more major flaws -- intentional or otherwise. These corresponded to what we wanted to evaluate as good faith and bad faith respectively.

To start with, we conducted a literature review of documentation quality assessment frameworks, where we identified the different methods used and studied the dimensions used as well as their role in the assessment of specific information and use by stakeholders. From this, we selected the Goal, Question, Metric (GQM) as our overall approach to structure the process. The GQM method provides a three-step process:

1. Define the conceptual goal
2. For each goal, identify questions to describe the characteristics or model
3. For each question, what must be evaluated as a metric i.e. an assessment that produces a measurable result

Using GQM, and considering the broader objectives of the public summary as established in the AI Act as well as in the explanatory note accompanying the template, we chose the following two as our broad goals:

1. Transparency: The extent to which the public summary provides the required information.
2. Usefulness: The extent to which the provided information can be utilised for the intended purposes.

To evaluate each of these, we utilised our literature survey to select 'dimensions' or 'categories of quality' that would provide the best model or description for the information in the public summary. For Transparency these were:

1. Clarity: Is the information unambiguous, easy to understand, and avoids misinterpretation?
2. Completeness: Is the information relevant to what is asked is the necessary information provided?
3. Consistency: Does the information use the appropriate terminology, format, and structure in a manner that is consistent within and across the document(s)?
4. Correctness: Is the information accurate, validated, and reliable?

By using these, a public summary that has a high degree of transparency can be understood as the document and the information being clear, complete, consistent, and correct. We identified similar dimensions for assessing Usefulness:

1. Accessibility: Is the information easy to obtain, navigate, interact with, & includes or follows accessibility standards?
2. Comprehension: Is the information understandable and interpretable for the intended audience?

The choice of dimensions was not merely from theoretical considerations. We also discussed common issues such as providing information but in an obfuscated or confusing manner (clarity), not providing information in a specific field (completeness), using different terms across sections (consistency), providing information that is inaccurate or is later shown to be invalid (correctness), publishing the summary but in a manner that is not easily found (accessibility), and using highly technical jargon that cannot be understood by specific stakeholders (comprehension). Thus, the dimensions not only help to assess the quality of provided information, but also help categorise and assess issues.

To evaluate the selected dimensions over the public summary, we chose to assess each section of the public summary independently as it pertains to a specific topic. For example, Section 2.1 concerns public data whereas Section 2.4 concerns user data. Each of these sections would be of interest to specific stakeholders, and therefore they may be interested in the quality only for that specific section. In addition to these 'stakeholder-oriented' sections, we also considered the document itself as a separate category to assess criteria such as the manner in which it was provided and the metadata fields that are not part of any section. In total, we identified 8 sections:

1. Document: entire document
2. General information: Section 1
3. Public Data Sources: Section 2.1
4. Private Data Sources: Section 2.2
5. Scraped/Crawled Data: Section 2.3
6. User Data: Section 2.4
7. Synthetic & Other Data: Section 2.5 & 2.6
8. Data Processing: Section 3

Assessing the quality of each section meant assessing the transparency and usefulness of each section independently, which meant assessing the 6 dimensions (or questions under GQM) for each section. To do so, we created evaluation metrics using the following process:

1. First, separate each independent information field in the template. This was necessary as the template uses numbers for a group of fields, and where one field can ask several pieces of information. To evaluate each asked information on its own merit, we split the field into separate parts as necessary. For example, there is a single field for the Provider name and contact. Here, 'name' and 'contact' are independent pieces of information, and thus we created two fields to represent these.
2. We started from existential assessments i.e. is the field filled in or not (completeness).
3. We then considered what would be required for this information to be clear. For example, if the field for contact provides a generic contact that is not distinguishable from other contacts (e.g. a generic postal address), then this would not be clear.
4. We then considered whether the information across the fields is consistent e.g. the terms used in the model section are different from those in later sections (consistency).
5. We assumed that the information in specific fields in the public summary is correct for the moment, unless there are discrepancies in the public summary itself which invalidate this assumption (correctness). For these fields, if later it is found that the public summary has inaccurate or invalid information, this field would be used to identify and represent these issues.
6. Once we had ensured that the information is provided in a transparent manner, we focused on how this information could be used. For example, the Provider name must indicate the specific legal entity (comprehension). We also considered identifiers used being comprehensible, and key information that affects what obligations apply -- such as whether the model is a new model or has been fine-tuned on existing models.
7. For information where links would be needed, or where links are provided in a field, we considered how to evaluate the use of links and the information provided through the links, such as if these links lead to relevant locations (accessibility).

By using this approach, we identified a total of 242 evaluation metrics. This means that to assess the quality of a public summary that has all sections filled in, we would have to assess 242 things to evaluate its overall quality. Or, if we wanted to evaluate the quality only of a specific section, then the number of evaluations would be a subset of 242 (likely including the document section as well since it contains evaluation of how the public summary itself is provided). We think these 242 metrics are currently sufficient, based on our understanding of the template as well as how it may be filled in. However, as with any quality assessment framework, once we perform a number of evaluations, we would likely update the metrics -- including adding new ones -- to reflect the evolving practices and the need to identify and highlight specific practices (as good or bad quality).

In quality assessment methods, different information is likely to have a differing interpretation of importance. For example, in the public summary, the format of the date is not as important as contact details or the identifiers for the model. To reflect this difference, weights are assigned to specific evaluations, such as that the weighted score reflects the impact of that field being correctly or incorrectly provided. To determine the weights, we discussed each metric and what would be the implication of that metric on the use of the public summary. We assigned weights for metrics from 1 to 25, with 1 representing the lowest impact and 25 the highest. Note: The subjective choosing of weights is an accepted practice in developing quality assessment frameworks, as long as the method for assigning weights is justified. In our case, we determined the weights based on the perceived 'importance' of the information and the 'impact' of not having that information.

Using the weights, the scoring process becomes as follows: First, we evaluate the metrics, and then we multiply each metric by its score. For example, if there are two metrics in a section: A and B, with weights 25 and 1 respectively, then the scoring is as follows: (A x 25) + (B x 1). This means that if A is not provided and B is provided, the score would be much lower than if A is provided and B is not.

As the public summaries are assessed, certain fields and corresponding metrics may emerge to be more prominent -- either because they require additional considerations or because they are the areas where most summaries do not have an adequate quality. To address these, updating quality assessment frameworks also includes an assessment of whether the current set of weights is sufficient or should be changed.

1. Evaluating whether a given model meets the generality requirement.
2. Evaluating whether a given model likely meets the FLOP requirement.
3. Evaluating whether a given model is produced by a commercial entity or is provided in the course of a commercial activity.
4. Evaluating whether a given model is placed on the EU market.

• We expected the public summary to be explicitly indicated as such (i.e. titled 'public summary') or by referring to AI Act's Article 53(1)(d) obligations or broadly referring to AI Act compliance.
• We utilised a combination of search engines (using queries such as "public summary of training content" and "Article 53(1)(d) public summary") and utilising experimental web-search features of GenAI services such as Google Search and ChatGPT.
• We also undertook a manual discovery process where we looked at:
  1. The webpage or README in the repository where the model weights or model card are provided;
  2. The legal or compliance page on the model provider's webpage;
  3. The technical report or paper where the model's architecture and functioning are described.

1. The evaluator scores each (granular) metric by assesses the corresponding information field and element, and records a value that reflects their findings as:
   1. sufficient (scored as 1) to indicate the provided information meets or satisfies the requirement e.g. by being clear, or being complete;
   2. insufficient (scored as 0) to indicate either an absence of information or where the information does not meet the criteria of the metric e.g. by being obfuscated or being incomprehensible; and
   3. partially sufficient (scored as 0.5) to indicate that information partially satisfies the requirement.
2. The final score for each metric is then multiplied by the weight.
3. Note: If a specific metric or section is not applicable, then it is not given a '0' but is marked as 'N/A'. There is no penalty for the public summary providing no information in non-applicable sections, and the maximum possible score includes only those sections/metrics that are applicable.
4. Each dimension within a section is scored by taking the sum of all metrics associated with that dimension and section.
5. The transparency and usefulness scores are then calculated for each section by taking the sum of the scores of their respective dimensions.
6. The scores are then converted to percentages for convenience and ease of comparison across documents by dividing them by the maximum possible score i.e. the sum of maximum scores for each selected/evaluated metric. (as noted above, if a metric/field is not applicable, it is not included in the scoring).
7. A letter grade system from A as the highest quality and F as the lowest quality is assigned based on the scores to enable stakeholders to understand quality implications at a glance.