{ "note": "R1-R5 mechanical pass. gold_engram_ids are null -- resolved after the ingest completes (this script ran before Step 2 finished, per this session's Step-2-then-Step-1 ordering). gold_position (before/after the question's own location in reading order) is filled in at the same resolution pass. Distractor-only doc (08) excluded entirely -- not R1-eligible for question/gold. selection_source per pair: 'heading-mechanical' (R2/R3 directly extracted from a real heading), 'CC-suggested' (this session's hand-picked candidate, not authoritative), 'pending-operator-selection' (no confident suggestion offered), or, after Step 1 ratification, 'operator-selected'.", "corpus_distractor_only": [ "08_ory_hydra_flow_cache_design_doc.md" ], "summary": { "total_docs_processed": 12, "mechanical_extraction": 2, "needs_manual_selection": 10, "heading_mechanical_r4_included": 1, "heading_mechanical_r4_excluded": 1, "cc_suggested_r4_included": 7, "cc_suggested_r4_excluded": 0, "no_suggestion_offered": 3 }, "pairs": [ { "doc_id": "09_backstage_adr002_default_catalog_file_format.md", "family": "effect-from-action", "extraction": "needs-manual-selection", "selection_source": "operator-selected", "selection_note": "no Decision/Consequences structure -- recommend drop", "reason": "no clean R2/R3 section heading found in this document (plain prose or a bulleted Q&A template, not markdown-headed sections)", "question_text": null, "question_location": null, "gold_text": null, "gold_location": null, "jaccard": null, "r4_status": "pending-manual-selection", "gold_engram_ids": null, "gold_position": null, "candidates": [ { "text": "---", "start_line": 1, "end_line": 1, "tags": [] }, { "text": "id: adrs-adr002", "start_line": 2, "end_line": 2, "tags": [] }, { "text": "title: 'ADR002: Default Software Catalog File Format'", "start_line": 3, "end_line": 3, "tags": [] }, { "text": "description: Architecture Decision Record (ADR) log on Default Software Catalog File Format", "start_line": 4, "end_line": 4, "tags": [] }, { "text": "---", "start_line": 5, "end_line": 5, "tags": [] }, { "text": "Backstage comes with software catalog functionality that you can use to track", "start_line": 9, "end_line": 9, "tags": [] }, { "text": "all your software components and more. It can be powered by data from various", "start_line": 10, "end_line": 10, "tags": [] }, { "text": "sources, and one of them that is included with the package is a custom database", "start_line": 11, "end_line": 11, "tags": [] }, { "text": "backed catalog. It has the ability to keep itself updated automatically based on", "start_line": 12, "end_line": 12, "tags": [] }, { "text": "the contents of little descriptor files in your version control system of", "start_line": 13, "end_line": 13, "tags": [] }, { "text": "choice. Developers create these files and maintain them side by side with their", "start_line": 14, "end_line": 14, "tags": [] }, { "text": "code, and the catalog system reacts accordingly.", "start_line": 15, "end_line": 15, "tags": [] }, { "text": "This ADR describes the default format of these descriptor files.", "start_line": 17, "end_line": 17, "tags": [] }, { "text": "Internally at Spotify, a homegrown software catalog system is used heavily and", "start_line": 21, "end_line": 21, "tags": [] }, { "text": "forms a core part of Backstage and other important pieces of the infrastructure.", "start_line": 22, "end_line": 22, "tags": [] }, { "text": "The user experience, learnings, and certain pieces of metadata from that catalog", "start_line": 23, "end_line": 23, "tags": [] }, { "text": "are being carried over to the open source effort.", "start_line": 24, "end_line": 24, "tags": [] }, { "text": "The file format described herein also draws heavy inspiration from the", "start_line": 26, "end_line": 26, "tags": [] }, { "text": "[kubernetes object format](https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/).", "start_line": 27, "end_line": 27, "tags": [] }, { "text": "There are a number of descriptor files, all of whose locations (e.g., within a", "start_line": 31, "end_line": 31, "tags": [] }, { "text": "version control system) are registered with the software catalog. The method of", "start_line": 32, "end_line": 32, "tags": [] }, { "text": "registration is not covered in this document; it could happen either manually", "start_line": 33, "end_line": 33, "tags": [] }, { "text": "inside Backstage, or by push events from a CI/CD pipelines, or by webhook", "start_line": 34, "end_line": 34, "tags": [] }, { "text": "triggers from the version control system, etc.", "start_line": 35, "end_line": 35, "tags": [] }, { "text": "Each file describes one or more entities in accordance with the", "start_line": 37, "end_line": 37, "tags": [] }, { "text": "[Backstage System Model](https://github.com/backstage/backstage/issues/390). All", "start_line": 38, "end_line": 38, "tags": [] }, { "text": "of these entities have a common structure and nomenclature, and they are stored", "start_line": 39, "end_line": 39, "tags": [] }, { "text": "in the software catalog from which they then can be queried.", "start_line": 40, "end_line": 40, "tags": [] }, { "text": "Entities have distinct names, and they may reference each other by those names.", "start_line": 42, "end_line": 42, "tags": [] }, { "text": "Descriptor files use the [YAML](https://yaml.org/spec/1.2/spec.html) format.", "start_line": 46, "end_line": 46, "tags": [] }, { "text": "They may be written by hand or created using automated tools. Each file may", "start_line": 47, "end_line": 47, "tags": [] }, { "text": "consist of several YAML documents (separated by `---`), where each document", "start_line": 48, "end_line": 48, "tags": [] }, { "text": "describes a single entity.", "start_line": 49, "end_line": 49, "tags": [] }, { "text": "This is an example entity definition with some mocked data.", "start_line": 51, "end_line": 51, "tags": [] }, { "text": "```yaml", "start_line": 53, "end_line": 53, "tags": [] }, { "text": "---", "start_line": 54, "end_line": 54, "tags": [] }, { "text": "apiVersion: backstage.io/v1alpha1", "start_line": 55, "end_line": 55, "tags": [] }, { "text": "kind: Component", "start_line": 56, "end_line": 56, "tags": [] }, { "text": "metadata:", "start_line": 57, "end_line": 57, "tags": [] }, { "text": "name: frobs-awesome", "start_line": 58, "end_line": 58, "tags": [] }, { "text": "description: |", "start_line": 59, "end_line": 59, "tags": [] }, { "text": "Backend service that implements the Frobs API, as defined", "start_line": 60, "end_line": 60, "tags": [] }, { "text": "in [the Frobs RFC](https://example.com/spec/frob.html).", "start_line": 61, "end_line": 61, "tags": [] }, { "text": "labels:", "start_line": 62, "end_line": 62, "tags": [] }, { "text": "system: frobs", "start_line": 63, "end_line": 63, "tags": [] }, { "text": "lifecycle: production", "start_line": 64, "end_line": 64, "tags": [] }, { "text": "example.com/service-discovery-name: frobsawesome", "start_line": 65, "end_line": 65, "tags": [] }, { "text": "annotations:", "start_line": 66, "end_line": 66, "tags": [] }, { "text": "circleci.com/project-slug: github/example-org/frobs-awesome", "start_line": 67, "end_line": 67, "tags": [] }, { "text": "spec:", "start_line": 68, "end_line": 68, "tags": [] }, { "text": "type: service", "start_line": 69, "end_line": 69, "tags": [] }, { "text": "```", "start_line": 70, "end_line": 70, "tags": [] }, { "text": "The root fields `apiVersion`, `kind`, `metadata`, and `spec` are part of the", "start_line": 72, "end_line": 72, "tags": [] }, { "text": "_envelope_, defining the overall structure of all kinds of entities. Likewise, the", "start_line": 73, "end_line": 73, "tags": [] }, { "text": "`name`, `namespace`, `labels`, and `annotations` metadata fields are of special", "start_line": 74, "end_line": 74, "tags": [] }, { "text": "significance and have reserved purposes and distinct shapes.", "start_line": 75, "end_line": 75, "tags": [] }, { "text": "See below for details about these fields.", "start_line": 77, "end_line": 77, "tags": [] }, { "text": "The root envelope object has the following structure.", "start_line": 81, "end_line": 81, "tags": [] }, { "text": "The `kind` is the high level entity type being described, typically from the", "start_line": 85, "end_line": 85, "tags": [] }, { "text": "[Backstage system model](https://github.com/backstage/backstage/issues/390). The", "start_line": 86, "end_line": 86, "tags": [] }, { "text": "first versions of the catalog will focus on the `Component` kind.", "start_line": 87, "end_line": 87, "tags": [] }, { "text": "The `apiVersion` is the version of the specification format for that particular", "start_line": 89, "end_line": 89, "tags": [] }, { "text": "entity that this file is written against. The version is used for being able to", "start_line": 90, "end_line": 90, "tags": [] }, { "text": "evolve the format, and the tuple of `apiVersion` and `kind` should be enough for", "start_line": 91, "end_line": 91, "tags": [] }, { "text": "a parser to know how to interpret the contents of the rest of the document.", "start_line": 92, "end_line": 92, "tags": [] }, { "text": "Backstage specific entities have an `apiVersion` that is prefixed with", "start_line": 94, "end_line": 94, "tags": [] }, { "text": "`backstage.io/`, to distinguish them from other types of objects that share the", "start_line": 95, "end_line": 95, "tags": [] }, { "text": "same type of structure. This may be relevant when co-hosting these", "start_line": 96, "end_line": 96, "tags": [] }, { "text": "specifications with, e.g., Kubernetes object manifests.", "start_line": 97, "end_line": 97, "tags": [] }, { "text": "Early versions of the catalog will be using alpha/beta versions, e.g.,", "start_line": 99, "end_line": 99, "tags": [] }, { "text": "`backstage.io/v1alpha1` to signal that the format may still change. After that,", "start_line": 100, "end_line": 100, "tags": [] }, { "text": "we will be using `backstage.io/v1` and up.", "start_line": 101, "end_line": 101, "tags": [] }, { "text": "A structure that contains metadata about the entity, i.e., things that aren't", "start_line": 105, "end_line": 105, "tags": [] }, { "text": "directly part of the entity specification itself. See below for more details", "start_line": 106, "end_line": 106, "tags": [] }, { "text": "about this structure.", "start_line": 107, "end_line": 107, "tags": [] }, { "text": "The actual specification data that describes the entity.", "start_line": 111, "end_line": 111, "tags": [] }, { "text": "The precise structure of the `spec` depends on the `apiVersion` and `kind`", "start_line": 113, "end_line": 113, "tags": [] }, { "text": "combination, and some kinds may not even have a `spec` at all. See further down", "start_line": 114, "end_line": 114, "tags": [] }, { "text": "in this document for the specification structure of specific kinds.", "start_line": 115, "end_line": 115, "tags": [] }, { "text": "The `metadata` root field has the following nested structure.", "start_line": 119, "end_line": 119, "tags": [] }, { "text": "The name of the entity. This name is both meant for human eyes to recognize the", "start_line": 123, "end_line": 123, "tags": [] }, { "text": "entity, and for machines and other components to reference the entity (e.g. in", "start_line": 124, "end_line": 124, "tags": [] }, { "text": "URLs or from other entity specification files).", "start_line": 125, "end_line": 125, "tags": [] }, { "text": "Names must be unique per kind, within a given namespace (if specified), at any", "start_line": 127, "end_line": 127, "tags": [] }, { "text": "point in time. This uniqueness constraint is also case insensitive. Names may be", "start_line": 128, "end_line": 128, "tags": [] }, { "text": "reused at a later time, after an entity is deleted from the registry.", "start_line": 129, "end_line": 129, "tags": [] }, { "text": "Names are required to follow a certain format. Entities that do not follow those", "start_line": 131, "end_line": 131, "tags": [] }, { "text": "rules will not be accepted for registration in the catalog. The ruleset is", "start_line": 132, "end_line": 132, "tags": [] }, { "text": "configurable to fit your organization's needs, but the default behavior is as", "start_line": 133, "end_line": 133, "tags": [] }, { "text": "follows.", "start_line": 134, "end_line": 134, "tags": [] }, { "text": "- Strings of length at least 1, and at most 63", "start_line": 136, "end_line": 136, "tags": [] }, { "text": "- Must consist of sequences of `[a-z0-9A-Z]` possibly separated by one of", "start_line": 137, "end_line": 137, "tags": [] }, { "text": "`[-_.]`", "start_line": 138, "end_line": 138, "tags": [] }, { "text": "Example: `visits-tracking-service`, `CircleciBuildsDs_avro_gcs`", "start_line": 140, "end_line": 140, "tags": [] }, { "text": "The `name` of a namespace that the entity belongs to. This field is optional", "start_line": 144, "end_line": 144, "tags": [] }, { "text": "and currently has no special semantics apart from bounding the name uniqueness", "start_line": 145, "end_line": 145, "tags": [] }, { "text": "constraint if specified. It is reserved for future use and may get broader", "start_line": 146, "end_line": 146, "tags": [] }, { "text": "semantic implication.", "start_line": 147, "end_line": 147, "tags": [] }, { "text": "Namespaces may also be part of the catalog and are `v1` / `Namespace` entities,", "start_line": 149, "end_line": 149, "tags": [] }, { "text": "i.e., not Backstage specific but the same as in Kubernetes.", "start_line": 150, "end_line": 150, "tags": [] }, { "text": "A human readable description of the entity to be shown in Backstage. Should be", "start_line": 154, "end_line": 154, "tags": [] }, { "text": "kept short and informative, suitable to give an overview of the entity's purpose", "start_line": 155, "end_line": 155, "tags": [] }, { "text": "at a glance. More detailed explanations and documentation should be placed", "start_line": 156, "end_line": 156, "tags": [] }, { "text": "elsewhere.", "start_line": 157, "end_line": 157, "tags": [] }, { "text": "Labels are optional key/value pairs that are attached to the entity, and", "start_line": 161, "end_line": 161, "tags": [] }, { "text": "their use is identical to", "start_line": 162, "end_line": 162, "tags": [] }, { "text": "[kubernetes object labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/).", "start_line": 163, "end_line": 163, "tags": [] }, { "text": "Their main purpose is for references to other entities and for information that", "start_line": 165, "end_line": 165, "tags": [] }, { "text": "is in one way or another classifying for the current entity. They are often used", "start_line": 166, "end_line": 166, "tags": [] }, { "text": "as values in queries or filters.", "start_line": 167, "end_line": 167, "tags": [] }, { "text": "Both the key and the value are strings, subject to the following restrictions.", "start_line": 169, "end_line": 169, "tags": [] }, { "text": "Keys have an optional prefix followed by a slash, and then the name part, which", "start_line": 171, "end_line": 171, "tags": [] }, { "text": "is required. The prefix must be a valid lowercase domain name, at most 253", "start_line": 172, "end_line": 172, "tags": [] }, { "text": "characters in total. The name part must be sequences of `[a-zA-Z0-9]` separated", "start_line": 173, "end_line": 173, "tags": [] }, { "text": "by any of `[-_.]`, at most 63 characters in total.", "start_line": 174, "end_line": 174, "tags": [] }, { "text": "The `backstage.io/` prefix is reserved for use by Backstage core components.", "start_line": 176, "end_line": 176, "tags": [] }, { "text": "Some keys, such as `system` also have predefined semantics.", "start_line": 177, "end_line": 177, "tags": [] }, { "text": "Values are strings that follow the same restrictions as `name` above.", "start_line": 179, "end_line": 179, "tags": [] }, { "text": "An object with arbitrary non-identifying metadata attached to the entity,", "start_line": 183, "end_line": 183, "tags": [] }, { "text": "identical in use to", "start_line": 184, "end_line": 184, "tags": [] }, { "text": "[kubernetes object annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/).", "start_line": 185, "end_line": 185, "tags": [] }, { "text": "Their purpose is mainly, but not limited, to reference into external systems.", "start_line": 187, "end_line": 187, "tags": [] }, { "text": "This could, for example, be a reference to the git ref the entity was ingested", "start_line": 188, "end_line": 188, "tags": [] }, { "text": "from, to monitoring and logging systems, to pagerduty schedules, etc.", "start_line": 189, "end_line": 189, "tags": [] }, { "text": "Both the key and the value are strings, subject to the following restrictions.", "start_line": 191, "end_line": 191, "tags": [] }, { "text": "Keys have an optional prefix followed by a slash, and then the name part, which", "start_line": 193, "end_line": 193, "tags": [] }, { "text": "is required. The prefix must be a valid lowercase domain name, at most 253", "start_line": 194, "end_line": 194, "tags": [] }, { "text": "characters in total. The name part must be sequences of `[a-zA-Z0-9]` separated", "start_line": 195, "end_line": 195, "tags": [] }, { "text": "by any of `[-_.]`, at most 63 characters in total.", "start_line": 196, "end_line": 196, "tags": [] }, { "text": "The `backstage.io/` prefix is reserved for use by Backstage core components.", "start_line": 198, "end_line": 198, "tags": [] }, { "text": "Values can be of any length but are limited to being strings.", "start_line": 200, "end_line": 200, "tags": [] }, { "text": "The `spec` object for this kind is as follows:", "start_line": 209, "end_line": 209, "tags": [] } ], "operator_disposition": "drop" }, { "doc_id": "10_backstage_adr005_catalog_core_entities.md", "family": "effect-from-action", "extraction": "mechanical", "selection_source": "operator-selected", "question_text": "Backstage should eventually support the following core entities:\n\n- **Components** are individual pieces of software\n- **APIs** are the boundaries between different components\n- **Resources** are physical or virtual infrastructure needed to operate a\n component\n\n![Catalog Core Entities](../assets/architecture-decisions/catalog-core-entities.png)\n\nFor now, we'll start by only implementing support for the Component entity in\nthe Backstage catalog. This can later be extended to APIs, Resources and other\npotentially useful entities.", "question_location": "lines 14-25 of the ## Decision section (excludes the ### Component/API/Resource subsections and YAML blocks)", "gold_text": "We will continue fleshing out support for the Component entity in the Backstage\ncatalog.", "gold_location": "lines 110-112 (## Consequences)", "jaccard": 0.1316, "r4_status": "included", "gold_engram_ids": null, "gold_position": null, "operator_disposition": "keep, trimmed (CC-proposed trim boundary, APPROVED as-is)" }, { "doc_id": "11_backstage_adr009_entity_references.md", "family": "effect-from-action", "extraction": "needs-manual-selection", "selection_source": "operator-selected", "selection_note": "no Decision/Consequences structure -- recommend drop", "reason": "no clean R2/R3 section heading found in this document (plain prose or a bulleted Q&A template, not markdown-headed sections)", "question_text": null, "question_location": null, "gold_text": null, "gold_location": null, "jaccard": null, "r4_status": "pending-manual-selection", "gold_engram_ids": null, "gold_position": null, "candidates": [ { "text": "---", "start_line": 1, "end_line": 1, "tags": [] }, { "text": "id: adrs-adr009", "start_line": 2, "end_line": 2, "tags": [] }, { "text": "title: 'ADR009: Entity References'", "start_line": 3, "end_line": 3, "tags": [] }, { "text": "description: Architecture Decision Record (ADR) log on Entity References", "start_line": 4, "end_line": 4, "tags": [] }, { "text": "---", "start_line": 5, "end_line": 5, "tags": [] }, { "text": "While the spec for the catalog file format is well described in", "start_line": 9, "end_line": 9, "tags": [] }, { "text": "[ADR002](./adr002-default-catalog-file-format.md), guidance was not provided as", "start_line": 10, "end_line": 10, "tags": [] }, { "text": "to how one is expected to express references to other entities in the catalog.", "start_line": 11, "end_line": 11, "tags": [] }, { "text": "There was also some confusion on how to reference entities in URLs in the", "start_line": 12, "end_line": 12, "tags": [] }, { "text": "Backstage frontend.", "start_line": 13, "end_line": 13, "tags": [] }, { "text": "Following discussion in", "start_line": 15, "end_line": 15, "tags": [] }, { "text": "[Issue 1947](https://github.com/backstage/backstage/issues/1947), a decision was", "start_line": 16, "end_line": 16, "tags": [] }, { "text": "made.", "start_line": 17, "end_line": 17, "tags": [] }, { "text": "The textual format, as written by humans, to reference entities by name is on", "start_line": 21, "end_line": 21, "tags": [] }, { "text": "the following form, where square brackets denote optionality:", "start_line": 22, "end_line": 22, "tags": [] }, { "text": "```text", "start_line": 24, "end_line": 24, "tags": [] }, { "text": "[:][/]", "start_line": 25, "end_line": 25, "tags": [] }, { "text": "```", "start_line": 26, "end_line": 26, "tags": [] }, { "text": "That is, it is composed of between one and three parts in this specific order,", "start_line": 28, "end_line": 28, "tags": [] }, { "text": "without any additional encoding, with those exact separator characters.", "start_line": 29, "end_line": 29, "tags": [] }, { "text": "Optionality of `kind` and `namespace` are contextual, and they may or may not", "start_line": 30, "end_line": 30, "tags": [] }, { "text": "have default contextual fallback values.", "start_line": 31, "end_line": 31, "tags": [] }, { "text": "When that format is insufficient or when machine made interchange formats wish", "start_line": 33, "end_line": 33, "tags": [] }, { "text": "to express such relations in a more expressive form, a nested structure on the", "start_line": 34, "end_line": 34, "tags": [] }, { "text": "following form can be used:", "start_line": 35, "end_line": 35, "tags": [] }, { "text": "```yaml", "start_line": 37, "end_line": 37, "tags": [] }, { "text": "kind: ", "start_line": 38, "end_line": 38, "tags": [] }, { "text": "namespace: ", "start_line": 39, "end_line": 39, "tags": [] }, { "text": "name: ", "start_line": 40, "end_line": 40, "tags": [] }, { "text": "```", "start_line": 41, "end_line": 41, "tags": [] }, { "text": "Of these, only `name` is always required. Optionality of `kind` and `namespace`", "start_line": 43, "end_line": 43, "tags": [] }, { "text": "are contextual, and they may or may not have default contextual fallback values.", "start_line": 44, "end_line": 44, "tags": [] }, { "text": "All other possible key values in this structure are reserved for future use.", "start_line": 45, "end_line": 45, "tags": [] }, { "text": "A system or user wanting to express a full entity name that is always valid", "start_line": 47, "end_line": 47, "tags": [] }, { "text": "shall supply the entire triplet, whether using the string form or the compound", "start_line": 48, "end_line": 48, "tags": [] }, { "text": "form.", "start_line": 49, "end_line": 49, "tags": [] }, { "text": "A full description of the format can be found", "start_line": 51, "end_line": 51, "tags": [] }, { "text": "[in the documentation](https://backstage.io/docs/features/software-catalog/references).", "start_line": 52, "end_line": 52, "tags": [] }, { "text": "Where entities are referenced by name in the Backstage frontend, the URL", "start_line": 56, "end_line": 56, "tags": [] }, { "text": "containing the reference shall take the following form:", "start_line": 57, "end_line": 57, "tags": [] }, { "text": "```text", "start_line": 59, "end_line": 59, "tags": [] }, { "text": ":namespace/:kind/:name", "start_line": 60, "end_line": 60, "tags": [] }, { "text": "```", "start_line": 61, "end_line": 61, "tags": [] }, { "text": "All three parts are required under all circumstances. The default value for the", "start_line": 63, "end_line": 63, "tags": [] }, { "text": "`namespace` in the catalog is the string `\"default\"` if the entity does not", "start_line": 64, "end_line": 64, "tags": [] }, { "text": "specify one explicitly in `metadata.namespace`.", "start_line": 65, "end_line": 65, "tags": [] }, { "text": "This means that we do not encourage the string form of entity references to be", "start_line": 67, "end_line": 67, "tags": [] }, { "text": "used as a single URL segment, due to the use of URL-unsafe characters leading to", "start_line": 68, "end_line": 68, "tags": [ "possible-gold(root-cause-language)" ] }, { "text": "possible risk, confusion, and uglier URLs.", "start_line": 69, "end_line": 69, "tags": [] } ], "operator_disposition": "drop" }, { "doc_id": "12_backstage_adr011_plugin_package_structure.md", "family": "effect-from-action", "extraction": "mechanical", "selection_source": "operator-selected", "question_text": "We will place all plugin related code in the `plugins/` directory. The\n`packages/` directory is reserved for core package of Backstage.\n\nWe follow this structure for plugin packages (where `x` is the plugin name, for\nexample `catalog` or `techdocs`):\n\n- `x`: Contains the main frontend code of the plugin.\n- `x-module-`: Contains optional modules related to the frontend plugin\n package.\n- `x-backend`: Contains the main backend code of the plugin.\n- `x-backend-module-`: Contains optional modules related to the backend\n plugin package.\n- `x-react`: Contains shared widgets, hooks and similar that both the plugin\n itself (`x`) and third-party frontend plugins can depend on.\n- `x-node`: Contains utilities for backends that both the plugin backend itself\n (`x-backend`) and third-party backend plugins can depend on.\n- `x-common`: An isomorphic package with platform agnostic models, clients, and\n utilities that all packages above or any third-party plugin package can depend\n on.\n\nWe prefix the package names with `@backstage/plugin-`.\n\nThis structure is based on a\n[suggestion in issue #3655](https://github.com/backstage/backstage/issues/3655#issuecomment-758166746).", "question_location": "lines 28-53 (## Decision)", "gold_text": "We will actively migrate existing packages that are part of a plugin to the\n`plugins/` folder. This affects packages like:\n\n- `packages/techdocs-common` which should be moved to `plugins/techdocs-node`\n and named `@backstage/plugin-techdocs-node`.\n- `packages/catalog-client` which will be part of a future\n `plugins/catalog-common` and named `@backstage/plugin-catalog-common`.\n- While the new location of `packages/catalog-model` should be\n `plugins/catalog-common` we might want to do an exception here, as it's a very\n central package.\n\nWe will actively migrate optional features of backend plugins into separate\n`x-backend-module-` packages, for example the more specialized processors\nin the catalog backend.\n\nThe limited set of rules might not be sufficient in the future. If additional\npackages are required, we will revisit this decision and extend the pattern.\n\nIf possible, we will add tools, such as lint rules, to help enforce the package\nnames and dependencies between them or CLI commands to generate these packages.\n\nThe distinction between core packages and plugins helps us to setup\n[CODEOWNERS](https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/about-code-owners)\nin the repository. We can set the code owners for the `packages/` folder to the\ncore team and create additional rules (like `plugins/x*`) for plugin\nmaintainers.", "gold_location": "lines 55-81 (## Consequences)", "jaccard": 0.1639, "r4_status": "excluded-overlap", "gold_engram_ids": null, "gold_position": null, "operator_disposition": "drop" }, { "doc_id": "01_cloudflare_outage_2025-11-18.md", "family": "cause-from-symptom", "extraction": "needs-manual-selection", "selection_source": "operator-selected", "selection_note": null, "reason": "no clean R2/R3 section heading found in this document (plain prose or a bulleted Q&A template, not markdown-headed sections)", "question_text": "On 18 November 2025 at 11:20 UTC (all times in this blog are UTC), Cloudflare's network began experiencing significant failures to deliver core network traffic. This showed up to Internet users trying to access our customers' sites as an error page indicating a failure within Cloudflare's network.", "question_location": "line 3", "gold_text": "The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind. Instead, it was triggered by a change to one of our database systems' permissions which caused the database to output multiple entries into a \u201cfeature file\u201d used by our Bot Management system. That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network.", "gold_location": "line 4", "jaccard": 0.0167, "r4_status": "included", "gold_engram_ids": null, "gold_position": null, "candidates": [ { "text": "On 18 November 2025 at 11:20 UTC (all times in this blog are UTC), Cloudflare's network began experiencing significant failures to deliver core network traffic. This showed up to Internet users trying to access our customers' sites as an error page indicating a failure within Cloudflare's network.", "start_line": 3, "end_line": 3, "tags": [ "possible-question(impact-language)" ] }, { "text": "The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind. Instead, it was triggered by a change to one of our database systems' permissions which caused the database to output multiple entries into a \u201cfeature file\u201d used by our Bot Management system. That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network.", "start_line": 4, "end_line": 4, "tags": [] }, { "text": "The software running on these machines to route traffic across our network reads this feature file to keep our Bot Management system up to date with ever changing threats. The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail.", "start_line": 5, "end_line": 5, "tags": [] }, { "text": "After we initially wrongly suspected the symptoms we were seeing were caused by a hyper-scale DDoS attack, we correctly identified the core issue and were able to stop the propagation of the larger-than-expected feature file and replace it with an earlier version of the file. Core traffic was largely flowing as normal by 14:30. We worked over the next few hours to mitigate increased load on various parts of our network as traffic rushed back online. As of 17:06 all systems at Cloudflare were functioning as normal.", "start_line": 6, "end_line": 6, "tags": [ "possible-gold(root-cause-language)" ] }, { "text": "We are sorry for the impact to our customers and to the Internet in general. Given Cloudflare's importance in the Internet ecosystem any outage of any of our systems is unacceptable. That there was a period of time where our network was not able to route traffic is deeply painful to every member of our team. We know we let you down today.", "start_line": 7, "end_line": 7, "tags": [ "possible-question(impact-language)" ] }, { "text": "This post is an in-depth recount of exactly what happened and what systems and processes failed. It is also the beginning, though not the end, of what we plan to do in order to make sure an outage like this will not happen again.", "start_line": 8, "end_line": 8, "tags": [ "possible-question(impact-language)" ] }, { "text": "The outage", "start_line": 9, "end_line": 9, "tags": [ "possible-question(impact-language)" ] }, { "text": "The chart below shows the volume of 5xx error HTTP status codes served by the Cloudflare network. Normally this should be very low, and it was right up until the start of the outage.", "start_line": 10, "end_line": 10, "tags": [ "possible-question(impact-language)" ] }, { "text": "The volume prior to 11:20 is the expected baseline of 5xx errors observed across our network. The spike, and subsequent fluctuations, show our system failing due to loading the incorrect feature file. What\u2019s notable is that our system would then recover for a period. This was very unusual behavior for an internal error.", "start_line": 11, "end_line": 11, "tags": [ "possible-gold(root-cause-language)", "possible-question(impact-language)" ] }, { "text": "The explanation was that the file was being generated every five minutes by a query running on a ClickHouse database cluster, which was being gradually updated to improve permissions management. Bad data was only generated if the query ran on a part of the cluster which had been updated. As a result, every five minutes there was a chance of either a good or a bad set of configuration files being generated and rapidly propagated across the network.", "start_line": 12, "end_line": 12, "tags": [] }, { "text": "This fluctuation made it unclear what was happening as the entire system would recover and then fail again as sometimes good, sometimes bad configuration files were distributed to our network. Initially, this led us to believe this might be caused by an attack. Eventually, every ClickHouse node was generating the bad configuration file and the fluctuation stabilized in the failing state.", "start_line": 13, "end_line": 13, "tags": [ "possible-gold(root-cause-language)" ] }, { "text": "Errors continued until the underlying issue was identified and resolved starting at 14:30. We solved the problem by stopping the generation and propagation of the bad feature file and manually inserting a known good file into the feature file distribution queue. And then forcing a restart of our core proxy.", "start_line": 14, "end_line": 14, "tags": [] }, { "text": "The remaining long tail in the chart above is our team restarting remaining services that had entered a bad state, with 5xx error code volume returning to normal at 17:06.", "start_line": 15, "end_line": 15, "tags": [] }, { "text": "The following services were impacted:", "start_line": 16, "end_line": 16, "tags": [ "possible-question(impact-language)" ] }, { "text": "As well as returning HTTP 5xx errors, we observed significant increases in latency of responses from our CDN during the impact period. This was due to large amounts of CPU being consumed by our debugging and observability systems, which automatically enhance uncaught errors with additional debugging information.", "start_line": 25, "end_line": 25, "tags": [ "possible-gold(root-cause-language)", "possible-question(impact-language)" ] }, { "text": "How Cloudflare processes requests, and how this went wrong today", "start_line": 26, "end_line": 26, "tags": [] }, { "text": "Every request to Cloudflare takes a well-defined path through our network. It could be from a browser loading a webpage, a mobile app calling an API, or automated traffic from another service. These requests first terminate at our HTTP and TLS layer, then flow into our core proxy system (which we call FL for \u201cFrontline\u201d), and finally through Pingora, which performs cache lookups or fetches data from the origin if needed.", "start_line": 27, "end_line": 27, "tags": [] }, { "text": "We previously shared more detail about how the core proxy works here.", "start_line": 28, "end_line": 28, "tags": [] }, { "text": "As a request transits the core proxy, we run the various security and performance products available in our network. The proxy applies each customer\u2019s unique configuration and settings, from enforcing WAF rules and DDoS protection to routing traffic to the Developer Platform and R2. It accomplishes this through a set of domain-specific modules that apply the configuration and policy rules to traffic transiting our proxy.", "start_line": 29, "end_line": 29, "tags": [] }, { "text": "One of those modules, Bot Management, was the source of today\u2019s outage.", "start_line": 30, "end_line": 30, "tags": [ "possible-question(impact-language)" ] }, { "text": "Cloudflare\u2019s Bot Management includes, among other systems, a machine learning model that we use to generate bot scores for every request traversing our network. Our customers use bot scores to control which bots are allowed to access their sites \u2014 or not.", "start_line": 31, "end_line": 31, "tags": [ "possible-question(impact-language)" ] }, { "text": "The model takes as input a \u201cfeature\u201d configuration file. A feature, in this context, is an individual trait used by the machine learning model to make a prediction about whether the request was automated or not. The feature configuration file is a collection of individual features.", "start_line": 32, "end_line": 32, "tags": [] }, { "text": "This feature file is refreshed every few minutes and published to our entire network and allows us to react to variations in traffic flows across the Internet. It allows us to react to new types of bots and new bot attacks. So it\u2019s critical that it is rolled out frequently and rapidly as bad actors change their tactics quickly.", "start_line": 33, "end_line": 33, "tags": [] }, { "text": "A change in our underlying ClickHouse query behaviour (explained below) that generates this file caused it to have a large number of duplicate \u201cfeature\u201d rows. This changed the size of the previously fixed-size feature configuration file, causing the bots module to trigger an error.", "start_line": 34, "end_line": 34, "tags": [] }, { "text": "As a result, HTTP 5xx error codes were returned by the core proxy system that handles traffic processing for our customers, for any traffic that depended on the bots module. This also affected Workers KV and Access, which rely on the core proxy.", "start_line": 35, "end_line": 35, "tags": [ "possible-question(impact-language)" ] }, { "text": "Unrelated to this incident, we were and are currently migrating our customer traffic to a new version of our proxy service, internally known as FL2. Both versions were affected by the issue, although the impact observed was different.", "start_line": 36, "end_line": 36, "tags": [ "possible-question(impact-language)" ] }, { "text": "Customers deployed on the new FL2 proxy engine, observed HTTP 5xx errors. Customers on our old proxy engine, known as FL, did not see errors, but bot scores were not generated correctly, resulting in all traffic receiving a bot score of zero. Customers that had rules deployed to block bots would have seen large numbers of false positives. Customers who were not using our bot score in their rules did not see any impact.", "start_line": 37, "end_line": 37, "tags": [ "possible-question(impact-language)" ] }, { "text": "Throwing us off and making us believe this might have been an attack was another apparent symptom we observed: Cloudflare\u2019s status page went down. The status page is hosted completely off Cloudflare\u2019s infrastructure with no dependencies on Cloudflare. While it turned out to be a coincidence, it led some of the team diagnosing the issue to believe that an attacker may be targeting both our systems as well as our status page. Visitors to the status page at that time were greeted by an error message:", "start_line": 38, "end_line": 38, "tags": [ "possible-question(impact-language)" ] }, { "text": "In the internal incident chat room, we were concerned that this might be the continuation of the recent spate of high volume Aisuru DDoS attacks:", "start_line": 39, "end_line": 39, "tags": [] }, { "text": "The query behaviour change", "start_line": 40, "end_line": 40, "tags": [] }, { "text": "I mentioned above that a change in the underlying query behaviour resulted in the feature file containing a large number of duplicate rows. The database system in question uses ClickHouse\u2019s software.", "start_line": 41, "end_line": 41, "tags": [] }, { "text": "For context, it\u2019s helpful to know how ClickHouse distributed queries work. A ClickHouse cluster consists of many shards. To query data from all shards, we have so-called distributed tables (powered by the table engine Distributed) in a database called default. The Distributed engine queries underlying tables in a database r0. The underlying tables are where data is stored on each shard of a ClickHouse cluster.", "start_line": 42, "end_line": 42, "tags": [] }, { "text": "Queries to the distributed tables run through a shared system account. As part of efforts to improve our distributed queries security and reliability, there\u2019s work being done to make them run under the initial user accounts instead.", "start_line": 43, "end_line": 43, "tags": [] }, { "text": "Before today, ClickHouse users would only see the tables in the default database when querying table metadata from ClickHouse system tables such as system.tables or system.columns.", "start_line": 44, "end_line": 44, "tags": [ "possible-question(impact-language)" ] }, { "text": "Since users already have implicit access to underlying tables in r0, we made a change at 11:05 to make this access explicit, so that users can see the metadata of these tables as well. By making sure that all distributed subqueries can run under the initial user, query limits and access grants can be evaluated in a more fine-grained manner, avoiding one bad subquery from a user affecting others.", "start_line": 45, "end_line": 45, "tags": [ "possible-question(impact-language)" ] }, { "text": "The change explained above resulted in all users accessing accurate metadata about tables they have access to. Unfortunately, there were assumptions made in the past, that the list of columns returned by a query like this would only include the \u201cdefault\u201d database:", "start_line": 46, "end_line": 46, "tags": [ "possible-question(impact-language)" ] }, { "text": "SELECT", "start_line": 47, "end_line": 47, "tags": [] }, { "text": "name,", "start_line": 48, "end_line": 48, "tags": [] }, { "text": "type", "start_line": 49, "end_line": 49, "tags": [] }, { "text": "FROM system.columns", "start_line": 50, "end_line": 50, "tags": [] }, { "text": "WHERE", "start_line": 51, "end_line": 51, "tags": [] }, { "text": "table = 'http_requests_features'", "start_line": 52, "end_line": 52, "tags": [] }, { "text": "order by name;", "start_line": 53, "end_line": 53, "tags": [] }, { "text": "Note how the query does not filter for the database name. With us gradually rolling out the explicit grants to users of a given ClickHouse cluster, after the change at 11:05 the query above started returning \u201cduplicates\u201d of columns because those were for underlying tables stored in the r0 database.", "start_line": 54, "end_line": 54, "tags": [ "possible-question(impact-language)" ] }, { "text": "This, unfortunately, was the type of query that was performed by the Bot Management feature file generation logic to construct each input \u201cfeature\u201d for the file mentioned at the beginning of this section.", "start_line": 55, "end_line": 55, "tags": [] }, { "text": "The query above would return a table of columns like the one displayed (simplified example):", "start_line": 56, "end_line": 56, "tags": [] }, { "text": "However, as part of the additional permissions that were granted to the user, the response now contained all the metadata of the r0 schema effectively more than doubling the rows in the response ultimately affecting the number of rows (i.e. features) in the final file output.", "start_line": 57, "end_line": 57, "tags": [] }, { "text": "Memory preallocation", "start_line": 58, "end_line": 58, "tags": [] }, { "text": "Each module running on our proxy service has a number of limits in place to avoid unbounded memory consumption and to preallocate memory as a performance optimization. In this specific instance, the Bot Management system has a limit on the number of machine learning features that can be used at runtime. Currently that limit is set to 200, well above our current use of ~60 features. Again, the limit exists because for performance reasons we preallocate memory for the features.", "start_line": 59, "end_line": 59, "tags": [] }, { "text": "When the bad file with more than 200 features was propagated to our servers, this limit was hit \u2014 resulting in the system panicking. The FL2 Rust code that makes the check and was the source of the unhandled error is shown below:", "start_line": 60, "end_line": 60, "tags": [] }, { "text": "This resulted in the following panic which in turn resulted in a 5xx error:", "start_line": 61, "end_line": 61, "tags": [] }, { "text": "thread fl2_worker_thread panicked: called Result::unwrap() on an Err value", "start_line": 62, "end_line": 62, "tags": [] }, { "text": "Other impact during the incident", "start_line": 63, "end_line": 63, "tags": [ "possible-question(impact-language)" ] }, { "text": "Other systems that rely on our core proxy were impacted during the incident. This included Workers KV and Cloudflare Access. The team was able to reduce the impact to these systems at 13:04, when a patch was made to Workers KV to bypass the core proxy. Subsequently, all downstream systems that rely on Workers KV (such as Access itself) observed a reduced error rate.", "start_line": 64, "end_line": 64, "tags": [ "possible-question(impact-language)" ] }, { "text": "The Cloudflare Dashboard was also impacted due to both Workers KV being used internally and Cloudflare Turnstile being deployed as part of our login flow.", "start_line": 65, "end_line": 65, "tags": [ "possible-gold(root-cause-language)", "possible-question(impact-language)" ] }, { "text": "Turnstile was impacted by this outage, resulting in customers who did not have an active dashboard session being unable to log in. This showed up as reduced availability during two time periods: from 11:30 to 13:10, and between 14:40 and 15:30, as seen in the graph below.", "start_line": 66, "end_line": 66, "tags": [ "possible-question(impact-language)" ] }, { "text": "The first period, from 11:30 to 13:10, was due to the impact to Workers KV, which some control plane and dashboard functions rely upon. This was restored at 13:10, when Workers KV bypassed the core proxy system.", "start_line": 67, "end_line": 67, "tags": [ "possible-gold(root-cause-language)", "possible-question(impact-language)" ] }, { "text": "The second period of impact to the dashboard occurred after restoring the feature configuration data. A backlog of login attempts began to overwhelm the dashboard. This backlog, in combination with retry attempts, resulted in elevated latency, reducing dashboard availability. Scaling control plane concurrency restored availability at approximately 15:30.", "start_line": 68, "end_line": 68, "tags": [ "possible-question(impact-language)" ] }, { "text": "Remediation and follow-up steps", "start_line": 69, "end_line": 69, "tags": [] }, { "text": "Now that our systems are back online and functioning normally, work has already begun on how we will harden them against failures like this in the future. In particular we are:", "start_line": 70, "end_line": 70, "tags": [] }, { "text": "- Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input", "start_line": 71, "end_line": 71, "tags": [] }, { "text": "- Enabling more global kill switches for features", "start_line": 72, "end_line": 72, "tags": [] }, { "text": "- Eliminating the ability for core dumps or other error reports to overwhelm system resources", "start_line": 73, "end_line": 73, "tags": [] }, { "text": "- Reviewing failure modes for error conditions across all core proxy modules", "start_line": 74, "end_line": 74, "tags": [] }, { "text": "Today was Cloudflare's worst outage since 2019. We've had outages that have made our dashboard unavailable. Some that have caused newer features to not be available for a period of time. But in the last 6+ years we've not had another outage that has caused the majority of core traffic to stop flowing through our network.", "start_line": 75, "end_line": 75, "tags": [ "possible-question(impact-language)" ] }, { "text": "An outage like today is unacceptable. We've architected our systems to be highly resilient to failure to ensure traffic will always continue to flow. When we've had outages in the past it's always led to us building new, more resilient systems.", "start_line": 76, "end_line": 76, "tags": [ "possible-question(impact-language)" ] }, { "text": "On behalf of the entire team at Cloudflare, I would like to apologize for the pain we caused the Internet today.", "start_line": 77, "end_line": 77, "tags": [] } ], "operator_disposition": "keep" }, { "doc_id": "02_gitlab_db_outage_postmortem_2017-01-31.md", "family": "cause-from-symptom", "extraction": "needs-manual-selection", "selection_source": "operator-selected", "selection_note": "primary candidate (L4) fails R4 (jaccard 0.1724); L5 is a weaker fit for 'what was observed' but passes R4 (jaccard 0.0323) -- operator should consider both, this suggestion uses the R4-passing fallback", "reason": "no clean R2/R3 section heading found in this document (plain prose or a bulleted Q&A template, not markdown-headed sections)", "question_text": "This incident caused the GitLab.com service to be unavailable for many hours. We also lost some production data that we were eventually unable to recover. Specifically, we lost modifications to database data such as projects, comments, user accounts, issues and snippets, that took place between 17:20 and 00:00 UTC on January 31. Our best estimate is that it affected roughly 5,000 projects, 5,000 comments and 700 new user accounts. Code repositories or wikis hosted on GitLab.com were unavailable during the outage, but were not affected by the data loss. GitLab Enterprise customers, GitHost customers, and self-managed GitLab CE users were not affected by the outage, or the data loss.", "question_location": "line 4 (CC-proposed, replacing the prior L5 suggestion)", "gold_text": "On January 31st 2017, we experienced a major service outage for one of our products, the online service GitLab.com. The outage was caused by an accidental removal of data from our primary database server.", "gold_location": "line 3", "jaccard": 0.1311, "r4_status": "included", "gold_engram_ids": null, "gold_position": null, "candidates": [ { "text": "On January 31st 2017, we experienced a major service outage for one of our products, the online service GitLab.com. The outage was caused by an accidental removal of data from our primary database server.", "start_line": 3, "end_line": 3, "tags": [ "possible-gold(root-cause-language)", "possible-question(impact-language)" ] }, { "text": "This incident caused the GitLab.com service to be unavailable for many hours. We also lost some production data that we were eventually unable to recover. Specifically, we lost modifications to database data such as projects, comments, user accounts, issues and snippets, that took place between 17:20 and 00:00 UTC on January 31. Our best estimate is that it affected roughly 5,000 projects, 5,000 comments and 700 new user accounts. Code repositories or wikis hosted on GitLab.com were unavailable during the outage, but were not affected by the data loss. GitLab Enterprise customers, GitHost customers, and self-managed GitLab CE users were not affected by the outage, or the data loss.", "start_line": 4, "end_line": 4, "tags": [ "possible-question(impact-language)" ] }, { "text": "Losing production data is unacceptable. To ensure this does not happen again we're working on multiple improvements to our operations & recovery procedures for GitLab.com. In this article we'll look at what went wrong, what we did to recover, and what we'll do to prevent this from happening in the future.", "start_line": 5, "end_line": 5, "tags": [] }, { "text": "To the GitLab.com users whose data we lost and to the people affected by the outage: we're sorry. I apologize personally, as GitLab's CEO, and on behalf of everyone at GitLab.", "start_line": 6, "end_line": 6, "tags": [ "possible-question(impact-language)" ] }, { "text": "GitLab.com currently uses a single primary and a single secondary in hot-standby", "start_line": 7, "end_line": 7, "tags": [] }, { "text": "mode. The standby is only used for failover purposes. In this setup a single", "start_line": 8, "end_line": 8, "tags": [] }, { "text": "database has to handle all the load, which is not ideal. The primary's hostname", "start_line": 9, "end_line": 9, "tags": [] }, { "text": "is db1.cluster.gitlab.com, while the secondary's hostname is", "start_line": 10, "end_line": 10, "tags": [] }, { "text": "db2.cluster.gitlab.com.", "start_line": 11, "end_line": 11, "tags": [] }, { "text": "In the past we've had various other issues with this particular setup due to", "start_line": 12, "end_line": 12, "tags": [ "possible-gold(root-cause-language)" ] }, { "text": "db1.cluster.gitlab.com being a single point of failure. For example:", "start_line": 13, "end_line": 13, "tags": [] }, { "text": "On January 31st an engineer started setting up multiple PostgreSQL servers in our staging environment. The plan was to try out pgpool-II to see if it would reduce the load on our database by load balancing queries between the available hosts. Here is the issue for that plan: infrastructure#259.", "start_line": 14, "end_line": 14, "tags": [] }, { "text": "\u00b1 17:20 UTC: prior to starting this work, our engineer took an LVM snapshot of the production database and loaded this into the staging environment. This was necessary to ensure the staging database was up to date, allowing for more accurate load testing. This procedure normally happens automatically once every 24 hours (at 01:00 UTC), but they wanted a more up to date copy of the database.", "start_line": 15, "end_line": 15, "tags": [] }, { "text": "\u00b1 19:00 UTC: GitLab.com starts experiencing an increase in database load due to what we suspect was spam. In the week leading up to this event GitLab.com had been experiencing similar problems, but not this severe. One of the problems this load caused was that many users were not able to post comments on issues and merge requests. Getting the load under control took several hours.", "start_line": 16, "end_line": 16, "tags": [ "possible-gold(root-cause-language)", "possible-question(impact-language)" ] }, { "text": "We would later find out that part of the load was caused by a background job trying to remove a GitLab employee and their associated data. This was the result of their account being flagged for abuse and accidentally scheduled for removal. More information regarding this particular problem can be found in the issue \"Removal of users by spam should not hard delete\".", "start_line": 17, "end_line": 17, "tags": [ "possible-gold(root-cause-language)", "possible-question(impact-language)" ] }, { "text": "\u00b1 23:00 UTC: Due to the increased load, our PostgreSQL secondary's replication process started to lag behind. The replication failed as WAL segments needed by the secondary were already removed from the primary. As GitLab.com was not using WAL archiving, the secondary had to be re-synchronised manually. This involves removing the existing data directory on the secondary, and running pg_basebackup to copy over the database from the primary to the secondary.", "start_line": 18, "end_line": 18, "tags": [ "possible-gold(root-cause-language)" ] }, { "text": "One of the engineers went to the secondary and wiped the data directory, then", "start_line": 19, "end_line": 19, "tags": [] }, { "text": "ran pg_basebackup. Unfortunately pg_basebackup would hang, producing no", "start_line": 20, "end_line": 20, "tags": [] }, { "text": "meaningful output, despite the --verbose option being set. After a few tries", "start_line": 21, "end_line": 21, "tags": [] }, { "text": "pg_basebackup mentioned that it could not connect due to the master not having", "start_line": 22, "end_line": 22, "tags": [ "possible-gold(root-cause-language)" ] }, { "text": "enough available replication connections (as controlled by the max_wal_senders", "start_line": 23, "end_line": 23, "tags": [] }, { "text": "option).", "start_line": 24, "end_line": 24, "tags": [] }, { "text": "To resolve this our engineers decided to temporarily increase", "start_line": 25, "end_line": 25, "tags": [] }, { "text": "max_wal_senders from the default value of 3 to 32. When applying the", "start_line": 26, "end_line": 26, "tags": [] }, { "text": "settings, PostgreSQL refused to restart, claiming too many semaphores were being", "start_line": 27, "end_line": 27, "tags": [] }, { "text": "created. This can happen when, for example, max_connections is set too high. In", "start_line": 28, "end_line": 28, "tags": [] }, { "text": "our case this was set to 8000. Such a value is way too high, yet it had been", "start_line": 29, "end_line": 29, "tags": [] }, { "text": "applied almost a year ago and was working fine until that point. To resolve this", "start_line": 30, "end_line": 30, "tags": [] }, { "text": "the setting's value was reduced to 2000, resulting in PostgreSQL restarting", "start_line": 31, "end_line": 31, "tags": [] }, { "text": "without issues.", "start_line": 32, "end_line": 32, "tags": [] }, { "text": "Unfortunately this did not resolve the problem of pg_basebackup not starting", "start_line": 33, "end_line": 33, "tags": [] }, { "text": "replication immediately. One of the engineers decided to run it with strace to", "start_line": 34, "end_line": 34, "tags": [] }, { "text": "see what it was blocking on. strace showed that pg_basebackup was hanging in", "start_line": 35, "end_line": 35, "tags": [] }, { "text": "a poll call, but that did not provide any other meaningful information that might", "start_line": 36, "end_line": 36, "tags": [] }, { "text": "have explained why.", "start_line": 37, "end_line": 37, "tags": [] }, { "text": "\u00b1 23:30 UTC: one of the engineers thinks that perhaps pg_basebackup", "start_line": 38, "end_line": 38, "tags": [] }, { "text": "created some files in the PostgreSQL data directory of the secondary during the", "start_line": 39, "end_line": 39, "tags": [] }, { "text": "previous attempts to run it. While normally pg_basebackup prints an error when", "start_line": 40, "end_line": 40, "tags": [] }, { "text": "this is the case, the engineer in question wasn't too sure what was going on. It", "start_line": 41, "end_line": 41, "tags": [] }, { "text": "would later be revealed by another engineer (who wasn't around at the time) that", "start_line": 42, "end_line": 42, "tags": [] }, { "text": "this is normal behaviour: pg_basebackup will wait for the primary to start", "start_line": 43, "end_line": 43, "tags": [] }, { "text": "sending over replication data and it will sit and wait silently until that time.", "start_line": 44, "end_line": 44, "tags": [] }, { "text": "Unfortunately this was not clearly documented in our engineering", "start_line": 45, "end_line": 45, "tags": [] }, { "text": "runbooks nor in the official", "start_line": 46, "end_line": 46, "tags": [] }, { "text": "pg_basebackup document.", "start_line": 47, "end_line": 47, "tags": [] }, { "text": "Trying to restore the replication process, an engineer proceeds to wipe the PostgreSQL database directory, errantly thinking they were doing so on the secondary. Unfortunately this process was executed on the primary instead. The engineer terminated the process a second or two after noticing their mistake, but at this point around 300 GB of data had already been removed.", "start_line": 48, "end_line": 48, "tags": [] }, { "text": "Hoping they could restore the database the engineers involved went to look for the database backups, and asked for help on Slack. Unfortunately the process of both finding and using backups failed completely.", "start_line": 49, "end_line": 49, "tags": [] }, { "text": "This brings us to the recovery procedures. Normally in an event like this, one should be able to restore a database in relatively little time using a recent backup, though some form of data loss can not always be prevented. For GitLab.com we have the following procedures in place:", "start_line": 50, "end_line": 50, "tags": [] }, { "text": "pg_dump, this backup is uploaded", "start_line": 51, "end_line": 51, "tags": [] }, { "text": "to Amazon S3. Old backups are automatically removed after some time.", "start_line": 52, "end_line": 52, "tags": [] }, { "text": "At this point the replication process was broken and data had already been wiped from both the primary and secondary, meaning we could not restore from either host.", "start_line": 53, "end_line": 53, "tags": [] }, { "text": "When we went to look for the pg_dump backups we found out they were not there.", "start_line": 54, "end_line": 54, "tags": [] }, { "text": "The S3 bucket was empty, and there was no recent backup to be found anywhere.", "start_line": 55, "end_line": 55, "tags": [] }, { "text": "Upon closer inspection we found out that the backup procedure was using", "start_line": 56, "end_line": 56, "tags": [] }, { "text": "pg_dump 9.2, while our database is running PostgreSQL 9.6 (for Postgres, 9.x", "start_line": 57, "end_line": 57, "tags": [] }, { "text": "releases are considered major). A difference in major versions results in", "start_line": 58, "end_line": 58, "tags": [] }, { "text": "pg_dump producing an error, terminating the backup procedure.", "start_line": 59, "end_line": 59, "tags": [] }, { "text": "The difference is the result of how our Omnibus package works. We currently", "start_line": 60, "end_line": 60, "tags": [] }, { "text": "support both PostgreSQL 9.2 and 9.6, allowing users to upgrade (either manually", "start_line": 61, "end_line": 61, "tags": [ "possible-question(impact-language)" ] }, { "text": "or using commands provided by the package). To determine the correct version to", "start_line": 62, "end_line": 62, "tags": [] }, { "text": "use the Omnibus package looks at the PostgreSQL version of the database cluster", "start_line": 63, "end_line": 63, "tags": [] }, { "text": "(as determined by $PGDIR/PG_VERSION, with $PGDIR being the path to the data", "start_line": 64, "end_line": 64, "tags": [] }, { "text": "directory). When PostgreSQL 9.6 is detected Omnibus ensures all binaries use", "start_line": 65, "end_line": 65, "tags": [ "possible-question(impact-language)" ] }, { "text": "PostgreSQL 9.6, otherwise it defaults to PostgreSQL 9.2.", "start_line": 66, "end_line": 66, "tags": [] }, { "text": "The pg_dump procedure was executed on a regular application server, not the", "start_line": 67, "end_line": 67, "tags": [] }, { "text": "database server. As a result there is no PostgreSQL data directory present on", "start_line": 68, "end_line": 68, "tags": [] }, { "text": "these servers, thus Omnibus defaults to PostgreSQL 9.2. This in turn resulted in", "start_line": 69, "end_line": 69, "tags": [] }, { "text": "pg_dump terminating with an error.", "start_line": 70, "end_line": 70, "tags": [] }, { "text": "While notifications are enabled for any cronjobs that error, these notifications are sent by email. For GitLab.com we use DMARC. Unfortunately DMARC was not enabled for the cronjob emails, resulting in them being rejected by the receiver. This means we were never aware of the backups failing, until it was too late.", "start_line": 71, "end_line": 71, "tags": [] }, { "text": "Azure disk snapshots are used to generate a snapshot of an entire disk. These snapshots don't make it easy to restore individual chunks of data (e.g. a lost user account), though it's possible. The primary purpose is to restore entire disks in case of disk failure.", "start_line": 72, "end_line": 72, "tags": [] }, { "text": "In Azure a snapshot belongs to a storage account, and a storage account in turn is linked to one or more hosts. Each storage account has a limit of roughly 30 TB. When restoring a snapshot using a host in the same storage account, the procedure usually completes very quickly. However, when using a host in a different storage account the procedure can take hours if not days to complete. For example, in one such case it took over a week to restore a snapshot. As a result we try not to rely on this system too much.", "start_line": 73, "end_line": 73, "tags": [] }, { "text": "While enabled for the NFS servers, these snapshots were not enabled for any of the database servers as we assumed that our other backup procedures were sufficient enough.", "start_line": 74, "end_line": 74, "tags": [] }, { "text": "The LVM snapshots are primarily used to easily copy data from our production environment to our staging environment. While this process was working as intended, the produced snapshots are not really meant to be used for disaster recovery. At the time of the outage we had two snapshots available:", "start_line": 75, "end_line": 75, "tags": [ "possible-question(impact-language)" ] }, { "text": "When we generate a snapshot the following steps are taken:", "start_line": 76, "end_line": 76, "tags": [] }, { "text": "To recover GitLab.com we decided to use the LVM snapshot created 6 hours before the outage, as it was our only option to reduce data loss as much as possible (the alternative was to lose almost 24 hours of data). This process would involve the following steps:", "start_line": 77, "end_line": 77, "tags": [ "possible-question(impact-language)" ] }, { "text": "For our staging environment we were using Azure classic, without Premium Storage. This is primarily done to save costs as premium storage is quite expensive. As a result the disks are very slow, resulting in them being the main bottleneck in the restoration process. Because LVM snapshots are stored on the hosts they are taken for we had two options to restore data:", "start_line": 78, "end_line": 78, "tags": [] }, { "text": "In both cases the amount of data to copy would be roughly the same. Since copying over and restoring the data directory would be easier we decided to go with this solution.", "start_line": 79, "end_line": 79, "tags": [] }, { "text": "Copying the data from the staging to the production host took around 18 hours. These disks are network disks and are throttled to a really low number (around 60Mbps), there is no way to move from cheap storage to premium, so this was the performance we would get out of it. There was no network or processor bottleneck, the bottleneck was in the drives. Once copied we were able to restore the database (including webhooks) to the state it was at January 31st, 17:20 UTC.", "start_line": 80, "end_line": 80, "tags": [] }, { "text": "On February 1st at 17:00 UTC we managed to restore the GitLab.com database without webhooks. Restoring webhooks was done by creating a separate staging database using the LVM snapshot, but without triggering the removal of webhooks. This allowed us to generate a SQL dump of the table and import this into the restored GitLab.com database.", "start_line": 81, "end_line": 81, "tags": [] }, { "text": "Around 18:00 UTC we finished the final restoration procedures such as restoring the webhooks and confirming everything was operating as expected.", "start_line": 82, "end_line": 82, "tags": [] }, { "text": "In the spirit of transparency we kept track of progress and notes in a publicly visible Google document. We also streamed the recovery procedure on YouTube, with a peak viewer count of around 5000 (resulting in the stream being the #2 live stream on YouTube for several hours). The stream was used to give our users live updates about the recovery procedure. Finally we used Twitter (https://twitter.com/gitlabstatus) to inform those that might not be watching the stream.", "start_line": 83, "end_line": 83, "tags": [ "possible-question(impact-language)" ] }, { "text": "The document in question was initially private to GitLab employees and contained name of the engineer who accidentally removed the data. While the name was added by the engineer themselves (and they had no problem with this being public), we will redact names in future cases as other engineers may not be comfortable with their name being published.", "start_line": 84, "end_line": 84, "tags": [] }, { "text": "Database data such as projects, issues, snippets, etc. created between January 31st 17:20 UTC and 23:30 UTC has been lost. Git repositories and Wikis were not removed as they are stored separately.", "start_line": 85, "end_line": 85, "tags": [] }, { "text": "It's hard to estimate how much data has been lost exactly, but we estimate we have lost at least 5000 projects, 5000 comments, and roughly 700 users. This only affected users of GitLab.com, self-managed instances or GitHost instances were not affected.", "start_line": 86, "end_line": 86, "tags": [ "possible-question(impact-language)" ] }, { "text": "Since GitLab uses GitLab.com to develop GitLab the outage meant that for some it was harder to get work done. Most developers could continue working using their local Git repositories, but creating issues and such had to be delayed. To publish the blog post \"GitLab.com Database Incident\" we used a private GitLab instance we normally use for private/sensitive workflows (e.g. security releases). This allowed us to build and deploy a new version of the website while GitLab.com was unavailable.", "start_line": 87, "end_line": 87, "tags": [ "possible-question(impact-language)" ] }, { "text": "We also have a public monitoring website located at https://dashboards.gitlab.com/. Unfortunately the current setup for this website was not able to handle the load produced by users using this service during the outage. Fortunately our internal monitoring systems (which dashboards.gitlab.com is based on) were not affected.", "start_line": 88, "end_line": 88, "tags": [ "possible-question(impact-language)" ] }, { "text": "To analyse the root cause of these problems we'll use a technique called \"The 5 Whys\". We'll break up the incident into 2 main problems: GitLab.com being down, and it taking a long time to restore GitLab.com.", "start_line": 89, "end_line": 89, "tags": [ "possible-gold(root-cause-language)" ] }, { "text": "Problem 1: GitLab.com was down for about 18 hours.", "start_line": 90, "end_line": 90, "tags": [ "possible-question(impact-language)" ] }, { "text": "Problem 2: restoring GitLab.com took over 18 hours.", "start_line": 91, "end_line": 91, "tags": [] }, { "text": "We are currently working on fixing and improving our various recovery procedures. Work is split across the following issues:", "start_line": 92, "end_line": 92, "tags": [] }, { "text": "We are also working on setting up multiple secondaries and balancing the load amongst these hosts. More information on this can be found at:", "start_line": 93, "end_line": 93, "tags": [] }, { "text": "Our main focus is to improve disaster recovery, and making it more obvious as to", "start_line": 94, "end_line": 94, "tags": [] }, { "text": "what host you're using; instead of preventing production engineers from running", "start_line": 95, "end_line": 95, "tags": [] }, { "text": "certain commands. For example, one could alias rm to something safer but in", "start_line": 96, "end_line": 96, "tags": [] }, { "text": "doing so would only protect themselves against accidentally running rm -rf /important-data, not against disk corruption or any of the many other ways you", "start_line": 97, "end_line": 97, "tags": [] }, { "text": "can lose data.", "start_line": 98, "end_line": 98, "tags": [] }, { "text": "An ideal environment is one in which you can make mistakes but easily and quickly recover from them with minimal to no impact. This in turn requires you to be able to perform these procedures on a regular basis, and make it easy to test and roll back any changes. For example, we are in the process of setting up procedures that allow developers to test their database migrations. More information on this can be found in the issue \"Tool for executing and reverting Rails migrations on staging\".", "start_line": 99, "end_line": 99, "tags": [ "possible-question(impact-language)" ] }, { "text": "We're also looking into ways to build better recovery procedures for the entire GitLab.com infrastructure, and not just the database; and to ensure there is ownership of these procedures. The issue for this is \"Disaster recovery for everything that is not the database\".", "start_line": 100, "end_line": 100, "tags": [] }, { "text": "Monitoring wise we also started working on a public backup monitoring dashboard,", "start_line": 101, "end_line": 101, "tags": [] }, { "text": "which can be found at https://dashboards.gitlab.com/dashboard/db/postgresql-backups.", "start_line": 102, "end_line": 102, "tags": [] }, { "text": "Currently this dashboard only contains data of our pg_dump backup procedure,", "start_line": 103, "end_line": 103, "tags": [] }, { "text": "but we aim to add more data over time.", "start_line": 104, "end_line": 104, "tags": [] }, { "text": "One might notice that at the moment our pg_dump backups are 3 days old. We", "start_line": 105, "end_line": 105, "tags": [] }, { "text": "perform these backups on a secondary as pg_dump can put quite a bit of", "start_line": 106, "end_line": 106, "tags": [] }, { "text": "pressure on a database. Since we are in the process of rebuilding our", "start_line": 107, "end_line": 107, "tags": [] }, { "text": "secondaries the pg_dump backup procedure is suspended for the time being. Fear", "start_line": 108, "end_line": 108, "tags": [] }, { "text": "not however, as LVM snapshots are now taken every hour instead of once per 24", "start_line": 109, "end_line": 109, "tags": [] }, { "text": "hours. Enabling Azure disk snapshots is something we're still looking into.", "start_line": 110, "end_line": 110, "tags": [] }, { "text": "Finally, we're looking into improving our abuse reporting and response system. More information regarding this can be found in the issue \"Removal of users by spam should not hard delete\".", "start_line": 111, "end_line": 111, "tags": [ "possible-question(impact-language)" ] }, { "text": "If you think there are additional measures we can take to prevent incidents like this please let us know in the comments.", "start_line": 112, "end_line": 112, "tags": [] }, { "text": "Pushing to the default branch will automatically update the merge request so that it's aware of there not being any differences between the source and target branch. At this point you can safely close the merge request.", "start_line": 113, "end_line": 113, "tags": [] }, { "text": "There are 3 options to resolve this:", "start_line": 114, "end_line": 114, "tags": [] }, { "text": "Go to your project, then \"Pipelines\", \"New Pipeline\", use \"master\" as the branch, then create the pipeline. This will create and start a new pipeline using your master branch, which should result in your website being updated.", "start_line": 115, "end_line": 115, "tags": [] }, { "text": "Most likely they were, but the database is not aware of this. To solve this, create a new pipeline using the right branch and run it.", "start_line": 116, "end_line": 116, "tags": [] }, { "text": "Pushing new commits should automatically solve this. Alternatively you can try force pushing to the target branch.", "start_line": 117, "end_line": 117, "tags": [] }, { "text": "Project details are stored in the database. This meant that this data was lost for projects created after 17:20. We ran a procedure to restore these projects based on their Git repositories that were still stored in our NFS cluster. This procedure however was only able to restore projects in their most basic form, without associated data such as issues and merge requests.", "start_line": 118, "end_line": 118, "tags": [] } ], "operator_disposition": "keep, question text CC-proposed (APPROVED as-is)", "r4_note": "Recomputed against build_probe.py's own jaccard(): 0.1311, INCLUDED. The pre-registration doc's original adjudication table stated 0.1724/EXCLUDED for this same L3-vs-L4 pairing -- that figure does not reproduce under the pinned tokenizer/jaccard implementation against the exact stored candidate text and is flagged here as a correction, not silently swapped in." }, { "doc_id": "03_gitlab_issue_17057.md", "family": "cause-from-symptom", "extraction": "needs-manual-selection", "selection_source": "operator-selected", "selection_note": null, "reason": "no clean R2/R3 section heading found in this document (plain prose or a bulleted Q&A template, not markdown-headed sections)", "question_text": "- Customers trying to access GitLab.com between 2023-10-30 15:27 UTC to 16:15 UTC. Duration of 48 minutes of service disruptions.", "question_location": "line 14 (CC-proposed, replacing the prior thin L10 candidate)", "gold_text": "The root cause was database saturation on the merge_requests table caused by bulk import jobs. #17054 (comment 1627154963). Bulk import support was introduced a few months ago and a feature flag was flipped enabling that functionality. This caused:", "gold_location": "line 18", "jaccard": 0.0, "r4_status": "included", "gold_engram_ids": null, "gold_position": null, "candidates": [ { "text": "Incident Review: 2023-10-30 Gitlab.com is down", "start_line": 3, "end_line": 3, "tags": [] }, { "text": "Incident issue: #17054 (closed)", "start_line": 4, "end_line": 4, "tags": [] }, { "text": "The DRI for the incident review is the issue assignee.", "start_line": 5, "end_line": 5, "tags": [] }, { "text": "- If applicable, ensure that the exec summary is completed at the top of the associated incident issue, the timeline tab is updated and relevant graphs are included.", "start_line": 6, "end_line": 6, "tags": [] }, { "text": "- If there are any corrective actions or infradev issues, ensure they are added as related issues to the original incident.", "start_line": 7, "end_line": 7, "tags": [] }, { "text": "- Fill out relevant sections below or link to the meeting review notes that cover these topics https://docs.google.com/document/d/1jrX-Z2NJrNjBBcywY7emQKwaKRqVAlDRdGG0Krk76ys/edit#", "start_line": 8, "end_line": 8, "tags": [] }, { "text": "- Who was impacted by this incident? (i.e. external customers, internal customers)", "start_line": 9, "end_line": 9, "tags": [ "possible-question(impact-language)" ] }, { "text": "- All GitLab.com users.", "start_line": 10, "end_line": 10, "tags": [ "possible-question(impact-language)" ] }, { "text": "- What was the customer experience during the incident? (i.e. preventing them from doing X, incorrect display of Y, ...)", "start_line": 11, "end_line": 11, "tags": [] }, { "text": "- The primary database for GitLab.com was saturated due to a spike in bulk import jobs. This caused an intermittent disruption in accessing GitLab.com including web services, API services, and Git operations.", "start_line": 12, "end_line": 12, "tags": [ "possible-gold(root-cause-language)" ] }, { "text": "- How many customers were affected?", "start_line": 13, "end_line": 13, "tags": [ "possible-question(impact-language)" ] }, { "text": "- Customers trying to access GitLab.com between 2023-10-30 15:27 UTC to 16:15 UTC. Duration of 48 minutes of service disruptions.", "start_line": 14, "end_line": 14, "tags": [ "possible-question(impact-language)" ] }, { "text": "- If a precise customer impact number is unknown, what is the estimated impact (number and ratio of failed requests, amount of traffic drop, ...)?", "start_line": 15, "end_line": 15, "tags": [ "possible-question(impact-language)" ] }, { "text": "- All GitLab.com users were impacted.", "start_line": 16, "end_line": 16, "tags": [ "possible-question(impact-language)" ] }, { "text": "A detailed root cause discussion is available here.", "start_line": 17, "end_line": 17, "tags": [ "possible-gold(root-cause-language)" ] }, { "text": "The root cause was database saturation on the merge_requests table caused by bulk import jobs. #17054 (comment 1627154963). Bulk import support was introduced a few months ago and a feature flag was flipped enabling that functionality. This caused:", "start_line": 18, "end_line": 18, "tags": [ "possible-gold(root-cause-language)" ] }, { "text": "- High UPDATE rate on merge_requests caused the frequent queries against that table to become progressively less efficient, saturating the db connection pool.", "start_line": 19, "end_line": 19, "tags": [] }, { "text": "- Replication lag exceeding the configured staleness tolerance caused additional demand on the primary db's connection pool, since most read-only queries were no longer being offloaded to the replica dbs.", "start_line": 20, "end_line": 20, "tags": [] }, { "text": "- How was the incident detected?", "start_line": 21, "end_line": 21, "tags": [ "possible-question(impact-language)" ] }, { "text": "- Noticed drop on RPS (Requests per second) and DB saturation. Initially received alerts about Apdex SLO drop for all front end components. #17054 (comment 1627147834)", "start_line": 22, "end_line": 22, "tags": [ "possible-question(impact-language)" ] }, { "text": "- How could detection time be improved?", "start_line": 23, "end_line": 23, "tags": [] }, { "text": "- Looking at the timeline in the incident and the alert from Pagerduty, the engineer on call (EOC) received an alert at 15:27 UTC and declared an incident in that same moment, so the detection time was immediate.", "start_line": 24, "end_line": 24, "tags": [] }, { "text": "- Looking at the timeline in the incident and the alert from Pagerduty, the engineer on call (EOC) received an alert at", "start_line": 25, "end_line": 25, "tags": [] }, { "text": "- How was the root cause diagnosed?", "start_line": 26, "end_line": 26, "tags": [ "possible-gold(root-cause-language)" ] }, { "text": "- After some investigation we noticed that patroni-main primary node was experiencing high CPU load. We also saw table bloat and dead tuple increase for the following tables: merge_requests ,merge_requests_diffs_commits . After reviewing the database activity we identified a set of queries with INSERT/UPDATE statements to themerge_requests table, that were related to a specific correlation ID. This helped us identify the problem was related to a bulk import and theBulkImports::PipelineBatchWorker class. #17054 (comment 1627147834)", "start_line": 27, "end_line": 27, "tags": [ "possible-question(impact-language)" ] }, { "text": "- After some investigation we noticed that patroni-main primary node was experiencing high CPU load. We also saw table bloat and dead tuple increase for the following tables:", "start_line": 28, "end_line": 28, "tags": [ "possible-question(impact-language)" ] }, { "text": "- How could time to diagnosis be improved?", "start_line": 29, "end_line": 29, "tags": [] }, { "text": "- In terms of the Direct Transfer/Bulk Import feature, it could be helpful to track what BulkImports are active at any given time. If there's a similar interruption of service, and we can see that a large number of bulk import jobs have been created, it could provide a clue that bulk import may have caused the interruption.", "start_line": 30, "end_line": 30, "tags": [] }, { "text": "- Grafana dashboards for individual services should contain links to their related runbooks. This would make it easier for the EOC to refer to them, without having to search the whole repo.", "start_line": 31, "end_line": 31, "tags": [] }, { "text": "- How did we reach the point where we knew how to mitigate the impact?", "start_line": 32, "end_line": 32, "tags": [ "possible-question(impact-language)" ] }, { "text": "- After identifying an increase in the merge_requests table being caused by BulkImports, we temporarily disabled import functionality for all of GitLab.com via the Admin setting. This prevented the creation of new import jobs and allowed us time to continue investigating. We then managed to identify a specific bulk import job as a perpetrator also disabled thebulk_imports_batched_import_export feature via a feature flag to prevent another occurrence. Once the feature flag was disabled, we reenabled the import functionality using individual jobs rather than batched jobs.", "start_line": 33, "end_line": 33, "tags": [ "possible-gold(root-cause-language)" ] }, { "text": "- After identifying an increase in the", "start_line": 34, "end_line": 34, "tags": [] }, { "text": "- How could time to mitigation be improved?", "start_line": 35, "end_line": 35, "tags": [] }, { "text": "- Once the diagnoses were made, the time to mitigation was pretty good. It could have been improved by having a PM from the import and integrate team on the zoom call, simply to make relevant decisions faster, but communication was done async in slack and there were no inordinate delays. I also think that time to mitigation shouldn't be rushed as such, as mitigations can have adverse customer impact.", "start_line": 36, "end_line": 36, "tags": [ "possible-question(impact-language)" ] }, { "text": "- Did we have other events in the past with the same root cause?", "start_line": 37, "end_line": 37, "tags": [ "possible-gold(root-cause-language)" ] }, { "text": "- No", "start_line": 38, "end_line": 38, "tags": [] }, { "text": "- Do we have existing backlog items that would've prevented or greatly reduced the impact of this incident?", "start_line": 39, "end_line": 39, "tags": [ "possible-question(impact-language)" ] }, { "text": "- No", "start_line": 40, "end_line": 40, "tags": [] }, { "text": "- Was this incident triggered by a change (deployment of code or change to infrastructure)? If yes, link the issue.", "start_line": 41, "end_line": 41, "tags": [] }, { "text": "- Enabled bulk_imports_batched_import_export feature flag on.com which was introduced in gitlab-org/gitlab!124434 (merged) (issue link: gitlab-org/gitlab#391224 (closed)). Note that this change was done several months prior.", "start_line": 42, "end_line": 42, "tags": [] }, { "text": "- Enabled", "start_line": 43, "end_line": 43, "tags": [] }, { "text": "- Lots of help from different team members jumping into help. We were able to quickly pull in the necessary domain expertise from DBRE and the Import groups.", "start_line": 44, "end_line": 44, "tags": [] }, { "text": "- Google doc was used for organizing notes while .com was down.", "start_line": 45, "end_line": 45, "tags": [] }, { "text": "- Customers and e-group were kept informed with regular status updates", "start_line": 46, "end_line": 46, "tags": [ "possible-question(impact-language)" ] } ], "operator_disposition": "keep, question text CC-proposed (APPROVED as-is)" }, { "doc_id": "04_gitlab_issue_18490.md", "family": "cause-from-symptom", "extraction": "needs-manual-selection", "selection_source": "operator-selected", "selection_note": null, "reason": "no clean R2/R3 section heading found in this document (plain prose or a bulleted Q&A template, not markdown-headed sections)", "question_text": "- Customer Impact: Most GitLab services unavailable for all users", "question_location": "line 6", "gold_text": "- Root cause: Manual database restart during scheduled production change and maintenance work.", "gold_location": "line 9", "jaccard": 0.0, "r4_status": "included", "gold_engram_ids": null, "gold_position": null, "candidates": [ { "text": "2024-09-03: GitLab.com is down", "start_line": 3, "end_line": 3, "tags": [] }, { "text": "Services on GitLab.com that depend on postgres were not reachable for several minutes because the cluster had been restarted.", "start_line": 4, "end_line": 4, "tags": [] }, { "text": "The patroni clusters for v14-main and v14-ci were restarted (postgres was restarted). They are recovered now.", "start_line": 5, "end_line": 5, "tags": [] }, { "text": "- Customer Impact: Most GitLab services unavailable for all users", "start_line": 6, "end_line": 6, "tags": [ "possible-question(impact-language)" ] }, { "text": "- Service Impact:", "start_line": 7, "end_line": 7, "tags": [ "possible-question(impact-language)" ] }, { "text": "- Impact Duration: 5:00 UTC - 5:20 UTC (20 minutes)", "start_line": 8, "end_line": 8, "tags": [ "possible-question(impact-language)" ] }, { "text": "- Root cause: Manual database restart during scheduled production change and maintenance work.", "start_line": 9, "end_line": 9, "tags": [ "possible-gold(root-cause-language)" ] }, { "text": "Recent Events (available internally only):", "start_line": 10, "end_line": 10, "tags": [] }, { "text": "- Feature Flag Log - Chatops to toggle Feature Flags Documentation", "start_line": 11, "end_line": 11, "tags": [] }, { "text": "- Infrastructure Configurations", "start_line": 12, "end_line": 12, "tags": [] }, { "text": "- GCP Events (e.g. host failure)", "start_line": 13, "end_line": 13, "tags": [] }, { "text": "Use the following links to create related issues to this incident if additional work needs to be completed after it is resolved:", "start_line": 14, "end_line": 14, "tags": [] }, { "text": "Note: In some cases we need to redact information from public view. We only do this in a limited number of documented cases. This might include the summary, timeline or any other bits of information, laid out in our handbook page. Any of this confidential data will be in a linked issue, only visible internally. By default, all information we can share, will be public, in accordance to our transparency value.", "start_line": 15, "end_line": 15, "tags": [] }, { "text": "Security Note: If anything abnormal is found during the course of your investigation, please do not hesitate to contact security.", "start_line": 16, "end_line": 16, "tags": [] } ], "operator_disposition": "drop" }, { "doc_id": "05_gitlab_issue_18596.md", "family": "cause-from-symptom", "extraction": "needs-manual-selection", "selection_source": "operator-selected", "selection_note": null, "reason": "no clean R2/R3 section heading found in this document (plain prose or a bulleted Q&A template, not markdown-headed sections)", "question_text": "- Customer Impact: Users of GitLab.com would see slow API, Web and git responses.", "question_location": "line 6", "gold_text": "- Root cause: Excessive traffic to a specific endpoint from a small number of users.", "gold_location": "line 9, trimmed to the root-cause sentence only (drops the unrelated 'See internal note' tail)", "jaccard": 0.0526, "r4_status": "included", "gold_engram_ids": null, "gold_position": null, "candidates": [ { "text": "2024-09-21: Elevated error rate across api, git and frontend services", "start_line": 3, "end_line": 3, "tags": [] }, { "text": "Customers were impacted for approximately 2 hour period in total where the response from the API was slow or produced errors. This started at at 11:46 UTC to 14:04 UTC. See logs", "start_line": 4, "end_line": 4, "tags": [ "possible-question(impact-language)" ] }, { "text": "More information will be added as we investigate the issue. For customers believed to be affected by this incident, please subscribe to this issue or monitor our status page for further updates.", "start_line": 5, "end_line": 5, "tags": [ "possible-question(impact-language)" ] }, { "text": "- Customer Impact: Users of GitLab.com would see slow API, Web and git responses.", "start_line": 6, "end_line": 6, "tags": [ "possible-question(impact-language)" ] }, { "text": "- Service Impact:", "start_line": 7, "end_line": 7, "tags": [ "possible-question(impact-language)" ] }, { "text": "- Impact Duration: 11:46UTC - 14:04UTC (138mins)", "start_line": 8, "end_line": 8, "tags": [ "possible-question(impact-language)" ] }, { "text": "- Root cause: Excessive traffic to a specific endpoint from a small number of users. See internal note", "start_line": 9, "end_line": 9, "tags": [ "possible-gold(root-cause-language)", "possible-question(impact-language)" ] }, { "text": "Recent Events (available internally only):", "start_line": 10, "end_line": 10, "tags": [] }, { "text": "- Feature Flag Log - Chatops to toggle Feature Flags Documentation", "start_line": 11, "end_line": 11, "tags": [] }, { "text": "- Infrastructure Configurations", "start_line": 12, "end_line": 12, "tags": [] }, { "text": "- GCP Events (e.g. host failure)", "start_line": 13, "end_line": 13, "tags": [] }, { "text": "Use the following links to create related issues to this incident if additional work needs to be completed after it is resolved:", "start_line": 14, "end_line": 14, "tags": [] }, { "text": "Note: In some cases we need to redact information from public view. We only do this in a limited number of documented cases. This might include the summary, timeline or any other bits of information, laid out in our handbook page. Any of this confidential data will be in a linked issue, only visible internally. By default, all information we can share, will be public, in accordance to our transparency value.", "start_line": 15, "end_line": 15, "tags": [] }, { "text": "Security Note: If anything abnormal is found during the course of your investigation, please do not hesitate to contact security.", "start_line": 16, "end_line": 16, "tags": [] } ], "operator_disposition": "keep as edited (operator-ratified gold edit)" }, { "doc_id": "06_gitlab_issue_15999.md", "family": "cause-from-symptom", "extraction": "needs-manual-selection", "selection_source": "operator-selected", "selection_note": null, "reason": "no clean R2/R3 section heading found in this document (plain prose or a bulleted Q&A template, not markdown-headed sections)", "question_text": "- GitLab.com was unavailable on 2023-07-07 from 16:25 UTC to 18:42 UTC. During this time the web and API interfaces were not available (503). Customers were able to perform git actions via the command line.", "question_location": "line 10", "gold_text": "- The root cause was an out-of-sync infrastructure configuration plan (Terraform) executed against our production environment.", "gold_location": "line 21", "jaccard": 0.0, "r4_status": "included", "gold_engram_ids": null, "gold_position": null, "candidates": [ { "text": "The DRI for the incident review is the issue assignee.", "start_line": 3, "end_line": 3, "tags": [] }, { "text": "- If applicable, ensure that the exec summary is completed at the top of the associated incident issue, the timeline tab is updated and relevant graphs are included.", "start_line": 4, "end_line": 4, "tags": [] }, { "text": "- If there are any corrective actions or infradev issues, ensure they are added as related issues to the original incident.", "start_line": 5, "end_line": 5, "tags": [] }, { "text": "- Fill out relevant sections below or link to the meeting review notes that cover these topics", "start_line": 6, "end_line": 6, "tags": [] }, { "text": "- Who was impacted by this incident? (i.e. external customers, internal customers)", "start_line": 7, "end_line": 7, "tags": [ "possible-question(impact-language)" ] }, { "text": "- All users of gitlab.com, including external and internal customers.", "start_line": 8, "end_line": 8, "tags": [ "possible-question(impact-language)" ] }, { "text": "- What was the customer experience during the incident? (i.e. preventing them from doing X, incorrect display of Y, ...)", "start_line": 9, "end_line": 9, "tags": [] }, { "text": "- GitLab.com was unavailable on 2023-07-07 from 16:25 UTC to 18:42 UTC. During this time the web and API interfaces were not available (503). Customers were able to perform git actions via the command line.", "start_line": 10, "end_line": 10, "tags": [ "possible-question(impact-language)" ] }, { "text": "- For customers that did not have DNS records cached, Container Registry was unavailable on 2023-07-07 from 16:25 UTC to 19:36 UTC.", "start_line": 11, "end_line": 11, "tags": [ "possible-question(impact-language)" ] }, { "text": "- A small number of git pushes on 2023-07-07 from 15:55 UTC to 16:17 UTC are not available on GitLab.com until the changes are pushed again from a local copy.", "start_line": 12, "end_line": 12, "tags": [] }, { "text": "- We have restored data to known recovery points, and a small subset of customer projects requires a refresh using their local copy.", "start_line": 13, "end_line": 13, "tags": [] }, { "text": "- The impacted project owners have been notified and were advised to re-push their changes.", "start_line": 14, "end_line": 14, "tags": [ "possible-question(impact-language)" ] }, { "text": "- How many customers were affected?", "start_line": 15, "end_line": 15, "tags": [ "possible-question(impact-language)" ] }, { "text": "- All customers.", "start_line": 16, "end_line": 16, "tags": [ "possible-question(impact-language)" ] }, { "text": "- Ongoing investigation: https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/24086", "start_line": 17, "end_line": 17, "tags": [] }, { "text": "- If a precise customer impact number is unknown, what is the estimated impact (number and ratio of failed requests, amount of traffic drop, ...)?", "start_line": 18, "end_line": 18, "tags": [ "possible-question(impact-language)" ] }, { "text": "- All customers.", "start_line": 19, "end_line": 19, "tags": [ "possible-question(impact-language)" ] }, { "text": "An outdated production configuration was applied to our production environment, which caused several GitLab.com production services to be removed and replaced.", "start_line": 20, "end_line": 20, "tags": [] }, { "text": "- The root cause was an out-of-sync infrastructure configuration plan (Terraform) executed against our production environment.", "start_line": 21, "end_line": 21, "tags": [ "possible-gold(root-cause-language)" ] }, { "text": "- This infrastructure configuration plan was prepared 3-weeks prior in preparation for our production database upgrade.", "start_line": 22, "end_line": 22, "tags": [] }, { "text": "- Environmental drifts accumulated during those 3 weeks causing the planned configuration and production environment to become out-of-sync.", "start_line": 23, "end_line": 23, "tags": [] }, { "text": "- Executing the out-of-sync plan caused an unintended removal of production services which resulted in the outage.", "start_line": 24, "end_line": 24, "tags": [ "possible-question(impact-language)" ] }, { "text": "- We typically execute configuration plans shortly after they are prepared. However, the execution of this 3-week-old configuration plan exposed a gap in our process.", "start_line": 25, "end_line": 25, "tags": [] }, { "text": "-", "start_line": 26, "end_line": 26, "tags": [] }, { "text": "How was the incident detected?", "start_line": 27, "end_line": 27, "tags": [ "possible-question(impact-language)" ] }, { "text": "- 16:10 UTC - A job begins to apply that is applying an old TF configuration", "start_line": 28, "end_line": 28, "tags": [] }, { "text": "- 16:15 UTC - EOC received Blackbox PagerDuty notifications about https://cdn-artifacts.gitlab-static.net URLs failing", "start_line": 29, "end_line": 29, "tags": [] }, { "text": "- 16:15 UTC - DBRE posted in the #production channel that they were applying a terraform plan via CI", "start_line": 30, "end_line": 30, "tags": [] }, { "text": "- 16:18 UTC - EOC declared an S2 incident", "start_line": 31, "end_line": 31, "tags": [] }, { "text": "- 16:20 UTC - Based on slack reports and personal observation of 5xx errors on GitLab.com, EOC attempted to upgrade incident to an S1.", "start_line": 32, "end_line": 32, "tags": [] }, { "text": "- It was five minutes from when the job began applying destructive changes until the EOC was notified of an initial problem. And ten minutes until it was clear that this was an S1 site outage incident.", "start_line": 33, "end_line": 33, "tags": [ "possible-question(impact-language)" ] }, { "text": "-", "start_line": 34, "end_line": 34, "tags": [] }, { "text": "How could detection time be improved?", "start_line": 35, "end_line": 35, "tags": [] }, { "text": "- NA.", "start_line": 36, "end_line": 36, "tags": [] }, { "text": "-", "start_line": 37, "end_line": 37, "tags": [] }, { "text": "How was the root cause diagnosed?", "start_line": 38, "end_line": 38, "tags": [ "possible-gold(root-cause-language)" ] }, { "text": "- 18:25 UTC - Checked Cloudflare status page", "start_line": 39, "end_line": 39, "tags": [] }, { "text": "- 18:25 UTC - DBRE brings the calls attention to a running Terraform apply job", "start_line": 40, "end_line": 40, "tags": [] }, { "text": "- 18:28 UTC - First look at the MR attached to the pipeline seems harmless", "start_line": 41, "end_line": 41, "tags": [] }, { "text": "- 18:30 UTC - Examining the running Terraform apply job revealed several resources being destroyed. Referencing the plan for the pipeline showed 617 resources to be destroyed.", "start_line": 42, "end_line": 42, "tags": [] }, { "text": "- 18:31 UTC - The job was stopped to try and prevent further destruction.", "start_line": 43, "end_line": 43, "tags": [] }, { "text": "- 18:34 UTC - A local plan against the production environment was run by the EOC to see what cloud resources were missing.", "start_line": 44, "end_line": 44, "tags": [] }, { "text": "- 18:37 UTC - At this point, it appeared likely that the applied plan had caused the outage due to destroyed resources.", "start_line": 45, "end_line": 45, "tags": [ "possible-gold(root-cause-language)", "possible-question(impact-language)" ] }, { "text": "-", "start_line": 46, "end_line": 46, "tags": [] }, { "text": "How could time to diagnosis be improved?", "start_line": 47, "end_line": 47, "tags": [] }, { "text": "- We had to fall back to Google Docs to manage this incident since GitLab.com was unavailable, however, a GitLab issue was eventually created once GitLab.com was back online #15997 (closed). Having both the doc and issue caused back and forth copying and pasting of data which was inefficient while trying to manage the incident. One way to improve this is to use our ops instance for incident tracking rather than Google docs. Though, this would lead to additional problems and is currently discarded. Alternative solution are being discussed and a follow-up issue has been created to continue exploring them: Improve Incident Management process when gitlab... (gitlab-com/www-gitlab-com#34382 - moved)", "start_line": 48, "end_line": 48, "tags": [] }, { "text": "-", "start_line": 49, "end_line": 49, "tags": [] }, { "text": "How did we reach the point where we knew how to mitigate the impact?", "start_line": 50, "end_line": 50, "tags": [ "possible-question(impact-language)" ] }, { "text": "- Spent some time trying to \"fix\" terraform, or get a better handle on how a restore might work without having to slowly apply each difference one at a time.", "start_line": 51, "end_line": 51, "tags": [] }, { "text": "- While that was happening, work was put into trying to assess what specific disks and other systems were missing.", "start_line": 52, "end_line": 52, "tags": [] }, { "text": "- An attempt to remove the dependency on the redis cache cluster was put in progress to see if that would get the web fleet back to operating status.", "start_line": 53, "end_line": 53, "tags": [] }, { "text": "- The call was broken into two zoom chats. The main incident room was used to focus on restoring services without using Terraform. The other was focused on restoring resources via Terraform.", "start_line": 54, "end_line": 54, "tags": [] }, { "text": "- Getting the resources restored and the Terraform configuration into a clean state along with ensuring affected customers were notified was what we considered mitigated.", "start_line": 55, "end_line": 55, "tags": [ "possible-question(impact-language)" ] }, { "text": "-", "start_line": 56, "end_line": 56, "tags": [] }, { "text": "How could time to mitigation be improved?", "start_line": 57, "end_line": 57, "tags": [] }, { "text": "- Identifying a list of affected customers was delayed due to tooling (rails console and Teleport) being offline. This tooling was dependent on first getting our Terraform configuration into a clean state.", "start_line": 58, "end_line": 58, "tags": [ "possible-gold(root-cause-language)", "possible-question(impact-language)" ] }, { "text": "- Drafting messaging for affected customers required quite a lot of cross-functional efforts and approvals (i.e. engineering, product, and support to assess impact and work with corporate communications to draft message, marketing ops to send the message, legal / customer success sign off). This can be improved by having pre-approved messaging templates for these types of outages allowing us to move quicker. Follow-up issue: https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/34387+", "start_line": 59, "end_line": 59, "tags": [ "possible-question(impact-language)" ] }, { "text": "- Did we have other events in the past with the same root cause?", "start_line": 60, "end_line": 60, "tags": [ "possible-gold(root-cause-language)" ] }, { "text": "- Do we have existing backlog items that would've prevented or greatly reduced the impact of this incident?", "start_line": 61, "end_line": 61, "tags": [ "possible-question(impact-language)" ] }, { "text": "- ...", "start_line": 62, "end_line": 62, "tags": [] }, { "text": "- Was this incident triggered by a change (deployment of code or change to infrastructure)? If yes, link the issue.", "start_line": 63, "end_line": 63, "tags": [] }, { "text": "- A lot of people came together to help, even though it was a Friday, and Saturday!", "start_line": 64, "end_line": 64, "tags": [] }, { "text": "- Our response processes were put to test, with an immediate good feedback from everyone involved, for e.g. the breakdown of zoom calls, threads, and documents to tackle different parts of the incident was excellent.", "start_line": 65, "end_line": 65, "tags": [] } ], "operator_disposition": "keep" }, { "doc_id": "07_aws_s3_disruption_2017-02-28.md", "family": "cause-from-symptom", "extraction": "manual-sub-split", "selection_source": "operator-selected", "selection_note": "no confident suggestion -- recommend drop or operator manual sub-line splitting", "reason": "no clean R2/R3 section heading found in this document (plain prose or a bulleted Q&A template, not markdown-headed sections)", "question_text": "While these subsystems were being restarted, S3 was unable to service requests. Other AWS services in the US-EAST-1 Region that rely on S3 for storage, including the S3 console, Amazon Elastic Compute Cloud (EC2) new instance launches, Amazon Elastic Block Store (EBS) volumes (when data was needed from a S3 snapshot), and AWS Lambda were also impacted while the S3 APIs were unavailable.", "question_location": "line 4, impact clause (sub-split, CC-constructed)", "gold_text": "At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.", "gold_location": "line 4, root-cause clause (sub-split, CC-constructed)", "jaccard": 0.0167, "r4_status": "included", "gold_engram_ids": null, "gold_position": null, "candidates": [ { "text": "Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region", "start_line": 3, "end_line": 3, "tags": [] }, { "text": "We\u2019d like to give you some additional information about the service disruption that occurred in the Northern Virginia (US-EAST-1) Region on the morning of February 28th, 2017. The Amazon Simple Storage Service (S3) team was debugging an issue causing the S3 billing system to progress more slowly than expected. At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. The servers that were inadvertently removed supported two other S3 subsystems. One of these subsystems, the index subsystem, manages the metadata and location information of all S3 objects in the region. This subsystem is necessary to serve all GET, LIST, PUT, and DELETE requests. The second subsystem, the placement subsystem, manages allocation of new storage and requires the index subsystem to be functioning properly to correctly operate. The placement subsystem is used during PUT requests to allocate storage for new objects. Removing a significant portion of the capacity caused each of these systems to require a full restart. While these subsystems were being restarted, S3 was unable to service requests. Other AWS services in the US-EAST-1 Region that rely on S3 for storage, including the S3 console, Amazon Elastic Compute Cloud (EC2) new instance launches, Amazon Elastic Block Store (EBS) volumes (when data was needed from a S3 snapshot), and AWS Lambda were also impacted while the S3 APIs were unavailable.", "start_line": 4, "end_line": 4, "tags": [ "possible-question(impact-language)" ] }, { "text": "S3 subsystems are designed to support the removal or failure of significant capacity with little or no customer impact. We build our systems with the assumption that things will occasionally fail, and we rely on the ability to remove and replace capacity as one of our core operational processes. While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. S3 has experienced massive growth over the last several years and the process of restarting these services and running the necessary safety checks to validate the integrity of the metadata took longer than expected. The index subsystem was the first of the two affected subsystems that needed to be restarted. By 12:26PM PST, the index subsystem had activated enough capacity to begin servicing S3 GET, LIST, and DELETE requests. By 1:18PM PST, the index subsystem was fully recovered and GET, LIST, and DELETE APIs were functioning normally. The S3 PUT API also required the placement subsystem. The placement subsystem began recovery when the index subsystem was functional and finished recovery at 1:54PM PST. At this point, S3 was operating normally. Other AWS services that were impacted by this event began recovering. Some of these services had accumulated a backlog of work during the S3 disruption and required additional time to fully recover.", "start_line": 5, "end_line": 5, "tags": [ "possible-question(impact-language)" ] }, { "text": "We are making several changes as a result of this operational event. While removal of capacity is a key operational practice, in this instance, the tool used allowed too much capacity to be removed too quickly. We have modified this tool to remove capacity more slowly and added safeguards to prevent capacity from being removed when it will take any subsystem below its minimum required capacity level. This will prevent an incorrect input from triggering a similar event in the future. We are also auditing our other operational tools to ensure we have similar safety checks. We will also make changes to improve the recovery time of key S3 subsystems. We employ multiple techniques to allow our services to recover from any failure quickly. One of the most important involves breaking services into small partitions which we call cells. By factoring services into cells, engineering teams can assess and thoroughly test recovery processes of even the largest service or subsystem. As S3 has scaled, the team has done considerable work to refactor parts of the service into smaller cells to reduce blast radius and improve recovery. During this event, the recovery time of the index subsystem still took longer than we expected. The S3 team had planned further partitioning of the index subsystem later this year. We are reprioritizing that work to begin immediately.", "start_line": 6, "end_line": 6, "tags": [] }, { "text": "From the beginning of this event until 11:37AM PST, we were unable to update the individual services\u2019 status on the AWS Service Health Dashboard (SHD) because of a dependency the SHD administration console has on Amazon S3. Instead, we used the AWS Twitter feed (@AWSCloud) and SHD banner text to communicate status until we were able to update the individual services\u2019 status on the SHD. We understand that the SHD provides important visibility to our customers during operational events and we have changed the SHD administration console to run across multiple AWS regions.", "start_line": 7, "end_line": 7, "tags": [ "possible-question(impact-language)" ] }, { "text": "Finally, we want to apologize for the impact this event caused for our customers. While we are proud of our long track record of availability with Amazon S3, we know how critical this service is to our customers, their applications and end users, and their businesses. We will do everything we can to learn from this event and use it to improve our availability even further.", "start_line": 8, "end_line": 8, "tags": [ "possible-question(impact-language)" ] } ], "operator_disposition": "add, both texts CC-constructed by sub-splitting L4 into clauses (APPROVED as-is)" }, { "doc_id": "13_twilio_billing_postmortem_2013.md", "family": "cause-from-symptom", "extraction": "needs-manual-selection", "selection_source": "operator-selected", "selection_note": null, "reason": "no clean R2/R3 section heading found in this document (plain prose or a bulleted Q&A template, not markdown-headed sections)", "question_text": "Twilio experienced an incident with its billing system on July 18, 2013.", "question_location": "line 6, first sentence only (operator: edit to first sentence only)", "gold_text": "At 1:35 AM PDT on July 18, a loss of network connectivity caused all billing redis-slaves to simultaneously disconnect from the master. This caused all redis-slaves to reconnect and request full synchronization with the master at the same time. Receiving full sync requests from each redis-slave caused the master to suffer extreme load, resulting in performance degradation of the master and timeouts from redis-slaves to redis-master.", "gold_location": "line 28 (the Redis network-connectivity-loss paragraph)", "jaccard": 0.0513, "r4_status": "included", "gold_engram_ids": null, "gold_position": "after", "candidates": [ { "text": "Billing Incident Post-Mortem: Breakdown, Analysis and Root Cause", "start_line": 3, "end_line": 3, "tags": [ "possible-gold(root-cause-language)" ] }, { "text": "Time to read:", "start_line": 4, "end_line": 4, "tags": [] }, { "text": "Twilio experienced an incident with its billing system on July 18, 2013. Although we\u2019ve shared how the incident unfolded, and the impact on our customers, we\u2019d like to detail the root cause, how we fixed it, and what we\u2019re doing to ensure this doesn\u2019t happen in the future.", "start_line": 5, "end_line": 5, "tags": [ "possible-gold(root-cause-language)", "possible-question(impact-language)" ] }, { "text": "Twilio experienced an incident with its billing system on July 18, 2013. This incident affected 1.4% of Twilio\u2019s customers in up to three ways:", "start_line": 6, "end_line": 6, "tags": [ "possible-question(impact-language)" ] }, { "text": "- If a customer made a credit card payment to Twilio, during the time of the incident, the account balance was not updated to reflect the payment.", "start_line": 7, "end_line": 7, "tags": [] }, { "text": "- If the credit card payment was triggered by an auto-recharge, the recharge was attempted multiple times as a result of the account balance not being updated to reflect the payment.", "start_line": 8, "end_line": 8, "tags": [] }, { "text": "- Some accounts were suspended as a result of a recharge attempted against a credit card deactivated by the repeated billing.", "start_line": 9, "end_line": 9, "tags": [] }, { "text": "Additionally, Twilio usage reports were delayed in reflecting the prices of billable items for all customers while the billing system was offline. Voice and SMS message services were not impacted during this incident, however prices were not calculated for calls and message made during the duration of the incident.", "start_line": 10, "end_line": 10, "tags": [ "possible-question(impact-language)" ] }, { "text": "July 18, 2013", "start_line": 11, "end_line": 11, "tags": [] }, { "text": "- 1:35 AM PDT / 8:35 UTC: We experienced a loss of network connectivity between all of our billing redis-slaves and our redis-master. This caused all redis-slaves to reconnect and request full synchronization with the master at the same time.", "start_line": 12, "end_line": 12, "tags": [ "possible-question(impact-language)" ] }, { "text": "- 2:39 AM PDT / 9:39 UTC: Services relying on the redis-master began to fail due to the load generated by the slave synchronization.", "start_line": 13, "end_line": 13, "tags": [ "possible-gold(root-cause-language)" ] }, { "text": "- 2:42 AM PDT / 9:42 UTC: Our on-call engineers restarted the redis-master to address the high load.", "start_line": 14, "end_line": 14, "tags": [] }, { "text": "- 3:28 AM PDT / 10:28 UTC: Our monitoring systems detected an anomaly in our billing systems, which resulted in erroneous credit card charges and in some cases account suspensions. 1.1% of all Twilio customers were impacted. The on call team immediately began our incident response.", "start_line": 15, "end_line": 15, "tags": [ "possible-question(impact-language)" ] }, { "text": "- 4:10 AM PDT / 11:10 UTC: Our billing system was taken offline to avoid additional credit card charges.", "start_line": 16, "end_line": 16, "tags": [] }, { "text": "- 6:24 AM PDT / 13:24 UTC: Service restored to all suspended accounts. The billing system remained offline, while our engineers investigated the root cause.", "start_line": 17, "end_line": 17, "tags": [ "possible-gold(root-cause-language)" ] }, { "text": "- 11:58 AM PDT / 18:58 UTC: Billing system was brought back online.", "start_line": 18, "end_line": 18, "tags": [] }, { "text": "- 12:36 PM PDT / 19:36 UTC: Monitoring detected a recurrence of the original billing anomalies, affecting another 0.3% of all Twilio customers. The billing system was taken offline again. Suspended accounts were immediately reactivated.", "start_line": 19, "end_line": 19, "tags": [ "possible-question(impact-language)" ] }, { "text": "- 2:57 PM PDT / 21:57 UTC: We began processing refunds for the erroneous credit card charges. This work continued for the following 24 hours.", "start_line": 20, "end_line": 20, "tags": [] }, { "text": "July 19, 2013", "start_line": 21, "end_line": 21, "tags": [] }, { "text": "- 3:00 PM PDT / 22:00 UTC: We finished processing refunds for all erroneous credit card charges.", "start_line": 22, "end_line": 22, "tags": [] }, { "text": "- 3:30 PM PDT / 22:30 UTC: All impacted accounts were given a credit equaling 10% of their last 30 days of Twilio spend.", "start_line": 23, "end_line": 23, "tags": [ "possible-question(impact-language)" ] }, { "text": "- 7:10 PM PDT / July 20 02:30 UTC: The billing system was activated progressively for groups of customer accounts.", "start_line": 24, "end_line": 24, "tags": [] }, { "text": "- 8:14 PM PDT / July 20 03:14 UTC: The billing system was activated for all customers.", "start_line": 25, "end_line": 25, "tags": [ "possible-question(impact-language)" ] }, { "text": "- 9:15 PM PDT / July 20 04:15 UTC: We gave the all-clear message, and all systems were restored.", "start_line": 26, "end_line": 26, "tags": [] }, { "text": "Twilio\u2019s billing system uses an in-memory Redis cluster to store in-flight account balances. This cluster is configured with a single master and multiple slaves distributed across data-centers for resiliency in the event of a host or data-center failure.", "start_line": 27, "end_line": 27, "tags": [] }, { "text": "At 1:35 AM PDT on July 18, a loss of network connectivity caused all billing redis-slaves to simultaneously disconnect from the master. This caused all redis-slaves to reconnect and request full synchronization with the master at the same time. Receiving full sync requests from each redis-slave caused the master to suffer extreme load, resulting in performance degradation of the master and timeouts from redis-slaves to redis-master.", "start_line": 28, "end_line": 28, "tags": [] }, { "text": "By 2:39 AM PDT the host\u2019s load became so extreme, services relying on redis-master began to fail. At 2:42 AM PDT, our monitoring system alerted our on-call engineering team of a failure in the Redis cluster. Observing extreme load on the host, the redis process on redis-master was misdiagnosed as requiring a restart to recover. This caused redis-master to read an incorrect configuration file, which in turn caused Redis to attempt to recover from a non-existent AOF file, instead of the binary snapshot. As a result of that failed recovery, redis-master dropped all balance data. In addition to forcing recovery from a non-existent AOF, an incorrect configuration also caused redis-master to boot as a slave of itself, putting it in read-only mode and preventing the billing system from updating account balances.", "start_line": 29, "end_line": 29, "tags": [] }, { "text": "With all account balances at zero and read-only, Twilio usage that resulted in a billing transaction (e.g. 1 cent for a SMS message or a phone call) triggered the billing system to attempt a recharge using the credit card associated with the customer\u2019s account. This only affected accounts with auto-recharge enabled.", "start_line": 30, "end_line": 30, "tags": [] }, { "text": "Consequently, the billing system charged customer credit cards to increase account balances without being able to update the balances themselves. This root cause produced the billing incident of customer credit cards being charged repeatedly. At 3:28 AM PDT, the billing system monitoring reported the anomalous activity. At 4:10 AM PDT, on-call engineers responding to the incident shut down the billing system to prevent further charges.", "start_line": 31, "end_line": 31, "tags": [ "possible-gold(root-cause-language)" ] }, { "text": "The billing system maintains independent double-bookkeeping for all balance data in a separate relational datastore. Following the shutdown of the billing system, this independent record was used to restore the account balances lost in the failed recovery of the redis-master. Once these balances were properly restored, the billing system was turned on at 11:58 AM PDT. Observing further anomalous behavior, the billing system was shut back down and engineering work focused on restoring service and refunding customers accounts.", "start_line": 32, "end_line": 32, "tags": [ "possible-question(impact-language)" ] }, { "text": "That work completed the following day and the billing service was restored slowly across the customer base, reaching full restoration at 9:15 AM PDT.", "start_line": 33, "end_line": 33, "tags": [] }, { "text": "In the process of resolving the incident, we replaced the original redis cluster that triggered the incident. The incorrect configuration for redis-master was identified and corrected. As a further preventative measure, Redis restarts on redis-master are disabled and future redis-master recoveries will be accomplished by pivoting a slave.", "start_line": 34, "end_line": 34, "tags": [] }, { "text": "The simultaneous loss of in-flight balance data and the ability to update balances also exposed a critical flaw in our auto-recharge system. It failed dangerously, exposing customer accounts to incorrect charges and suspensions. We are now introducing robust fail-safes, so that if billing balances don\u2019t exist or cannot be written, the system will not suspend accounts or charge credit cards. Finally, we will be updating the billing system to validate against our double-bookkeeping databases in real-time.", "start_line": 35, "end_line": 35, "tags": [] }, { "text": "All of us here at Twilio apologize for the impact this had on you, your business and your customers. We look forward to these steps to earn back your trust, and all the steps that follow.", "start_line": 36, "end_line": 36, "tags": [ "possible-question(impact-language)" ] } ], "operator_disposition": "FINAL: gold scoped to root-cause chunks (23,24), excluding the timeline-section duplicate (chunk 7) -- operator-approved. Question edited to first-sentence trim -- operator-approved. Discovered the trimmed question ALSO duplicates verbatim (chunk 0, title/lede block, vs chunk 2, real body-opening sentence) -- scoped to chunk 2 only, applying the same substantive-occurrence-over-incidental-duplicate principle just approved for gold; disclosed, not silently applied. Pair is now fully resolved and closed.", "gold_engram_ids_scoped": [ "0a031d7d-7063-4c47-ba6d-0b9e4a5594ef", "d47de9e7-f6c2-454b-bc86-8f40351b1609" ], "gold_engram_ids_mechanical_raw": [ "abe26fb9-885a-43b6-9c81-369bfdcf2eea", "0a031d7d-7063-4c47-ba6d-0b9e4a5594ef", "d47de9e7-f6c2-454b-bc86-8f40351b1609" ], "question_engram_ids_scoped": [ "084ff483-7d81-4a2f-b299-824986c15e5b" ], "question_engram_ids_mechanical_raw": [ "b0b6d6a7-ad17-423e-9572-b74ce8045ac7", "084ff483-7d81-4a2f-b299-824986c15e5b" ], "resolution_status": "resolved (scoped)" } ] }