Why diagram the OAS ecosystem?

As part of the efforts to design OAS 3.2 and 4.0 “Moonwalk”, I wanted to figure out how different sorts of tools work with the OAS. Moonwalk is an opportunity to re-think everything, and I want that re-thinking to make it easier to design, implement and maintain tools. We also want 3.2 to be an incremental step towards Moonwalk, so figuring out what improvements we can make to align with the needs of tools while maintaining strict compatibility with OAS 3.1 is also important.

To do that, we need to understand how the OAS-based tooling ecosystem works, and how the various tools relate to the current OAS version, 3.1. This eventually led me to create two groups of diagrams: One about the architecture of OAS-based tools, and one about the many objects and fields defined by the OAS and how they relate to what tools need to do. But this was not a simple process!

OAS tools do… stuff?

I was surprised at how difficult it was to find patterns in tool design and OAS usage. I thought I could go look at how tools were categorized on sites that list them, figure out what each category needs, and see how that aligns with the current and proposed future OAS versions. I was wrong. Looking the OpenAPI Initiative’s own tools site reveals categories that often overlap and tools that don’t map cleanly to categories.

One problem is that many “tools” aren’t so much OAS implementations as they are applications that include some OAS-based functionality. So maybe “tool” or “application” isn’t the right granularity for this research. Really, there are different tasks that can be done with an OpenAPI Description (OAD), and each OAS-based tool, application, or library does at least one such task.

What if one tool did it all?

I started thinking about all of the tasks, and how they relate to each other. The goal is not to really create a monolithic mega-tool architecture. Not all tasks happen on the same system, or at the same time in the development process. But it would help to understand which tasks directly relate to each other, which don’t, and what the interfaces among them ought to be.

An idealized architecture

After several iterations with valuable feedback from the regular OAS working group calls, here is what I came up with (based on OAS 3.1 plus ideas for 3.2 and Moonwalk):

The first thing to notice is the that this “idealized architecture” breaks down into three main purposes or functional areas:

• Parsing libraries: The foundation on which all other tools are built

• OAD tools: These work with OpenAPI Descriptions and their constituent documents

• API tools: These work with the API that the OAD describes but (mostly) aren’t concerned with the mechanics of OAD documents and references

Color coding and accessibility

Before we look at the three purposes, let’s talk about how I’m using color in these diagrams:

The colors here, named “pale grey”, “light yellow”, “pink”, “light cyan”, “pear”, “olive”, “light blue”, “orange”, and “mint”, are a set designed by Paul Tol to work well with dense grids of information labeled with black text, regardless of color vision ability.

I’ll always use these names (minus the “pale” and “light” qualifiers) to talk about the colors in these diagrams. Every diagram will include the color names so that no one has to guess which color is which, and all diagrams use the same colors to mean the same general things:

• Gray: Documents and metadata

• Yellow: References and implicit connections

• Pink: Parsing

• Cyan: Interface modeling

• Pear: Runtime data validation

• Olive: Extended validation

• Blue: Data encoding

• Orange: HTTP modeling

• Mint: API documentation

As we compare diagrams about tools with diagrams about the spec (which you’ll see in future posts), we will hope that consistent use of color will show clear alignment between these two perspectives. Wherever they do not, we probably have some work to do.

Parsing libraries

The most fundamental functional area, “Parsing Libraries”, consists of a stack of wide blocks stretching across the bottom. From the color code, you can see that these blocks deal with the low-level document interface (grey), resolving references and implicit connections (yellow), and parsing (pink). “Implicit connections” is a term we’re introducing in OAS 3.0.4 and 3.1.1 that encompasses things like mapping Security Requirement Object keys to Security Scheme Object names under the securitySchemes field of the Components Object.

Every tool, application, or library that works with OADs or the APIs that they describe has to parse OADs and do at least some resolution of internal connections. While some references can be preprocessed out, others cannot, nor can any of the implicit connections. We’ll explore these challenges, as well as the interactions between the grey, yellow, and pink blocks, more deeply in future posts.

The API Description API (ADA)

For now, the most important thing is that the upper pink block, which I’m tentatively calling the “API Description API (ADA)”, is intended to provide a clean interface to the parsed and resolved OAD that insulates tools from the complexities of document managing and referencing. This ADA would be somewhat akin to the DOM. There are, however, fundamental differences between OADs and web pages, so the analogy is not exact.

We’ve decided that we want to have an ADA in Moonwalk. It’s an open question as to whether it would be worthwhile to start defining it in 3.x. Would 3.x tools (or at least parsing libraries) would start supporting it? If you have thoughts on that concern, or on whether “ADA” is a good name, we’d like to hear from you!

OAD tools

The first change I would make to sites that classify OpenAPI tools would be to separate the tools that work with OADs from the tools the work with APIs.

These are fundamentally different use cases and mindsets, and they have different requirements when it comes to interacting with OADs. The blocks in this area are colored pink because they are primarily concerned with the structure of the OAD. The description semantics inform what can or should be done with the OAD, but it is the OAD rather than the described API that is the focus of these tools.

The left column includes OAD editors (including IDE plugins), validators, linters, and generators (producing an OAD from code). The right column includes tools connecting the OAD with other specifications, including the OpenAPI Initiative’s Arazzo and (forthcoming) Overlays specifications. The right column also contains translators from other formats such as RAML, or from formats that offer an alternative syntax that “compiles” down to an OAD.

Semantic operations, such as a linting rule that applies to every Response Object, no matter its location, benefit from semantic access via the ADA (pink downward arrow). Structural operations, such as an overlay that inserts a field into each object matching a JSON Path applied to a specific document (or anything that requires writing to documents), benefit from structural access via the underlying document interface (grey downward arrow).

API Tools

Now that we’ve accounted for shared parsing tasks and OAD-specific tools, we’re ready to examine how the OAS supports working with an API. We’ll start on the left edge where our hypothetical end user is looking at an interactive documentation site that includes a test interface to send requests and display the resulting responses.

While our discussion will follow the progress of a request on the client side, the steps are reversed to parse, decode, and validate the request on the server, and then run again on the server to validate, encode, and assemble a response, which is parsed, decoded, validated, and displayed to the end user on the client side.

API Documentation (mint)

Documentation is produce by mint-colored blocks that use the ADA to walk through the whole OAD, organize the information for human consumption, and render it to the end user in some format. Different formats could be handled as plugins in an extensible architecture, as shown above the main mint-colored block.

The API documentation block is the only one that looks at every part of the OAD that is visible through the ADA. Separating docs processing into its own block allows other blocks to ignore the fields that only impact documentation. This might influence how the ADA is designed.

If the documentation is interactive, then it would incorporate an interface generated by the next block that can take the appropriate session or operation input and render the results. The mint blocks are concerned with rendering that interface, but not with defining it.

Interface Modeling and Validation (cyan, pear, and olive)

Following the arrow to the right, we reach the API client code. The boundary between the three differently-colored types of blocks within this larger area is somewhat ambiguous, and varies depending on the programming language(s) and types of tools being used.

Interface definition (cyan)

Cyan represents the static code-level (as opposed to HTTP-level) interface to the API. The diagram shows how there might be several form factors for this interface stacked in the left column above the main cyan block: Code libraries on the client and server, or a web or CLI interface for interactive documentation or manual testing. Such an interface might be very generic, or it might be custom-generated with elaborate static type-checking.

Runtime data validation (pear)

The pear-colored module is for runtime data validation, verifying constraints that are not inherent in the interface’s type-checking but only depend on the data passed to that interface. It's mostly synonymous with JSON Schema validation, although not entirely as we'll see when we color code individual fields in the next post.

Extended validation (olive)

The olive-colored module is for validation such as readOnly that requires additional context about how the data is being used, which often builds on the runtime validation by using annotations.

A confusing boundary

As noted, the boundary between these three colors is ambiguous. JSON Schema is not a data definition system, and was not designed with a clear division between static data types (cyan) and runtime constraints (pear) in mind. Languages support differing type-checking paradigms, tools have variable levels of support for classes or other complex structures, and "code generation" can be taken to the extreme of generating a custom function for each JSON Schema.

Additionally, format values are sometimes handled as static types (cyan), sometimes as runtime validation (pear), and sometimes as annotation-based validation (olive).

All of this overlap and ambiguity is a major challenge for tooling vendors, an one we’ll examine in detail in a future post.

Data Encoding and Decoding (blue)

The code interface is defined in terms of JSON-compatible data structures. Encoding such a structure as an application/json request body is trivial. But validated data also needs to be mapped into other media types, and into various bits and pieces governed by other rules such as ABNF grammars.

In this diagram, I'm proposing a modular, extensible system to handle these encodings or translations. API innovation moves faster than specification development, and the OAS doesn’t currently have a good way to accommodate emerging data formats like JSON streams. We also haven’t ever addressed things like HTTP headers with complex structure.

Defining an extensible interface, where mappings to various targets could be defined in a registry and supported with a plug-in architecture, would dramatically increase the flexibility of the OAS tooling ecosystem, as well as its responsiveness to innovation. It could also help us get the substantial complexity of the data encoding fields, which we’ll start to explore in the next post, under control.

HTTP Modeling (orange)

Finally, we reach the orange block that is takes the bits and pieces of encoded data and assembles them into HTTP messages (or parses them back out). We see on our diagram how data passes through this module as a request to be constructed, then back through on the server side where the request is parsed, handled, and returned as a response.

This area breaks down into three sub-areas: Server configuration (which provides the URL prefix for HTTP calls), security configuration (which might involve HTTP interactions with an auth server), and the actual API operations.

Much of this HTTP parsing and serialization can be handled by standard libraries, which can also help define us the exact border with the data encoding blocks. Our goal is to organize the OAS aspect of things, not to re-invent the wheel of HTTP and related technologies.

That’s a pretty picture, but now what?

The next step is to analyze OAS 3.1 and the likely 3.2 proposals in detail and correlate the Objects and fields with this idealized architecture. That starts to show where we have mismatches between the spec and tools, and what we might want to do about them, which will be the subject of the next post.

However, one aspect of this diagram is already driving real work: This past week’s Moonwalk working group call featured a TSC member demo-ing a small ADA proof-of-concept. The idealized architecture may be theoretical, but its impacts on the future of the OAS are already very real.

What can 3.0 users do now to make the update to 3.1 easier, whether it is in the near or distant future? This is the first of a series of posts addressing that question, in which you will learn:

• the OAS road map and your possible update paths

• schema re-use patterns that illustrate the how and why of future-proofing for 3.1

• techniques to use in 3.0 to make it easier to quickly benefit from updating

This post provides a high-level overview. The next few posts will get into more concrete details, including a guide to actually updating your API descriptions to 3.1.

The OAS Road Map

Note: While I am very involved in the OpenAPI project, I do not speak for it! I am confident in the direction, but the details could change.

The key point of the proposed OpenAPI Specification road map are:

• All 3.x releases from 3.2 on will be 100% compatible with 3.1

• The 3.2+ releases will be small and incremental

• OAS 4.0 "Moonwalk" features will be backported to 3.2+ when possible

OAS 3.1 is much bigger than 3.0, and slightly incompatible with it. While there were good reasons for this, the newly proposed road map prioritizes painless updates from 3.1, rapid implementation of 3.2+, and an incremental path to 4.0.

Three re-use challenges

These challenges come from my experience with consulting clients who are working with complex data models, or are standardizing approaches across multiple APIs. If your API has a simple, self-contained data model, you may not have seen the difficulties, or may have easily worked around them. Even then, understanding these challenges will give you more design options to support your API's growth.

Our challenges are to:

• re-use object schemas while forbidding unknown properties

• re-use collection schemas with different contents

• recognize a standard component by a location-independent identifier

Let's look at these in a bit more detail before talking about the solutions in OAS 3.1 and beyond.

Object re-use and unknown properties

This challenge is most likely to show up if you are trying to align your schemas with the behavior of strictly-typed object-oriented languages. Particularly if you have a complex class hierarchy. The problem is that

1. re-using an object schema involves using $ref with allOf, oneOf, etc.

2. failing validation when an unexpected property is seen requires additionalProperties: false

3. these two techniques cannot be used together^[1]

components:
  schemas:
    extensibleBase:
      type: object
      properties:
        foo: {type: integer}
    sealedBase:
      # This schema fails if foo is present, because this
      # additionalProperties can't "see" the foo definition
      allOf:
      - $ref: "#/components/schemas/extensibleBase"
      additionalProperties: false

Re-using collection schemas

In strictly-typed languages, structures like collections correspond to generics (Java, C#) or templates (C++). These programming idioms have concise syntax, but in OAS 3.0's version of JSON Schema, you need to duplicate some of the generic structure for each wrapped type:

components:
  schemas:
    collection:
      type: object
      properties:
        metadata:
          $ref: "#/components/schemas/metadata"
        data:
          type: array
          items: {}   # allow anything
    thing:
      type: string
    thingCollection:
      allOf:
      - $ref: "#/components/schemas/collection"
      # Here we have to duplicate the path to the items
      properties:
        data:
          items:
            $ref: "#/components/schemas/thing"
    metadata:
      # Assume some object schema here for pagination, etc.

The three extra keywords and nesting levels here aren't too burdensome. But with a more complex container and many wrapped types, it becomes fragile to maintain.^[2]

Identifying standard components

Some APIs need interoperability with industry standards.^[3] For example, there are libraries for handling data that conforms to the GeoJSON standard (RFC 7946), which has standard JSON Schemas such as https://geojson.org/schema/Feature.json for a GeoJSON "feature." If you see "$ref": "https://geojson.org/schema/Feature.json" in any API, you know that you can use your GeoJSON library on that data.

OAS 3.0 describes reference URLs as locators, implying that tools should retrieve the target using the URL. However, not all tools support references to external documents. Even if they do, security policies, unreliable networks, or performance constraints can make them impractical or impossible to use. If you have to copy that GeoJSON schema somewhere else, code looking for the standard URL won't recognize the URL of the new location.

When challenges combine...

A workaround for the URL problem might be to compare the schemas instead of the references. However, experience shows that copies are often modified, for example to work around the other challenges. You might want a "strict" Feature schema that forbids unknown properties, and specializations of the Feature Collection schema with different extended or strict features. Even the most dedicated DRY programmers will resort to copy-paste(-modify) if proper re-use is too hard.

Challenges on the Road Map

So lets look at how our challenges are handled in each current or proposed release:

For re-using objects while forbidding unexpected properties, there's a solid, well-supported solution in OAS 3.1.

In theory, collections are also solved in OAS 3.1. In practice, the solution depends on keywords that are often not yet implemented, even by tools claiming 3.1 support. There was a similar lag in support for discriminator with past OAS versions.

Component recognition in OAS 3.1 only works for Schema Objects, and has similar keyword support challenges. Fortunately, this is likely to be thoroughly addressed in OAS 4.0, in a way that should be possible to backport to 3.x.

What should you do in 3.0?

This post introduced the proposed road map and three re-use challenges. The next few posts will get into the concrete details of each challenge. But the basic pattern is the same for any use of 3.0 that will benefit from new features in 3.1

1. Identify where you are working around limitations in 3.0, and why

2. Make sure your workarounds for each limitation are consistent and recognizable (and document them as best practices within your team)

3. If your API description is large, create an x- prefixed extension keyword for each workaround so that you (or an automated tool) can find them in the future

For these three challenges, you won't need to change anything just to move to a newer OAS 3.x version. But taking advantage of the new features will make your API descriptions more robust and maintainable.

We'll take a concrete look at how that works for the object re-use case in the next post in this series.

1. For several years, the most common complaint about JSON Schema was this additionalProperties: false problem. The debate on how to solve it ran for more than 500 GitHub comments before we settled on unevaluatedProperties!

2. I first saw the need for this sort of feature in Cloudflare's pre-OpenAPI documentation system, doca, which extended JSON Hyper-Schema with a limited templating keyword for their standard response envelope. This directly inspired JSON Schema draft 2019-09's $recursiveRef and $recursiveAnchor, which evolved into 2020-12's $dynamicRef and $dynamicAnchor. Cloudflare's current OAS 3.0.3 description of course lacks that capability, so you will see lots of "*-api-response-common" components that repeat the same envelope with different contents. Once they move to 3.1, they should be able to shorten their description file considerably!

3. Special thanks to my client, the Open Geospatial Consortium, for motivating and supporting this post, and particularly for highlighting the component identification use case. As a result, I'll be using geospatial data-related examples in this and related posts.

The OpenAPI Initiative

2024 is shaping up to be a very exciting year for OpenAPI, and I'm very much involved in the work to define exactly what the year will look like.

OpenAPI Release Landscape

The OpenAPI world is excited about the plan to publish Moonwalk (OAS 4.0) in 2024, but that's only half of the picture. I am (along with others) advocating for putting a lot of work into the 3.x line, including 3.0.y and 3.1.y patch releases, and a series of small, incremental 3.x releases starting with 3.2.

OpenAPI has developed a reputation for releasing giant changes every three to four years (3.1 qualifies despite being sincerely intended as "minor"). These then take another three to four years to be broadly implemented and adopted. It's produced a bit of understandable grumbling mixed in with the Moonwalk excitement among folks who are just getting to 3.1, or are still on 3.0.

We're hoping to provide a smoother path of increased functionality this time, and we're just getting started reviving and updating our processes for easier participation while we produce multiple releases. I'm excited to figure out what pace of change will work best for the OpenAPI community.

OASComply

Much of my focus last year was on an OpenAPI Description (OAD) parser intended to accomplish several things:

• Illustrate how the OpenAPI Initiative interprets the OpenAPI Specification's (OAS) parsing requirements

• Define a format for the parsed representation that can be used to test parsing compliance

• Serve as the basis for further work, such as linting, a formal compliance test suite, or possibly automated upgrading from 3.x to Moonwalk

Part of the way through the year we discovered ambiguities in the referencing requirements, so coding work is on hold while we resolve that (which will show up in the release planning somehow). But we expect to deliver a working tool this year.

JSON Schema Implementation

The OASComply work includes parsing both standard (but extended) JSON Schema (for OAS 3.1) and the "superset-subset" custom JSON Schema variation found in OAS 3.0. As I worked with a JSON Schema implementation to extend its capabilities, the owner eventually recommended that I make an authorized fork as the work needed to fully support OAS + JSON Schema is pretty extensive. I'll post more about this when I'm ready to release the first public version of the fork.

It's looking like a busy 2024!

As you can see, I'm doing a lot of both specification and coding work this year. One downside of specification work is you're often focused on things that won't reach your average end-user for 5+ years. I am hopeful we'll settle on an OpenAPI release approach that gets new features into the hands of end-users very quickly. I'm looking forward to writing more about that process as well as my tool development work in the very near future!

In this article, you'll learn:

• What it means for JSON Schema to be a constraint system

• How to think about designing constraints effectively

• Why "constraint" encompasses more than just JSON Schema assertions

Constraints vs Definitions

This is an empty schema (in all versions¹ of JSON Schema):

{}

It allows everything.

This is an empty class in Java:

public class Empty {
}

It does nothing.

Building Java classes and JSON Schemas for a data model work from opposite starting points and require different approaches:

Class definition systems are additive: You start from nothing and the more you specify, the more you can do. On the left, our empty Java class is a black disc with no other colors. You add fields to define the correct set of possible data.

Constraint systems are subtractive: You start with everything and the more you specify, the less you can do. On the right, our empty JSON Schema is a black disc covered by a chaotic riot of color blobs. You add keywords to exclude all incorrect data. This thought process runs in the opposite direction compared to defining a class in Java.

Both processes converge in the middle, where our correct data model is shown as three colors at the points of an equilateral triangle on a black disc. So let's look at how these different processes work.

Be forbidding!

If we were thinking about Java classes and JSON Schemas in parallel, our next step might be adding a property to each (using a single public boolean data attribute for simplicity — colors and shapes just made a better visual!)

public class Empty {
    public boolean isOn;
}

{
    "properties": {
        "isOn": {
            "type": "boolean"
        }
    }
}

While these look analogous, they are very different. Constraints in JSON Schema constrain as little as possible. This maximizes flexibility for schema authors. All this schema says is "if the instance is an object, and if it has a property named isOn, then that property must be a boolean."

In contrast, Java classes are inherently objects with named fields. Java classes only have the fields you declare. They always have those fields, even if you don't use them. The fields are initialized automatically, which for booleans means being set to false.

That's four additional constraints that we need to add to our JSON Schema (using JavaScript comment syntax):

{
   "type": "object",               // forbid non-objects
   "required": ["isOn"],           // forbid objects without "isOn"
   "properties": {
       "isOn": {
           "type": "boolean",      // forbid non-boolean "isOn"
           "default": false        // forbid assuming "isOn" is true
       }     
   },    
   "additionalProperties": false   // forbid props that aren't "isOn"
}

This might seem strange. Why would you want to consider a non-object valid if you are describing object properties?

When you're writing a Java class, you are starting with Java's inherent concept of a class, with all of its automatic behaviors (such as attribute initialization) and assumptions (classes always having exactly the set of attributes that were declared, no more, no less). Java knows that you only want the things you say you want.

On the other hand, JSON Schema has no idea what you're trying to do and can't make any assumptions for you. It doesn't know that you're trying to match an OOP language. Even if it did, you could be writing in Perl, where a reference to anything, even a number or string, can be "blessed" as a class instance and properties can be added or removed at any time.

This is why JSON Schema is a constraint system, and why you want to understand that instead of trying to use as if it were something else.² It has to work with anything that might be done in JSON, and it relies on you to ask yourself, "have I forbidden everything I don't want?"

Constraints vs validation assertions

JSON Schema has different keyword behaviors including assertions, which can cause validation to fail, and annotations, which can't fail validation but do provide additional information about valid instances.

All of the JSON Schema keywords in the example above are assertions except "default", which is an annotation. But if you're using this schema for code generation, "default" is a constraint in that context. A "default" of true would require a generated Java class to initialize isOn to true. Non-assertions can be constraints in non-validation use cases!

Conclusion

In this article, you've learned that:

• JSON Schema is a constraint system because it can't know how you will use it

• Java and similar languages know how you will use them, and make assumptions accordingly

• The empty JSON Schema allows everything

• The schema design mindset is: "Have I forbidden everything I don't want?"

• Whether a keyword functions as a constraint depends on how you are using the schema

Please let me know what you think in the comments, and please click the "Follow" button so you don't miss the next two posts: First, an explanation of dynamic scopes, which you'll want to understand so you get the most out of the following post: When to use "additionalProperties" vs "unevaluatedProperties"!

1. In real-world usage, you should always specify "$schema", as different implementations handle its absence differently. Some assume a default version (which might not be the one you expected), and others won't process such schemas at all because they don't want to risk giving an incorrect validation outcome with an inaccurate default assumption. To keep examples more readable, I'll often omit "$schema" when it doesn't change how the example is understood. ↩

2. There have been some misguided attempts to reconcile OOP and constraints. For example, recent versions of the popular AJV implementation of JSON Schema set a bunch of "strict mode" options by default. Many of these attempt to make JSON Schema behave more like a data definition system, but the resulting behavior is not compliant with the JSON Schema specification. In a future article, we'll look at when it is useful to use AJV's strict mode, and when it is problematic. While AJV's strict mode prevents certain valid schemas from being used rather than changing their outcome, this breaks interoperability as many things it prevents have valid use cases. Such schemas written for use with compliant implementations will fail with AJV, which is why the JSON Schema implementations page now notes implementations with noncompliant default behavior. ↩

I'm excited to show you how "modern" JSON Schema, meaning 2019-09, 2020-12, and later, can solve far more problems than "classical" JSON Schema (draft-07 and earlier). We'll also explore topics relevant to APIs in general and OpenAPI 3.1 in particular, and to all versions of JSON Schema. If you're using classical JSON Schema, earlier versions of OpenAPI, or other approaches to HTTP APIs, you'll find plenty to interest you here as well.

In this introductory post, you'll learn:

• Why I'm making a distinction between "modern JSON Schema" and "classical JSON Schema"
• Three areas of new functionality that modern JSON Schema has introduced

Why "modern" vs "classical" JSON Schema?

The JSON Schema landscape is pretty confusing right now. There are five versions in active use: draft-04, draft-06, draft-07, and drafts¹ 2019-09 and 2020-12. When you get into the complexities of IETF draft names vs meta-schema identifiers, it all becomes excessively complicated:

Talking about all of these versions is confusing. Talking about eras, only two of which are in use, is much simpler!

Simplifying the conversation is important, but there's a bit more to the name. "Modern JSON Schema" is inspired by the successful branding of C++11 and later as "modern C++". C++11 made a long list of changes to the core language, just as JSON Schema 2019-09 significantly expanded JSON Schema's capabilities. "Modern C++" is used in discussions and job descriptions to help people understand it as a distinct and desirable skill compared to C++03 and earlier.

I want to see a similar shift happen with JSON Schema. We can build excitement around "modern JSON Schema" much more easily than around a pile of draft names and numbers. We can focus on what unifies recent versions and sets them apart from the past. We can create a healthy ecosystem where modern JSON Schema is in demand, but only if enough people understand why they should demand it and support it.

My hope for this blog is that it will further the community's understanding and generate that demand.

Shiny modern things!

So let's explore what makes modern JSON Schema, "modern"! These three changes aren't the only changes between classical and modern JSON Schema, but they each dramatically expand what JSON Schema can do.

Runtime keyword communication

This change is about making schemas more re-usable. How many of you tried to write a schema like this at some point?

{
  "$schema": "https://json-schema.org/draft-07/schema",
  "type": "object",
  "allOf": [{"$ref": "https://example.com/schemas/sharedproperties"}],
  "additionalProperties": false
}

I did! But any instance that has any properties will fail validation against this schema. The "additionalProperties" keyword can't see into the https://example.com/schemas/sharedproperties schema, so it doesn't matter what it contains.

Modern JSON Schema allows keywords to record their runtime behavior for other keywords to read. This allows keywords to build on the results of other keywords in ways that classical JSON Schema does not allow, such as by taking into account which properties that were validated in a referenced schema.

In future posts, we'll use a series of examples to look at how this works, and what advantages and drawbacks it brings. You'll learn how modern JSON Schema makes it easy to solve several important use cases that used to be difficult or impossible.

Annotations for application-aware behavior

Even though JSON Schema has standard "readOnly" and "writeOnly" annotation keywords, it is challenging to use them to validate API request and response bodies if you're using classical JSON Schema.

JSON Schema validates the structure of the JSON data. In the process of doing that it learns which fields are annotated with information such as "readOnly": true. But it doesn't understand whether it's running on a client or a server, nor does it know the request and response semantics of different HTTP methods. OpenAPI HTTP tooling understands these things, but it doesn't know which fields in the structure are read-only. If these two systems could collaborate, they could validate the usage of the JSON data in the context of HTTP by extending standard JSON Schema tools.

Classical JSON Schema never defined what information made up an annotation, or how to report that information to applications such as OpenAPI tools. Modern JSON Schema defines five pieces of information that make up an annotation, along with machine-readable formats for reporting it. This allows OpenAPI tools to collaborate with standard JSON Schema implementations:

Here we see JSON Schema validating a PUT request payload on the server, and reporting one read-only and one write-only field as annotations. For the purpose of this example, read-only fields can be present as long as they don't change the field's value. The OpenAPI tool checks that the read-only field is unchanged, and knows that the write-only field is allowed because a PUT request is a write.

This shows how modern JSON Schema annotations enable standardized collaboration among tools that was not previously possible! We'll explore the five parts of an annotation along with many scenarios where applications can leverage JSON Schema annotations to perform a wide variety of tasks.

Vocabularies: You (yes, you!) can extend JSON Schema

Vocabularies allow anyone to define JSON Schema keywords that can be used reliably with any compliant implementations. Classical JSON Schema allows extension keywords, but you can't guarantee that they'll be respected. Unrecognized extension keywords are silently ignored, which results in invalid data being declared valid.

Modern JSON Schema introduces vocabularies, which allow you to define a group of keywords and identify them with a URI. Schema authors can then use that URI to tell implementations that the need to support the vocabulary in order to use the schema. If they can't, instead of failing validation, the implementation refuses to run the schema and indicates which vocabularies it doesn't understand.

This is just like finding out that you need to install a 3rd-party library to run a piece of software. Hopefully it will one day be easy to find vocabulary plugins when you need them. But even if you can't find support for a vocabulary, it's much better for an implementation to refuse to use the schema than it is to incorrectly claim that the data is valid! Getting the right error is the first step to fixing it.

We'll spend a whole series of posts understanding why you should care about vocabularies and what you need to know about them. We'll walk through examples of how to design and implement them, paying special attention to vocabularies that can be created and used without having to write custom code. This is an area that is still evolving, and I hope this blog will start conversations that help us finalize vocabularies in a way that makes them accessible to everyone who wants create keywords.

Conclusion

In this article, you've learned that:

• "Modern JSON Schema" is a way to simplify how we talk about JSON Schema and promote the expanded capabilities of its recent versions
• Runtime keyword communication makes modern JSON Schema more re-usable
• Modern JSON Schema's annotations and output formats allow applications to build on JSON Schema in a standardized way
• Vocabularies allow anyone to create new keywords and ensure that they will be used correctly rather than being silently ignored

Thank you for reading this introductory post! Future posts will go deeper into these and other topics, including detailed examples. My goal is to keep posts short (less than 1000 words) and frequent, with the occasional longer deep dive.

In the next post we'll examine how the mental model for working with JSON Schema differs from working with object-oriented programming. Understanding this is fundamental to really getting comfortable with how JSON Schema works. Please click the "+Follow" button at the top of the page as you will not want to miss it!

I am very excited to be writing this blog, and have several articles near completion that you can expect to see within the next week. There is so much to say about JSON Schema and APIs, and I would love to hear what topics would interest you! Please let me know in the comments! 👇

1. The "draft" label is by now mostly due to historical inertia, and will be dropped soon in favor of a release model with better compatibility guarantees. For modern JSON Schema I generally won't bother with the "draft" label. 2020-12 in particular is the base from which our new push towards stability will start. ↩