Introducing KaiZen OpenAPI 3.0 Parser: fast, flexible Java parsing & validation

Posted by Andy Lowry on Apr 12, 2017 04:22 PM
KaiZen OpenAPI Parser Logo (medium).png

Are you excited about OpenAPI 3.0?

Here's why it matters: The Swagger 2.0 specification was released 2Β½ years ago, emerging as the de facto industry standard for API definition and documentation. SmartBear made it official a year later, in November 2015 with the formation of the OpenAPI Initiative under the Linux Foundation, with Google, IBM, Microsoft and others as founding members.

The next generation of API modeling is now here. With the OpenAPI Spec v3 Draft release, developers can describe a broader range of modern REST APIs in greater detail, including hyperlinks, webhooks, better JSON Schema support, rich examples, and more.

At RepreZen, we're excited and working hard to bring the full range of OpenAPI 3.0's expanded capabilities into the RepreZen API Studio design experience. KaiZen OpenAPI Parser, now available on GitHub, is the first cornerstone of our open source stack for OpenAPI 3, based on the draft spec. It's built for speed, and highly configurable. If you're building OpenAPI 3 tooling or solutions in Java, this is a great place to start.

Design Goals

Two Java modules supplied by the OpenAPI project for Swagger 2.0, swagger-models (part of the swagger-core project), and swagger-parser, are core dependencies throughout Swagger's open source stack for Java, and instrumental in RepreZen API Studio's current Swagger support. The Swagger parser consumes an OpenAPI spec and constructs a representation of the spec using the Swagger model classes, with methods to access all of the details.

With OpenAPI 3.0, these and other software components will need to be upgraded or replaced. We've used the Java parser and model projects extensively, so this is a great opportunity to create a simpler, more efficient, and more functional replacement.

Some of the design goals included:

  • High Performance - The parser needs to be fast and efficient, for frequent parsing of large OpenAPI specs in our editing environment, without noticeable lag. Informal testing shows a 3x-4x performance improvement over the current Java Swagger 2.0 parser.
  • Tolerant Reader - Applying tolerant reader principles to the parser, we wanted to ensure that the parser handles gracefully any input that doesn't fully conform to the OpenAPI language spec.
  • DI-Ready - We wanted an interface-based design that allows substituting a custom implementation for the one we provide, using dependency injection or other mechanisms.
  • Conformance to Spec - Swagger-Models deviates from the OpenAPI 2.0 spec in subtle ways that required awkward pre- and post-processing. We want to ensure that the new core libraries faithfully represent OpenAPI's structure and validation rules.

We're currently in discussion with the OpenAPI community about making KaiZen OpenAPI Parser a core component of a reference implementation. Whether or not that happens, we'll continue to develop our parser as an open-source offering, and we welcome contributions from interested developers.

Java Interfaces

We provide a complete set of Java interfaces to encapsulate the OpenAPI object model, in addition to classes that implement those interfaces. This provides maximum flexibility to developers for scenarios like testing with mock instances, and opens the door to the creation of plug-compatible alternative implementations, substituted via dependency injection.

KOP-Type-Hierarchy.png

JSON Overlays

Our parser takes a novel approach to consuming its JSON input, by creating API classes that function as "Adapters" of Jackson's JsonNode object. We call these adapters JSON Overlays, and we've created a small, separate kernel framework of overlay classes for primitive types, collections, maps, and objects. The Overlay framework is the basis for our implementation of all the object types defined in the OpenAPI specification, from the top-level OpenAPI object to Paths, Responses, and all the rest.

A number of things are made much easier with this framework than with the more typical approach of deserializing JSON into target classes (often POJOs).

  • Separation of Parsing and Validation: An overlay object can bind to any JsonNode value, even if the node is incompatible with the overlay. An invalid node is treated as if it were missing when the overlay object is asked for its value. But the JsonNode is retained, so that a separate validation process can report on the error. This is in contrast to the typical approach of achieving robust validation by writing complicated Jackson custom deserializers, forcing this level of validation to occur during the parsing operation.
  • Bidirectional Navigation: Our overlay framework's base class keeps track of two bits of information that make upward navigation through the resulting structure very simple: each overlay object's parent overlay, and the key by which that parent knows this object. This makes it very easy, for example, to find the Path that an Operation belongs to, or the name under which a Schema is listed in the model's "components" section.
  • Speed: Informal testing suggests that our parser consumes OpenAPI models at roughly 3-4 times the rate of the existing Java Swagger 2.0 parser.

Generated Code

There is very little "logic" in the OpenAPI 3.0 specification, aside from validation-related requirements scattered throughout the document (e.g. each name in a security requirement MUST match the name of a security scheme defined in the components object).

Most what the KaiZen OpenAPI Parser does is driven by the prescribed structure of an OpenAPI 3.0 spec. The parser has to provide Java interfaces to represent that structure, validate that an OpenAPI instance conforms to the structure, and manage serialization and deserialization according to the structure.

For this reason we have chosen to generate the interfaces and implementation classes for all our OpenAPI object classes. We use a simple YAML DSL to define all the types and their properties- for example, an Info object has a title property of type String, a contact property of type Contact, etc.

Type-DSL-Info.png

In the few places where non-validation logic appears in the specification, some of these classes need to be augmented with hand-coded methods. For example, the getServices() method of the Operation object should return the list of services defined locally for that operation, if that list exists. Otherwise, it should fall back to the services defined for the enclosing path, or in the top-level model object.

So our generator preserves hand-crafted code by parsing an existing Java source file before overwriting it, and copying any members that are not marked with the @Generated annotation.

Reference Resolution

References are most conveniently resolved in the JSON form of an OpenAPI model, and that's where we do it. All references are resolved before the JSON structure is bound to overlay objects. References are left in place in the JSON structure, but are augmented internally with detailed information. The Java getter methods always provide, by default, resolved values for objects and their properties. However, the fact that a reference underlies a particular value is always available through additional methods, as are full details about the reference itself and the results of its resolution - or the causes of its resolution failure.

Validation

The first potential source of validation errors occurs in the parsing of the source JSON or YAML document into a JSON structure. We use the Jackson library for this, but without imposing any constraint other than that it should be valid JSON or YAML. If the input does not meet this base requirement, the parse will fail with a generic "malformed input" exception. (The Jackson exception can be examined for details.) Otherwise, parsing will yield an OpenAPI3 object that can be inspected to determine whether it is, in fact, a valid model.

All other validation is performed by our OpenAPI parser, in a separate step that follows the JSON or YAML parse. By default, the OpenAPI parser will perform validation on-the-fly, but this is optional.

Validation results - in the form of info, warning, and error items - are available via a method on the parsed Swagger object.  There are also methods to perform validation and to query the overall validity of the model, separately.  Here's a partial example from the Getting Started Guide:

    private static void processModel(URI modelUri, boolean validate) {
        OpenApi3 model = (OpenApi3) new OpenApiParser().parse(modelUri, validate);
        System.out.printf("== Model %s\n", modelUri);
        if (!validate || model.isValid()) {
            describeModel(model);
        } else {
            for (ValidationItem item : model.getValidationItems()) {
                System.out.println(item);
            }
        }
        System.out.printf("------\n\n");
    }

    private static void describeModel(OpenApi3 model) {
        System.out.printf("Title: %s\n", model.getInfo().getTitle());
        for (Path path : model.getPaths().values()) {
            System.out.printf("Path %s:\n", path.getKey());
            for (Operation op : path.getOperations().values()) {
                System.out.printf("  %s: [%s] %s\n", op.getKey().toUpperCase(), op.getOperationId(), op.getSummary());
                for (Parameter param : op.getParameters()) {
                    System.out.printf("    %s[%s]:, %s - %s\n", param.getName(), param.getIn(), getParameterType(param),
                            param.getDescription());
                }
            }
        }
    }

Validation comprises all requirements appearing in the OpenAPI specification, including all uses of the words MUST, SHALL, REQUIRED, and SHOULD (the latter giving rise to warnings), as well as requirements implied by property types (e.g. a "description" property whose value is something other than a JSON text node would result in an error item).

In addition, any unexpected JSON nodes appearing in the spec can be detected by the validator. The parser can be configured to treat these as errors or warnings, or to ignore them completely. In all cases, these will be properties of JSON object nodes that are not accounted for in the OpenAPI specification, since by the structure of the specification, any other unexpected JSON node will give rise to a wrong-type-node error. Unexpected property nodes may appear in object types (e.g. an address property in the JSON for a Contact object), or in maps with key restrictions (e.g. a property in the paths map that does not match either the /.+ regular expression for paths, or the x-.+ regular expression for extensions).

Read/Write Model

Most applications will probably involve parsing OpenAPI models from a JSON or YAML source and then working with the model programmatically. However, our Java interfaces also provide write access to every property of every model type, via a highly uniform write API.

KaiZen OpenAPI Parser, when complete, will supply a convenient builder for each type.

Serialization

Our serializer will output JSON or YAML that will, by default, reliably render properties within an object, or values within a map or list, in the order in which they were created. This is true whether the model was created by the parser, or from scratch using builders, or any combination of the two. Round-tripping through the parser will never reorder anything unless you specifically make use of ordering options.

Swagger 2.0, Too!

Using the same code generation tools we have developed for this parser, we plan to create a parser for Swagger-OpenAPI 2.0 as well. It will feature all the same capabilities, and the same uniform interfaces. And as long as your JSON or YAML input has the correct swagger or openapi property, we'll be able to detect the version and invoke the appropriate parser for you automatically.

Interested?

If you'd like more information, you can subscribe here for future blog posts that will dive into greater detail on some of the topics we've touched on.

If you'd like to get involved more deeply, check out our GitHub repository. We welcome serious contributors.  And by all means drop us a line at api.community@reprezen.com if you have questions, or just to let us know you're interested.

 

Topics: Swagger, OpenAPI

Welcome!

At RepreZen, we're building our business on two things:  thought leadership in API design, and great conversations.

This blog is one of many places where we'll have illuminating, mutually enriching conversations with our customers, partners, and the software community at large.  Please chime in with your thoughts and let's get started!

RepreZen API Studio - Free Trial!


Installing RepreZen API Studio is easy!  Here's how it works:

number_circle_1   Submit the trial registration form.
number_circle_2   Check your inbox for download instructions.
number_circle_3   Download and run the automated installer.

 

System Requirements

 

RepreZen API Studio runs on the following operating systems:

  • 64-bit Windows 7, 8.0 and 8.1
  • Mac OS X 10.7 or higher
RepreZen API Studio requires JDK 1.7 or higher.