Python API test automation framework (Part 6) API response schema validation

Logos in header image sources: Python, Requests, JSON, HTTP, Cerberus

This is the sixth post in a series on how to build an API framework using python.

You can read previous parts below:

Most APIs return JSON that adhere to some contract set between its clients (could be another API or a web app etc). While you can write consumer-driven contract tests, sometimes you might want to just test with the live API and see if its response schema conforms to a fixed structure.

There are many different data validation libraries in the Python ecosystem and you can choose one that meets your objectives. We will use Cerberus which is a quite popular library for this purpose. Other notable libraries are jsonschema, voluptuous etc

Setup

Install cerberus within your virtualenv

pipenv install cerberus

Schema test for Read operation in People API

Let’s say we want to check that our people API’s response conforms to a schema that we expect:

Below is the structure we get when we hit the read API

{
"fname": "Doug",
"lname": "Farrell",
"person_id": 1,
"timestamp": "2020-12-01T16:50:36.842997"
}

Cerberus works by defining a schema with all the fields inside the response object and their types and then validates if a sample response indeed met the schema need.

Your first schema test

Below is a test to validate if Read one operation of people API meets a defined schema

import json

import requests
from cerberus import Validator

from config import BASE_URI

schema = {
"fname": {'type': 'string'},
"lname": {'type': 'string'},
"person_id": {'type': 'integer'},
"timestamp": {'type': 'string'}
}


def test_read_one_operation_has_expected_schema():
response = requests.get(f'{BASE_URI}/1')
person = json.loads(response.text)

validator = Validator(schema)
is_valid = validator.validate(person)

assert_that(is_valid, description=validator.errors).is_true()

Let’s understand how this is constructed.

We start with defining the expected schema of the response object, Since it is a single object with certain keys like fname, lname etc, and values.

We can define a python dict with these schema details

"fname": {'type': 'string'}

Here for every field in the response, we specify key with the field name and value is another dict specifying the type like string, number, boolean, date etc.

See the full list of types on cerberus docs

Sweet, Below is how the schema looks like for the read response

schema = {
"fname": {'type': 'string'},
"lname": {'type': 'string'},
"person_id": {'type': 'integer'},
"timestamp": {'type': 'string'}
}

We then hit the GET API with an expected person id and then convert the response to a python dict using loads()

response = requests.get(f'{BASE_URI}/1')
person = json.loads(response.text)

Note: Hard coding a user id like 1 (in the request URL) is often something to be avoided. You might want to create a new user and then do this test, however since this API gets seeded with some dummy data, we are following this approach for demo purposes only.

We then initialize an instance of Validator class with this schema. Optionally, if we want to specify that all the keys are required in this schema then we can add require_all=True keyword argument. Or we could even specify this at a field level using'required': True/False in the schema itself.

validator = Validator(schema, require_all=True)

We can assess if the JSON matches this schema with below:

is_valid = validator.validate(person)

And if we want to raise an assertion error if Cerberus finds a mismatch then we can print validator.errors

assert_that(is_valid, description=validator.errors).is_true()

When we run the test for our current people API, we see the test passes.

To see how it would look like in case of a failure we can change the type of person_id from number to string and that would raise the error message below, notice we get to know the field that is mismatched and what the validation failure is.

AssertionError: [{'person_id': ['must be of string type']}] Expected <True>, but was not.

Test for read all operation

How does this test look like for the Read all operation?

def test_read_all_operation_has_expected_schema():
response = requests.get(BASE_URI)
persons = json.loads(response.text)

validator = Validator(schema, require_all=True)

with soft_assertions():
for person in persons:
is_valid = validator.validate(person)

Essentially, We get the list of persons and then repeat the same validation for all the records in this list while wrapping it with a soft assertion to ensure all the validation failures are collected and printed in the end.

Conclusion

Schema validation is an important component to include in your API automation framework and I hope you have a basic understanding of how to use a tool like Cerberus to achieve this. For understanding, all the nuances of this approach feel free to dig deep into Cerberus docs which lists a lot of the functionality that is available.

You can find the complete code for this course on Github at automationhacks/course-api-framework-python

If you found this post useful, Do share it with a friend or colleague and if you have thoughts, I’d be more than happy to chat over at twitter or comments. Until next time. Happy Testing.

Further reads:

Originally published at https://automationhacks.io on December 28, 2020.

Software Engineer at Meta, I ❤️ to build scalable test automation frameworks and teams. Blog at automationhacks.io 🇮🇳 pronouns: he/him/his

Software Engineer at Meta, I ❤️ to build scalable test automation frameworks and teams. Blog at automationhacks.io 🇮🇳 pronouns: he/him/his