Many successful SaaS businesses opened a JSON API to the public for their customers to integrate. Many times, the only protection against introducing a breaking change is automated tests. But as we‘ll see in this article, not all automated tests are protecting equally.

If you don't have thorough tests for your public API, it's probably for one or more of these reasons that we will or will not handle in this article:

• Your public API is just a byproduct of your primary SaaS offering, and you don't consider public APIs special enough to deserve more thorough testing than the usual backend development. After all, in the backend, the party using the backend API of a method or class will have their own tests anyway (see the next section, why it's problematic to ruin your customers' days).
• You want to move quickly or feel like you don't have enough time to do thorough testing. Thorough API testing requires an established testing culture in your team, adding factories/fixtures, and setting up the tests with more care. But most painfully, you’d need to actually THINK (🙀) about business requirements to come up with better test names and assertions (this article will rather go into how to test, which should save you some time in the long term; however, for the business value side of things and tooling setup, you will need continuing to be a responsible engineer and we won't fix it here).
• Many API tools and web development frameworks generate code and tests suggesting that API tests are as simple as checking for the response to be successful or that the data kinda matches (we'll see an example of generation and build upon it).
• The nature of subtle breaking changes is not apparent to you because you've never yet broken API clients’ integrations and ruined everyone’s day at least a couple of times (if you did have the experience, you probably already do thorough testing; if not, you can start here).

Ruining Your API Clients’ Day(s)

Not having thorough tests means increasing the probability that the response payload will change over time without anyone else on your team noticing it. This is very problematic because clients who integrate with your API will start to rely on the payloads that they received when initially developing the integration.

Let's consider an extreme example and imagine that your API responded with this payload for an endpoint like GET yourapp.com/api/contacts:

{
    "id": 1,
    "email_address": "hey+testing@richsteinmetz.com",
    "first_name": "Rich",
    "last_name": "Stone",
    "nick": "RichStone",
    "age": 142,
    "created_at": "2024-01-27T14:48:04.617Z",
    "updated_at": "2024-01-27T14:48:04.617Z"
}

But on the next deployment, someone decided to change things up slightly. Now your response looks a bit different to make things more developer- and user-friendly:

{
    "id": 1,
    "email": "hey+testing@richsteinmetz.com",
    "name": "Rich Stone",
    "created_at": "Jan. 27th 2024",
    "updated_at": "Jan. 27th 2024"
}

If they used this data to display and modify contacts in their own web app, then after your little update, their whole app or parts of it will be down because fields disappeared, got renamed, and the dates respond with a different format that needs to be parsed differently by the client.

When you offered your API version to the public, and someone started integrating with it, you both agreed on a contract, i.e., the available fields, the exact field names, and the format of their corresponding values. It’s on you to maintain the backward compatibility of any changes that you make to this version of the API.

So, it’s essential to test the exact responses you offer based on various incoming requests in integration tests and/or serializer tests so as not to break the contract and ruin your clients’ days.

Web API Testing Fallacies

In the next sections, you'll look into common testing practices by building an example from scratch. At the end, you'll also have a peek into actual production code and see how the big folks are doing it.

Because it’s so quick, you’ll use Ruby and Rails for this, the language and framework of the web and APIs. In a couple of commands, you have a fully working API with a fully functioning resource.

With this setup, you can now ask the API for contacts:

$ curl localhost:3000/contacts
[]

And run the pre-generated tests:

$ rails test
Running 5 tests in a single process (parallelization threshold is 50)
Run options: --seed 63461

# Running:

.....

Finished in 0.285483s, 17.5142 runs/s, 31.5255 assertions/s.
5 runs, 9 assertions, 0 failures, 0 errors, 0 skips

There are 5 controller tests that are auto-generated and are supposed to test the whole request/response pipeline, e.g. this one:

# contacts_controller_test.rb
test "should show contact" do
  get contact_url(@contact), as: :json
  assert_response :success
end

Automated testing is a wide field, so it’s out of scope for this article to get into test naming definition and implication wars around Controller, Integration, Contract testing, and Co. We’ll keep it simple and run Rails/minitest style “controller tests” that spin up the API, make actual requests to it, and ensure the contract that you are offering to your API consumers is correct until the end of the article.

You hopefully agree that this is not a production-ready test to back up the contract between you and your API clients. The only thing it tests is that nothing is seriously broken, and the endpoint returns a success code in the range between 200 and 299. Any breaking changes to the application code will go into your production code without any notice.

From here on, there are two main ways developers will often add tests “quickly”.

Object-to-Object testing fallacy

The quickest way to test all your offered output payload is to test the object in the database against the JSON output:

# contacts_controller_test.rb
test "should show contact" do
  get contact_url(@contact), as: :json

  assert_response :success
  assert_equal @contact.as_json, JSON.parse(response.body)
end

Now, you’d also be testing that the parsed JSON response object (Ruby's Hash, a.k.a. HashMap in other languages) equals the contact record's Hash that'd be passed to the serializer. Maybe you gained a marginal something, but these tests will fail once you add additional metadata to the response that is not saved in the database. For example, if this was your definition of the output that you enable for your clients:

# app/views/_contact.json.jbuilder
json.extract! contact, 
  :id, 
  :email_address, 
  :first_name, 
  :last_name, 
  :age, 
  :created_at, 
  :updated_at

Then, adding a calculated non-DB field would break your tests:

# app/views/_contact.json.jbuilder
json.extract! contact, 
  :id, 
  :email_address, 
  :first_name, 
  :last_name, 
  :age, 
  :created_at, 
  :updated_at
json.url contact_url(contact, format: :json) # This is calculated dynamically.

So, this would only kinda work without additional effort if you assume that your database is and always will be the source of truth for the API output.

But even then, if you have these kinds of tests with an API being used by others, you are all set up to ruin someone else’s day because these tests allow for breaking changes. For example, renaming or removing a column like contacts.first_name from your database will still happily let the tests pass because the object in the database still equals the API response.

Also, note that this test is just one big anti-pattern of a test. It doesn’t contribute to communicating, documenting, or enforcing requirements.

Variable attributes-based testing fallacy

An approach that is more common in API development is “variable attributes-based testing”:

# contacts_controller_test.rb
test "should show contact" do
    get contact_url(@contact), as: :json
    parsed_response = JSON.parse(response.body)

    assert_response :success
    assert_equal @contact.id, parsed_response["id"]
    assert_equal @contact.email_address, parsed_response["email_address"]
    assert_equal @contact.first_name, parsed_response["first_name"]
    assert_equal @contact.last_name, parsed_response["last_name"]
    assert_equal @contact.nick, parsed_response["nick"]
    assert_equal @contact.age, parsed_response["age"]
  end

This is an improvement because if the codebase changes to remove or rename one of the attributes, the test will fail. This is good because now our API clients’ days won’t be ruined as easily.

These tests are also less brittle because adding new attributes won’t fail them. But this is actually a bug, not a feature. If someone forgets to add a test for an attribute that’s introduced by your API, it could be changed later and, again, ruin your API clients’ days ⛈️

Did you notice something missing in the new test? Developers migrating from "response successful" testing or object-to-object testing to the variable attributes-based testing approach will often avoid the pain of dealing with dynamic fields or fields that are changed through the serializer. In this case, updated_at and created_at were left out because they don’t work out of the box:

# contacts_controller_test.rb
test "should show contact" do
  get contact_url(@contact), as: :json

  # ... more code ...

  assert_equal @contact.updated_at, parsed_response["updated_at"]
  assert_equal @contact.created_at, parsed_response["created_at"]
end

This will give you an error that seems not to be worth fiddling with for many devs:

$ rails test
Minitest::Assertion: --- expected
+++ actual
@@ -1 +1 @@
-Sat, 27 Jan 2024 15:51:05.981315000 UTC +00:00
+"2024-01-27T15:51:05.981Z"

test/controllers/contacts_controller_test.rb:32:in `block in <class:ContactsControllerTest>'

The problem is that @contact.updated_at returns an ActiveSupport::TimeWithZone, which was serialized by the serializer here deeper down the call stack:

# contacts_controller.rb
def show
  render json: @contact
end

The more dedicated developer will solve this issue by serializing the left side

# contacts_controller_test.rb
assert_equal @contact.updated_at.as_json, parsed_response["updated_at"]
assert_equal @contact.created_at.as_json, parsed_response["created_at"]

This situation is particularly interesting for our trusted API tests discussion because it brings us to an interesting point:

Looking at these tests, you won't know what values from the DB are tested against what other values from the response. This removes the extra documentation feature that tests can provide and makes them harder to understand at a glance. But more importantly, they do not secure the contract in more detail other than ensuring that the database still has the right fields on the model. In other words, you avoid being precise about the requirements to get your tests out quickly.

Imagine that our codebase changes in a way that it stops populating the updated_at field so it’s now always coming out as null or someone finds a "better" JSON serializer that suddenly serializes the datetime field as RFC 2822 Date Time Format ("Tue, 19 Mar 2023 16:30:45 +0000") instead of the previous "2024-01-27T15:51:05.981Z" ISO 8601 format. Now, your clients’ day is ruined because they can’t parse the data anymore properly. This could have been prevented if your collaborator was notified by a failing test that we actually promised this exact format of the field to our API clients.

Changing the expected value format or type in web APIs is a breaking change.

A better approach is the "invariable attribute-based testing" where you test for the actual values:

# contacts_controller_test.rb

  # ... this requires a more thorough setup; see the next article section...
  assert_equal @contact.id, 0
  assert_equal @contact.email_address, "hey@richsteinmetz.com"
  assert_equal @contact.first_name, "Rich"
  assert_equal @contact.last_name, "Stone"
  assert_equal @contact.nick, "Gepard"
  assert_equal @contact.age, 142
  assert_equal @contact.updated_at, "2024-01-27T15:51:05.981Z"
  assert_equal @contact.created_at, "2024-01-27T15:51:05.981Z"

This is already way better, but it still does not test the response as a whole. You or your teammates might miss adding a new field that was added to the response payload here in your tests, which would make your tests go out of sync with the contract yet again.

Web API Tests You Can Trust

So, how can we ensure that all the changes happening in our codebase won't mess with the contract that we offered to the API client?

Enter “Full Response Testing”.

First, extra data about the HTTP request, like particular headers and the status code, are part of the contract, so let’s at least test that we always return the right status code:

# contacts_controller_test.rb
test "should show contact" do
  get contact_url(@contact), as: :json
  parsed_response = JSON.parse(response.body)

  assert_response 200
  # ...
end

The status code is a vital part of each request, so it’s fine to include it when testing the contract, but other headers for security and further formalities and the absence of some headers should be tested in their own tests.

The test name could also deserve some love, so you see immediately which part of the ContactsController is being tested here and to what extent:

# contacts_controller_test.rb
test "#show responds with the serialized contact" do
  # ...

Next, testing the exact contract means specifying your expected requirements in an exact way. This means actually writing out the outputs that you expect.

test "#show responds with the serialized contact" do
  # Creating the contact in the test directly helps specifying requirements,
  # improving the tests-as-docs feature, and readability:
  #
  # Even better if you use factories or similar patterns to create this contact.
  contact = Contact.create(
    id: 0,
    email_address: "hey@richsteinmetz.com",
    first_name: "Rich",
    last_name: "Steinmetz",
    nick: "Gepard",
    age: 142,
    created_at: DateTime.parse("2024-01-27T15:51:05.981Z"),
    updated_at: DateTime.parse("2024-01-27T15:51:05.981Z")
  )
  get contact_url(contact), as: :json

  assert_response 200
  expected = {
    id: 0,
    email_address: "hey@richsteinmetz.com",
    first_name: "Rich",
    last_name: "Steinmetz",
    nick: "Gepard",
    age: 142,
    created_at: "2024-01-27T15:51:05.981Z",
    updated_at: "2024-01-27T15:51:05.981Z"
  }
  assert_equal expected.to_json, response.body
end

First thing that some developers dislike about this approach is the verbosity and duplication. It’s a trade off worth considering, though.

Your first instinct might be that expected is almost the same as the attributes passed to Contact.create. Though, you wouldn’t be able to reuse the attributes passed to the contact creation in the expected ones because DateTime.parse(…) in conjunction with .to_json will return something slightly different than the actual JSON response. But honestly, that’s not a big deal because in any reasonably complex API, you will write both by hand anyway, because there will be cool stuff like dynamically calculated attributes and dynamic attribute values (that are stored differently in the database and change right before serialization).

Also, hard-coded values make it crystal clear what's the original input vs what's expected.

Tests are a place, where DRYness can lead to more pain than good in many instances introducing cognitive overload when holding many variables in your head and scrolling back and forth through your test files trying to map those to the test context.

These wet tests of yours, are now enforcing application requirements and human/developer-readable documentation.

There Is More to Test

That’s a great baseline now! We are specifying the exact keys and values that are offered in the current version of the contract. If anything in the codebase modifies these expectations, we’ll be notified.

Keep in mind that this level of thorough testing is essential to have for the general happy path, but doesn't need to be a part of your tests. You may fallback to attribute-based testing, e.g. when testing corner-cases and looking at the dynamic behavior of one specific field or field group.

If you are reading this from the real world, your Contact model is probably a tiny bit more complex with a bunch of associations and many more fields. I would still advice you to employ "full response testing" for the most part. However, your API resource might include other API resources in full, like so:

{
    "id": 1,
    "email_address": "hey+testing@richsteinmetz.com",
    "first_name": "Rich",
    "purpose": {
      "id": 5,
      "title: "Riddling over the API Puzzle"
      // ... 100 more fields to describe the purpose...
    },
    "created_at": "2024-01-27T14:48:04.617Z",
    "updated_at": "2024-01-27T14:48:04.617Z"
}

The other API resources can become quite big and complex themselves, too. Apart from the fact that you might have a design problem, duplicating this test effort becomes unreasonably big. It also creates a test duplication that you might not want where failing tests do warn you about the issue but you might need to fix them in several places. In this case, it is better to fall back to a variable attribute-based assertions for this field only:

Further, depending on how your API is set up, you will now be adding different test scenarios to ensure that:

• dynamic attributes are serialized correctly in different cases
• proper authentication/authorization behavior is ensured
• different inputs are handled correctly
• different error cases are tested including their response formats, error messages and status codes

Trade-offs

In complex applications, these tests are more difficult to write because it's harder to get the test setup right and you'd need to create a bunch of objects by hand to test for all those values. If you find a codebase like this that already uses a strategy tactic that's not "full response testing", then it will be an unachievable task to migrate to all tests in one go. A more iterative and modular approach might make sense here.

Also, testing full responses might introduce an avoidable type of duplication if you have many resources that include other, already tested resources. For example, if the contact payload includes an address payload you may opt into not hardcoding the address, because the serialization of it is already tested elsewhere:

  expected = {
    id: 0,
    email_address: "hey@richsteinmetz.com",
    first_name: "Rich",
    last_name: "Steinmetz",
    nick: "Gepard",
    age: 142,
    address: address_attributes, # This is dynamic.
    created_at: "2024-01-27T15:51:05.981Z",
    updated_at: "2024-01-27T15:51:05.981Z"
  }

Other times, you will find that it's not as easy to hardcode dynamic values, e.g. a UUID that is always generated before save in which case your test might opt into asserting that contact.uuid is a string of a certain length, instead of . In other cases, stubbing values could also make sense to test for expected transformations of the filed values.

When Breaking an API Contract Is OK

Never.

You should never break a contract of a web API (or any other API) that is used or potentially used by others.

Once parts of a web API are exposed publicly, things become tricky. An early adopter might be spying for new changes in your API and begin their development shortly after you make it public.

Same with old functionality. You might look at your monitoring tools and see that no one used your endpoint for 3 months. You deactivate it, but it turns out that someone was about to connect to it after integrating against it for the past 3 months.

Web API Testing in the Wild

Let’s take a final look at how “real” open source APIs are doing it in the wild.

GitLab

https://docs.gitlab.com/ee/api/settings.html

GitLab has a huuuge test-base, so no accusations, cause one might miss some parts. But on the first glance, there seem to be a lot of variable attribute-based tests. Here’s an example though, where we can see more invariable attribute-based tests that actually test for the response values:

# ...
it "updates application settings", fips_mode: false do
  expect(response).to have_gitlab_http_status(:ok)
  expect(json_response['default_ci_config_path']).to eq('debian/salsa-ci.yml')
  expect(json_response['default_projects_limit']).to eq(3)
  expect(json_response['default_project_creation']).to eq(::Gitlab::Access::DEVELOPER_MAINTAINER_PROJECT_ACCESS)
  expect(json_response['password_authentication_enabled_for_web']).to be_falsey
  expect(json_response['repository_storages_weighted']).to eq({ 'default' => 100, 'custom' => 0 })
  expect(json_response['plantuml_enabled']).to be_truthy
  expect(json_response['plantuml_url']).to eq('<http://plantuml.example.com>')
  expect(json_response['diagramsnet_enabled']).to be_falsey
  # ... 70+ more expectations...

This approach is not optimal but kind of acceptable as long as it is always (somehow) ensured, that new fields are added to the list and json_response['attribute'] will error out when the key is suddenly missing in the response and/or on the model in the case where we test for stuff like .to be_falsey (which I actually doubt in this case unfortunately).

Exercism

My spot check for Exercism’s Tracks for Mentoring API might look like some kind of an “attribute-based testing fallacy” at first:

# website/test/controllers/api/mentoring/tracks_controller_test.rb
###
# Show
###
test "show retrieves all tracks" do
  user = create :user
  track = create :track
  create :track, slug: :javascript
  create(:user_track_mentorship, user:, track:)
  setup_user(user)

  get api_mentoring_tracks_path, headers: @headers, as: :json
  assert_response :ok

  # JUST TESTING SOME VARIABLE AGAINST A VARIABLE? 😒
  expected = {
    tracks: SerializeTracksForMentoring.(Track.all, user) 
  }
  assert_equal expected.to_json, response.body
end

But on a closer look the Track serializer is tested in more detail including different expectations for different use cases:

lass Mentor::Request::RetrieveTracksTest < ActiveSupport::TestCase
  test "serializes correctly without mentor" do

    # ... more code ...

    expected = [
      {
        slug: "csharp",
        title: "C#",
        icon_url: csharp.icon_url,
        num_solutions_queued: 6,
        median_wait_time: nil,
        links: {
          exercises: Exercism::Routes.exercises_api_mentoring_requests_url(track_slug: 'csharp')
        }
      },

There's still quite some dynamism in this test and you'll need to dig deeper into the different test classes and their method calls to get the full picture of what is actually tested. However, the complexity of an application, might sometimes call for more dynamism and complexity in your tests.

Save the Day!

No one likes big surprises and breaking software, especially when they put effort into working with your creation. So let’s show appreciation to our API clients’ efforts and save their days by doing our best in delivering on the promised contract 🙏