Building Smarter with Intuit: Stay in sync with CDC

As part of a new series of articles around best practices for developing applications with Intuit APIs called “Building Smarter with Intuit”, we’re sharing insights and tips on optimizing everything from authorization to advanced platform capabilities. Stay tuned for more articles in this series designed to help you level up your app development on the Intuit developer platform.

The third article in this series covers the change data capture (or CDC for short) operation. Let’s start with a quick recap. To maintain a seamless user experience, your app will likely want to store the latest state of a given transaction, list record, or entity. There are multiple ways to achieve this, including querying the API entity itself or using an event-driven approach such as webhooks. If instead, your architecture plays nicer with polling based approaches, CDC might be the solution for you.

So how does it work? Conceptually, CDC accepts a dateTime stamp as well as the list of comma separated entities your app is interested in. CDC will then set a look-back period based on the dateTime stamp and return all entities that were changed in this period.

CDC also has its own dedicated endpoint, namely:

https://quickbooks.api.intuit.com/v3/company/<realmId>/cdc

Additionally, with every CDC request, you’ll want to append two important parameters:

entities=<entityList>
changedSince=<dateTime>

Putting this all together, an example request looks like:

https://quickbooks.api.intuit.com/v3/company/<realmId>/cdc?entities=customer,invoice&changedSince=2023–08-01T12:00:00

Once your app receives the response, it should make updates to the respective records in its database. Be sure to find the correct record by relating against entity IDs.

There are some important characteristics of the CDC operation:

• CDC operations return full payloads for relevant entities, not just the attributes that were changed.
• CDC operations track changes within the last 30 days.
• The maximum number of objects that can be returned in a response is 1000. Consider querying for shorter time periods if you are hitting this upper limit.
• Objects are grouped by type and then in order of last updated time within the group. Objects deleted within the look-back period are returned after active objects.
• The CDC operation can be used for all API entities except the following:
  • journalCode
  • taxAgency
  • timeActivity
  • taxCode
  • taxRate
• The content type should be text/plain.
• The accepted date format is ISO (YYYY-MM-DDT00:00:00).
• The response only lists entities that have changed since the look-back date.
• If there were no changes in the look-back period, the response will be mostly empty.

Now that we’re familiar with CDC, let’s cover a few use cases.

Note: for greater legibility, we’ll only include a small number of attributes in the snippets below.

Use case 1: refreshing on a given frequency

Let’s say your app wants to refresh data on a given frequency, perhaps once each hour or once each day. To start, you should think about the entities your app is interested in. Depending on your use case, this might mirror the entities you commonly make requests against. For example, if your app mainly interacts with invoices, payments, and customers, you’ll likely want to add all of these entities to your CDC request. If instead, your app calls bills, expenses, and AP reports, you’ll only want to include bills and expenses in your CDC request while leaving reports out entirely.

For the logic in your code, consider the following:

• In addition to a variable placeholder for realmId, you’ll also need one for a dateTime stamp.
• A way to increment the dateTime stamp based on your given refresh/sync frequency.

An example of a summarized response looking at the customer and invoice entities is as follows:

"CDCResponse": [
  {
   "QueryResponse": [
    {
     "Invoice": [
        {
	  <full payload for invoice A>
        },
        {
	  <full payload for invoice B>
        }
      ],
      "startPosition": 1,
      "maxResults": 2,
      "totalCount": 2
    },
    {
     "Customer": [
        {
	  <full payload for customer X>
        },
        {
	  <full payload for customer Y>
        }
      ],
      "startPosition": 1,
      "maxResults": 2,
    }
  ]
 }
]

From here, scrape the IDs and relevant fields from the response and relate against current records in your app’s database using entity IDs.

Use case 2: user initiated refreshes

You may have a ‘sync’ or ‘refresh’ button in your app that allows the user to refresh QBO data on their schedule. When a user clicks this, consider starting the back-end flow with a CDC call. The dateTime stamp in this case would be dependent on the time your app last synced with QBO.

Post-CDC call, if your app is syncing the latest version of a given set of entities from QBO then it’s business as usual and you’ll make the required changes to the respective records. If instead, your app has an open queue of jobs that will carry out write operations to QBO in addition to syncing the latest versions of entities, then you’ll want to examine your use case and decide which operation (write to QBO vs. write to DB) should commence first. By doing so, you’ll minimize the amount of potential errors.

Use case 3: durable architecture using webhooks + CDC

To truly make your architecture durable, consider employing both CDC and webhooks. At face value, this might seem strange. Both of these capabilities ultimately strive to do the same thing – that is to alert you about changes to entities. However, both do so in slightly different ways (e.g., event-driven vs. polling).

One such way to implement this is as follows:

1. Listen to the relevant event triggers for the entities your app is interested in.
2. As your app receives webhooks, process them as you normally would.
3. On a given frequency, perhaps nightly, make a CDC call.
   • Note: the dateTime stamp here should mirror your CDC frequency. In this case, 24 hours ago.
4. Using the entity IDs in the response and dateTime columns in your database tables, reconcile updated records in the last 24 hours.
5. If you find any relevant records where updates did not happen but should have, use the payload returned in the CDC response to make the required updates.

You now have a durable architecture with redundancies and extra validation built in. Your customers will have even greater confidence that both systems are in sync and you, as the developer, will encounter fewer business validation errors when calling Intuit APIs.

We hope this helps you implement CDC operations on the Intuit developer platform. Keep an eye out for the next article in our “Building Smarter with Intuit” series. In the meantime, you might want to read a similar article on batch operations, which you can find here.