Spark API Replication
If you'd like to replicate data and are not sure if you've been provided a key for replication, or find a problem with this outline, please contact firstname.lastname@example.org.
The Spark API endpoint for replicating data is:
https://replication.sparkapi.com. Developers replicating data
must use this endpoint.
Developers who wish to replicate data are granted Personal Access Tokens with special replication access. These keys are granted a few additional privileges to make replicating data easier. For example:
- The maximum
_limitis raised to
1000for listings and accounts, and
200for other resources.
- The accounts resource does not apply company or office restrictions when searching user records for IDX roles.
CustomFieldsExpandedRawexpansions are available for listings when retrieving multiple records.
These keys are otherwise identical to regular API keys, with access to the same resources and governed by one of our data roles
While the listings resource is used in these examples, the same general guidelines also apply to replicating other resources.
1. Initial Download
Optionally, you may include the
_select parameter, but no other parameters should be specified.
ListingKey orderby is applied by default, ensuring that the order of the results
is consistent while paginating with
2. Update Data
Keeping your result set updated involves making requests against the
ModificationTimestamp field using the timestamp of your last successful update minus 20 minutes to account for server side caching. Here is a pseudocode example to demonstrate:
query_timestamp = last_successful_retrieval_timestamp - 20 #minutes ?_filter=ModificationTimestamp Gt query_timestamp
Paginating through the update result set is once again accomplished by using
During this step it is recommended that you also update any listing subresources of interest. Note that our listing data has several timestamps that can be used to determine if the corresponding data needs to be updated:
3. Purging Stale Data
Repeat step 1 with the
_select=ListingKey parameter applied, paginate through the result set identically to step 1, and compare all
ListingKey values returned with the values in your database. Purge any records that do not match.