The little-known secret of incremental synchronization: What to do, if your organization creates hundreds of records each day

white iPad phone and server syncing

Synchronization filters are usually the tools of choice to limit the amount of records pulled down to a mobile device during synchronization.
While they work perfectly during the full synchronization, there are some scenarios where they might not work as expected during incremental sync.
*For more details on the difference between full and incremental sync, see this blog post.
Imagine a sync filter on the Order entity, which defines that only the Orders which belong to current user should be downloaded.
During the full sync (empty database), everything works as expected. The server query is basically equal to the sync filter definition so only the Orders that belong to current user are queried and downloaded.
The situation is very different during an incremental synchronization (every subsequent sync after full). Only the changes – modified or newly created records since last synchronization – are queried on the server. Notice that the sync filter doesn’t play any role in the query; regardless of the sync filter definition, all changes are downloaded!
Only afterwards, once all changed and newly created records are present locally, the records that DO NOT satisfy the sync filter are deleted. So it is true that after synchronization only the records which comply with sync filters are present in the local database (only the Orders that belong to the current user), but the process how this is achieved is different than you might expect.
While this might not seem as significant difference, it can result in unexpected synchronization behavior and extend the sync time.
When (in our hypothetical organization) there are only a couple of Orders created each day, it doesn’t make much of a difference. But imagine a large organization with hundreds (if not thousands) of Orders being created or modified each day – and only a handful belonging to each respective user. This would result in thousands of Orders being downloaded, when in fact only e.g. ten are needed – and the rest is deleted (locally).
The reason for this behavior is performance optimization. We have implemented sync filters this way because querying all new or modified records that comply with the sync filter would need to be executed as two independent queries on the server (until Dynamics CRM 2015), which would significantly increase the synchronization time – and we’re not even taking into consideration the combination of the two results, to get the records present on both lists.

So is there anything you can do?

In scenarios with a large amount of new or modified records each day, it is worth reaching out to a different tool for defining user’s data subset – Dynamics CRM permissions rather than using the sync filters. If the users don’t have the need to see all the records (but have this privilege out of convenience) the situation can be easily sorted out by allowing the users to see only their records (records belonging to them). This will result not only in shorter synchronization times on the mobile, but also better performance on the server side and the web application.
And what if the situation requires the users to see all the records on the server – in the web application – but not in your mobile scenario? Well luckily for us all, with Dynamics CRM 2015 the Fetch query language was updated and now it is possible to write more complex queries. What this means in practice is that now we can create one query to get the new or modified records that comply the sync filter – with CRM 2015 that is. So watch out for the forthcoming updates of Resco MobileCRM as this improvement will be included soon!