Support

Fine-tuning the sync downloader

The purpose of this guide is to explain the ways how to speed up MCRM synchronization. We start by explaining internals of the synchronization process. Then we introduce main variables that control sync performance. Finally, we’ll explain how to formally write the setup string.

What is the SyncDownloader?

SyncDownloader executes perhaps the most important part of the synchronization – download of “normal” entity records.

  • It downloads accounts, contacts, orders, orderdetails…
  • It does not download some special entities such as NN entities or activityparty’s.
  • It does not download attachments, images.
  • It does not execute upload of client changes to the server.
  • It does not perform sync with SharePoint, OneDrive, Google Drive etc.

For a typical configuration most of the sync time is spent in the SyncDownloader.
Next we’ll describe in deeper detail how the SyncDownloader works. If you’ll understand this, you’ll be able to fine-tune the SyncDownloader to maximize its performance. Let’s start with a simple question: How can we measure the sync performance?

First of all we shall compare FullSync times. (Incremental sync with its low data traffic and relatively large overhead is difficult to judge.)

Secondly, we have two obvious metrics:

  1. Total time spent in the download. You can find this number in the sync log and it looks like “... Entitys=54141ms...” (ms stands for milliseconds). This measure is good when we have relatively stable set of server data. However, it does not allow comparing sync performance against different servers.
  2. Alternative measure is the Average time needed to download 1 record. Let’s say that we have this sync log excerpt:
    <Summary ... Sent=0 Recv=366590 Result=Normal/> ... <FullSync> Entitys=54141ms... 
    The average time per record = 54141/366590 = 0.15 ms/rec.
    (Speed ~1 ms/rec can be considered as a solid performance. Above number is much better and was achieved through the steps explained in the following chapter.)

How does the SyncDownloader work?

Entities are sorted somehow. (The exact criteria are unimportant.) The downloader opens (usually) 3 download threads that work simultaneously – hence the term multi-downloader. Each thread takes first entity that was not yet downloaded and executes the download of all relevant entity records. When the download is ready, the downloader takes next free entity. If there is no such entity, the download thread quits. The downloader finishes when all its download threads terminate.

The download of entity records is done in steps, also denoted as pages. By default one page consists of (up to) 500 records.
For example if we need to download 1253 account records, the downloader will issue 3 web requests that will return subsequently 500, 500 and 253 records.

What happens to the downloaded records?
The downloader has a cache, where the downloaded records are stored. All download threads write to the cache and the upper layer (SyncEngine) takes over the cached records and stores them in the database.

The cache has limited space and it often happens that it gets filled. Here is a typical scenario:

  • 3 download threads download accounts, contacts and orders subsequently.
  • SyncEngine processes the account entity (The account entity is active).
  • Hence the account records do not remain in the cache (they are consumed immediately), but contacts/orders are stored in the cache at least until account download finishes.

So what happens when the cache gets filled?
Of course, the download of inactive entities gets interrupted temporarily. (It will be resumed when the cache free space allows.)

Download pauses slow down the overall performance. MCRM is rather conservative in allocating the cache space, hence pauses appear rather often, especially if downloaded entities have many records.

Parameters that control the download process

The exact usage of these parameters will be clarified later by examples. Here we provide just the formal definitions.

NumDownloadThreads – Count of download threads used by the downloader. Default=0, i.e. MCRM default applies*). Allowed non-zero values: 3-10.
DownloadCacheSize – In MB. Default=0, i.e. MCRM default applies*). Allowed non-zero values: 10-1000 for Windows platforms, 10-500 for mobile platforms.
DownloadPageSize – Page size of download fetches. Default=0, i.e. MCRM default applies*). Allowed non-zero values: 100-5000.

The last parameter that can be changed is the entity order. More precisely, user can assign priority to selected entities. Such entities will be downloaded at the
beginning, i.e. before other entities. You can do so for up to (NumDownloadThreads-1) entities.

*) MCRM defaults (as of MCRM 11.1):
NumDownloadThreads = 3 (Unless the app setting MultiThreadSync is switched off, in which case the SyncDownloader is not used. This is a very slow option – not recommended.)
DownloadCacheSize = 250 MB for Desktop/WinRT, 50 MB for WinUWP, 10-50 MB for Android/iOS (depending on the device parameters)
DownloadPageSize = 100 for emails, 500 otherwise.

Typical problems

Before we discuss typical problems and try to formulate recommendations how to modify download parameters, let’s explain how the client-server communication works.

MCRM typically communicates over https. Here is a simplified web request life cycle:

  1. Request composition. (Done instantly, which means the time spent in this step is negligible with respect to other steps.)
  2. Sending request to the server. The duration – called latency – depends upon the connection band width and the distance between the client and the server.
  3. Server processing. This as a rule translates down to an SQL database query. Usually fast, but there are exceptions. Some queries may even timeout the server.
  4. Sending server response to the client. This also counts to the latency.
  5. Client processing. Includes response parsing and processing – usually writing to the client database. While these actions represent considerable effort, they typically represent only a fraction of the time spent in the web communication.

 

Problem: Web latency

Mobile networks are known for high latency. If you use VPN, latency gets larger and if your server is 10000 km away, latency grows even more. Under these circumstances the whole web request life cycle may take well over 1 second.

Theoretically, if the latency is 1.5 second and the 3 download threads run at the full speed (500 records per request), we get 1500 records in 1.5 second, i.e. 1 ms/record.
However, this does not count for any overhead, for requests returning less than 500 records (they have the same latency!) and for download pauses. Hence the real
performance will be much smaller.

Recommendations:

  • Increase DownloadPageSize. Doubling the page size will halve the latency.
    Risk: Higher memory consumption.
    Tip: You may use different page size for different entities. If an entity has small records*), you can use much higher page size.
  • Increase NumDownloadThreads.

*) “Small record” means record with small data. Record can be large either because of many nonempty columns (empty columns do not count) or because of large text column(s).
Records containing a few 100s of Bytes are considered small.
Special case: Columns containing so-called attachments (activitymimeattachment.body, annotation.documentbody, salesliteratureitem.documentbody) are not counted.
These columns use to contain big data and that’s way they are excluded from SyncDownloader fetches. Instead, they are downloaded in a separate step as attachments.

 

Problem: Dominant entity

Imagine a model situation: 30 entities, 50000 contacts, remaining 29 entities have all together 50000 records as well. (This is what we call dominant entity.)
How efficient is a multi-downloader in this situation? Let’s do simple math:

For the simplicity, the single thread download speed is 1000 records/second.

  1. Single thread downloader needs 100 seconds.
  2. Multi-downloader (3 threads), contact entity is the last one: 67 seconds. (First 50000 records with the speed 3000 rec/sec, then 50000 records with the speed 1000 rec/sec.)
  3. Multi-downloader (3 threads), contact entity is the first one: 50 seconds. (Download of all non-contact records is “for free”.)

To complete the picture: If there was no dominant entity, then the theoretical multi-downloader speed would be 3000 rec/sec and it would need 33 seconds.

Recommendations:

  • Dominant entity: Set on priority so that the entity download starts right at the beginning.
  • Dominant entity: Increase the page size as high as possible. (Decreasing the latency)
  • NumDownloadThreads = 2 (Motivation: Remove every possible obstacle for the dominant entity. Probably a minor effect only.)

Platforms, devices

You can specify conditional setup valid for some platforms only; see the examples bellow. Here is the list of supported platforms (the names are case sensitive):

  • android – Android devices
  • ios – iPhone, iPad
  • Desktop, WinRT, WinUWP – Windows platforms
  • Windows – This is a shortcut for Desktop + WinRT + WinUWP
  • Mobiles – This is a shortcut for android + ios

Another device-related condition is OnlyHighEndDevices='true'.
As of MobileCrm 11.1 high end devices are:

  • All Windows devices
  • iPhone5+, iPad3+
  • Android: Devices with >100MB of available memory & able to allocate 50MB continuous memory block

(Note: This definition might change in the future.)

How to specify Sync setup

The setup is part of the customization and is specified through Woodford.

SyncLog information

To double check the setup values really used in the synchronization*) check the log from the last synchronization. Look for a text similar to:

<AdvancedSetup>
DownloadCacheSize=500
DownloadPageSize=2000
Entities: my_entity1[priority, DownloadPageSize=5000] my_entity2[DownloadPageSize=5000]
</AdvancedSetup>

This text is permanently stored in the log before the first use of the advanced settings. (For the case something goes wrong.)

Sync log contains additional information that is important for designing downloader cache. It might look similar to
<Downloader CacheSize=200MB UsedCache=50MB />

In this case we designed unnecessary large cache. This does not present any problem because the cache grows dynamically (DownloadCacheSize is just theoretical upper limit), yet it is good to know real memory consumption.

The downloader summary may contain also the information about the pauses as for example
<Downloader CacheSize=100MB UsedCache=81MB Paused=18% />

In this case we know that the downloader paused for considerable time. As the single possible reason is insufficient cache size, increasing the cache size would improve the performance.

 

*) Advanced settings generally contain several platform-dependent setups. SyncEngine selects settings applicable to the current platform. Moreover, settings may be justified to allowed ranges.

Warning

Default MCRM settings are rather conservative to fit all supported devices. Millions of successful synchronizations prove this point.

Here is how the settings evolved:

  • MCRM used DownloadPageSize=500 since the very beginning. (Except emails which used page size 20. Long emails presented a problem in those days.)
  • January 2013: MCRM introduced multi-downloader with 3 threads.
  • July 2016: DownloadCacheSize was increased for the last time.
  • June 2017: Page size for emails was moved to 100. (More precisely, EmailFetchPageSize app setting was introduced with the default value 100.)

As mobile devices improve every year, there is certainly place for moving up from the default values. However, keep in mind that by doing so you increase the memory stress on the device. Never use these setting in the production without thorough preliminary testing.

Examples

5 threads, 5000 records per page, 1GB cache size. Valid for all platforms. Should result in very fast sync – unless there will be a memory problem (mobile devices). Nevertheless, the setup should work for the desktop.

<SESetup Version='1'>
<SyncDownloaderSetup>
  <Setup DownloadCacheSize='1000' DownloadPageSize='5000' NumDownloadThreads='5' />
</SyncDownloaderSetup>
</SESetup>

 

Desktop: 5000 records per page, 1GB cache size. High end devices: 5000 records per page, 300 MB cache size. Other devices: MCRM defaults apply.

<SESetup Version='1'>
<SyncDownloaderSetup>
  <Setup Platform='Desktop' DownloadCacheSize='1000' DownloadPageSize='5000' NumDownloadThreads='5' />
  <Setup DownloadCacheSize='300' DownloadPageSize='5000'  OnlyHighEndDevices='true'/>
</SyncDownloaderSetup>
</SESetup>

 

All Windows platforms: 5000 records per page, 1GB cache size. Mobile devices: MCRM defaults apply.

<SESetup Version='1'>
<SyncDownloaderSetup>
  <Setup Platform='Desktop WinRT WinUWP' DownloadCacheSize='1000' DownloadPageSize='5000' />
</SyncDownloaderSetup>
</SESetup>

 

Yet another example of platform-dependent setup.

<SESetup Version='1'>
<SyncDownloaderSetup>
  <Setup Platform='Desktop WinRT WinUWP' NumDownloadThreads='5' DownloadCacheSize='1000' DownloadPageSize='5000' />
  <Setup Platform='android ios' DownloadCacheSize='1000' DownloadPageSize='5000' />
</SyncDownloaderSetup>
</SESetup>

 

Same effect as above.

<SESetup Version='1'>
<SyncDownloaderSetup>
  <Setup Platform='Desktop WinRT WinUWP' NumDownloadThreads='5' DownloadCacheSize='1000' DownloadPageSize='5000' />
  <Setup Platform='Mobiles' DownloadCacheSize='1000' DownloadPageSize='5000' />
</SyncDownloaderSetup>
</SESetup>

 

The new feature here is specifying priority for an entity. Multi-downloader will start its job by downloading this entity. (Suitable for a dominant entity.)

<SESetup Version='1'>
<SyncDownloaderSetup>
  <Setup Platform='Desktop' DownloadCacheSize='1000' DownloadPageSize='5000'>
   <Entities>
    <Entity Name='mvow_serialnumber' HasPriority='true' />
   </Entities>
  </Setup>
</SyncDownloaderSetup>
</SESetup>

 

This complex setup illustrates how competing platform setups are evaluated. In this case there are two setups applicable to Desktop platform. The last one is selected as it provides the most specific platform specification.

<SESetup Version='1'>
<SyncDownloaderSetup>
  <Setup Platform='Desktop WinRT WinUWP' DownloadCacheSize='1000'>
   <Entities>
    <Entity Name='mvow_serialnumber' DownloadPageSize='2000' HasPriority='true' />
    <Entity Name='mvow_accountaddress' DownloadPageSize='2000' />
    <Entity Name='mvow_itemnumber' DownloadPageSize='2000' />
   </Entities>
  </Setup>
  <Setup Platform='Mobiles' OnlyHighEndDevices='true'>
   <Entities>
    <Entity Name='mvow_serialnumber' DownloadPageSize='1000' HasPriority='true' />
    <Entity Name='mvow_accountaddress' DownloadPageSize='1000' />
    <Entity Name='mvow_itemnumber' />
   </Entities>
  </Setup>
  <Setup Platform='Desktop' DownloadCacheSize='1000' NumDownloadThreads='5'>
   <Entities>
    <Entity Name='mvow_serialnumber' DownloadPageSize='4000' HasPriority='true' />
    <Entity Name='mvow_accountaddress' DownloadPageSize='2000' />
    <Entity Name='mvow_itemnumber' DownloadPageSize='2000' />
   </Entities>
  </Setup>
</SyncDownloaderSetup>
</SESetup>