Documents in Resco Mobile CRM (SharePoint, DropBox, Google Drive, OneDrive) – a technical deep dive – Part 4
We’ve looked at the basic terminology and the presentation of documents in the app’s user interface in the first part of this series and continued with document filtering, viewing, editing, storage and safety. Last week, we examined the details of how Resco Mobile CRM works with file hosting services (especially SharePoint). And now we’d like to wrap up our technical deep dive into document integration with a couple of final observations and tips for improving the app’s performance.
8. Downloader performance
Synchronization with a file server is a 2-step process. The client starts by sending series of server queries about what has changed in a particular folder. If a file reference is returned that needs to be downloaded, that reference is then handed over to the FileDownloadQueue which works in an independent parallel thread.
Usually it is the download queue which takes the most time, but occasionally – mainly for SharePoint – the opposite is true.
FileDownloadQueue executes simultaneous downloads of 5 (mobile devices) to 10 (desktop) files and is therefore very fast. The speed depends primarily on the connection bandwidth and the server performance.
MCRM measures the download speed; you will find it in the SyncLog. Here is an example showing download stats for a case with extremely high performance:
Notice that the speed number shown in the SyncLog (39.2 mbps – close to 8 MB/s) does not represent the true download speed as it is based on uncompressed data* and it also does not exclude download pauses.
*) Web transfer normally uses gzip compression. Counting uncompressed data means that (for example) plain text files will show much higher speed than jpeg images. Note that today’s most commonly used document formats employ compression – e.g. docx (which is a zip file), jpeg, pdf, mp4 just to name a few.
9. Planning the documents
The basic issue goes like this: You may have tons of files on your servers, but limited capacity of your mobile devices, weak bandwidth, etc. forces you to make compromises.
So what can you do?
Let’s start by simple bookkeeping:
- Get basic estimates for the documents stored on your servers – total size, document size distribution.
- Estimate the space that you can afford to allocate for documents on your mobile devices. 1 GB? 10 GB? Or even more? (MCRM documents are stored encrypted, but uncompressed.)
- Is your data well-compressible? If so (docs, text files, etc.) the download speed will be higher, if not (images) the speed will be lower.
- – Make download time estimates. For example, with a download speed of 8 mbps (a respectable number) the download of 1 GB of data will take 1000 seconds. Is it acceptable for a FullSync to take this long?
Design basic configuration:
- Select a reasonable MaxFileSize value (in Woodford or simply do so in the client Setup form). Files larger than this value will be left on the server (but still available for download on demand).
- If you need more fine-grained control, use BlobStoreFilter.
Now that you have some insight, it’s time to start an experiment. We recommend to use the desktop MCRM client because:
- The overall download stats won’t differ too much from mobile devices. (Mobiles are slower at data processing, but have more efficient web traffic.)
- You can organize the tests more efficiently and you get more detailed download stats (see the tips below).
Ready? Open MCRM and:
- Double check that proper servers are connected. (Setup form, Accounts section)
- Execute FullSync and then exit MCRM when finished.
- Evaluate: Analyze SyncLog. Go to MCRM AppData folder and analyze downloaded files (total size and number, largest files)
Satisfied with the result? If not, modify the configuration and repeat the test.
The following tips explain how to analyze the data and how to speed up testing (applicable for testing carried out on the Windows platform).
TIP: Your experiments will be easier with proper MCRM setup
To make testing faster and more transparent:
- Switch off background sync (Setting RequireSyncLogin). This makes synchronization predictable.
- Switch off auditing to eliminate unnecessary uploads.
- Switch off Exchange. Emails or other Exchange data are irrelevant for our testing.
TIP: How to avoid waiting for the download of CRM data
Maybe the download of your CRM data takes too long and you don’t want to waste this time for each test iteration?
Someone may come up with the idea of limiting CRM data (by using stronger sync filters or simply by setting low ‘Max Sync Records’ in the Setup form). However, that’s not the way to go. Because if you eliminate CRM records, you exclude also documents related to these records. And the test results will be corrupted!
Instead, take advantage of the fact that you are on the desktop and you have full access to MCRM AppData data:
- Make first FullSync with all file servers switched off (so that only CRM data and nothing else is downloaded).
- Switch on the file servers again and exit MCRM. (Do not sync!)
- Find the MCRM AppData directory and make a backup copy of it. Now whatever you do in MCRM (probably some experiment with documents), you can always return to the same basic state by restoring the MCRM directory.
TIP: Understand MCRM AppData folder
The MCRM AppData folder is typically located at C:\Users\\AppData\Roaming\MobileCRM (WinRT uses more complex path).
Database_8_0.sdf is the main DB file which contains mainly CRM data.
To modify the MCRM configuration manually (instead of using Woodford):
config.xml – You can change basic settings here.
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/MobileCrm/Configuration/ – Folder containing basic Woodford configuration. Contains amongst others these files:
SyncFilter.xml – Sync filter. You can edit this file manually.
BlobStoreFilter.xml – Document filter. You can create or edit this file manually.
blob/ – Blob store folder (contains all documents); has these important subfolders:
resco_sharepointdoc/ – All file documents are stored here
annotation/ – Notes are stored here
activitymimeattachment – Email attachments are stored here
salesliteratureitem/ – Sales attachments are stored here
TIP: What can be extracted from the SyncLog?
syncLog.txt can be found in the MCRM AppData folder. Following nodes contain document-related information:
SharePointDownload – contains stats related to SharePoint communication.
OneDriveDownload, GoogleDriveDownload, DropBoxDownload – stats related to the communication with file hosting services
FileDownloadQueue – stats for actual doc downloads. (SharePoint and all file hosting services share common file download queue.)
SyncLog may also contain additional details if you set the configuration setting to LogSyncDetails=true.
And that’s it for now. However, if you’d like this series and are looking for even more technical insight into Resco Mobile CRM, join us at resco.next 2018 (March 14-15) in Atlanta. You will enjoy 2 days full of hands on trainings, sessions, best practices, customer stories and fun.