How to speed up questionnaire sync and optimize storage use with the new JSON (+Comparison with the legacy formats)

What is the purpose of collecting data if they get stuck somewhere between the field and office? Or what is the purpose for mobile workers to use digital tech if it causes more hassle for them?
That’s why for us at Resco, the synchronization functionality and its adoption have always been a priority, as it enables mobile workers and office teams to work better together.

The recent update of Resco solutions takes this to the next level again.

The new JSON storage of questionnaires brings a lot of possibilities for improving synchronization’s speed and efficiency.

That’s why we have put together the most up-to-date information and a short guide so you can make the best out of it.

In this article you will learn about:

• What were the previous storage options and how they worked
• How the new JSON stores data
• Comparison of storage space used by the new and legacy JSON formats
• What are the recommended actions for using JSON
• How to create questionnaires with the correct setup

The legacy storage options

Record-based storage

The first option was record-based, where each newly completed questionnaire included its own questionnaire record, question records, and question group records.

Over time, this could cause a lot of records to be stored in the database, and could lead to a long synchronization time and even server timeouts.

That could lead to potential integrity problems, such as incomplete questionnaire. This would happen when the synchronization was interrupted while only a part of the question records were transferred.

Template dependency

This option creates a distinction between the core and template-dependent fields.

Core fields cannot be duplicated for other questionnaires and posses unique information as ‘value’, ‘score’, and ‘answeredon’ fields.

Template-dependent fields are ones that can be used for multiple questionnaires. That means the data doesn’t need to be stored for each completed questionnaire, which saves a lot of space.

Such fields are ‘resco_styles’, ‘resco_localization’, ‘resco_rules’ and ‘resco_autoreport’, that usually tend to take the most storage.

This method improved upon the records-based storage and decreased the amount of space needed, but also makes the questionnaire not complete without the template, and cannot run properly without it.

Note that template dependency comes hand in hand with template versioning, which addresses the problem with template changes.

Legacy JSON

With this storage method, we serialized all the fields presented in the ‘question’ and ‘question_group’ records into a one JSON string.

The data string was stored in ‘resco_serializedanswers’ when a new questionnaire was created. This method eliminated the need for question and ‘question_group’ records.

The origin JSON improved the storage significantly, but it was still storing all the fields, some of which were not necessary (if the template dependency was not turned on). The format used standard .NET serialization and was not optimal as far as storage was concerned

Recommended actions for the older storage methods

Currently, you can use all of the mentioned methods to store questionnaires, but the status will change with the update in December, 2023, where we will be deprecating legacy JSON and record-based storage.

Recommended actions regarding the legacy storage formats are:

• If you don’t have any integration with third-party tools that parse JSON data, we recommend switching to the new JSON right away to start benefiting from a reduced database footprint and shorter sync times
• • If you do have an integration with third-party tools that work with questionnaire answers in legacy JSON format, check it’s working with the latest version and adapt your processes during the transition period
• In either case, you have the option to convert your existing questionnaire data to the new JSON format with Resco developed tool called Questionnaire Converter. To learn more technical details, please, contact Resco support

How does the new JSON storage option work

The newest storage option is made to be modular by giving you the option of selecting which information should be stored and which shouldn’t. It also includes versioning and dependency settings that are mandatory for this method.

There are two options available, Minimal JSON and Flexible JSON:

Minimal JSON

This option stores the least amount of information, just key-value pairs, where the key is question name and the value is its answer.

Also, questions containing a default value can be excluded from the storage, as the answer is known and can be reconstructed.

With this method you can achieve the most efficient storage possible yet.

Let’s see some example where we have a question on the one side and answer on the other. Firstly, we have a questionnaire form filled:

This is how it will look in the JSON string format (note that the actual string is stored as just one line):

As you can see, there is only essential information about the questions, groups, and answers.

Flexible JSON

Using the Flexible JSON method, you can customize the format by choosing which information is stored.

It also improves on the problems of the original JSON as it omits almost all of the information that can be reconstructed from the template.

Let’s see some example using the same questionnaire data from the Minimal JSON:

As you can see, the amount of information here almost doubled from the minimal JSON, and it also includes data such as score of the question answer, lookup, and option set labels.

Size comparison of different storage methods

Firstly, we need to look at the impact of the template dependency for storage-based method. As you can see from the graph below, it decreases the storage required to about 8% of the original one.

In this example, there are 2300 questionnaires answered with 21 question groups and 183 questions for each one.

Next, we can have a look at Minimal JSON. Looking at the same scenario and given that all of the questions are answered, it looks something like this:

The Minimal JSON improves the storage 6 times, but a more realistic scenario, where not all of the questions are answered, would be around 10 times.

In the next graph, we can see the difference between Flexible JSON and Record-based method. The scenario is the same as with the previous example:

With Flexible JSON there are more fields present in the string, but the storage efficiency is still about 5 times better.

Here you can see all four scenarios together:

How to create a questionnaire with answers stored in JSON format

Now we can have a look at how you can create your questionnaire with these settings enabled to help you with decreasing your storage requirement.

Currently, when you create a questionnaire, the settings are set to default for the new JSON method with versioning and dependency enabled (and these settings are required for the method).

We recommend keeping these settings on, because as mentioned previously, we are planning to deprecate the other legacy methods by the end of 2023.

When you open your questionnaire and set up your question and question groups, you can have a look at the JSON string in the preview mode:

With this setup, you have a handy overview of how the storage file is going to look like and what information is included.

How can you further optimize Minimal and Flexible JSON?

It’s important to note that with both Minimal and Flexible JSON, length of the questions names has a significant impact on storage.

It’s especially valuable to consider this if you are managing a large number of questionnaires.

For example, imagine a questionnaire with 183 questions where a single question contains an average of 18 characters.

Now, let’s say we would be able to optimize the length to just 1 character per question, which would save over 3K characters per questionnaire.

If you would be able to shorten the names across 2300 questionnaires per day, you could save approximately 150MB of storage per month.

You can achieve this while designing your questionnaire for each of the questions:

What’s next?

With the introduction of the new JSON storage option, you can choose to store only the information you need and reduce the size of your database, resulting in shorter synchronization times.

And with the deprecation of older storage methods in December 2023, it’s important to make the switch as soon as possible. Check out Resco’s Wiki for more information on the Resco Inspections data model and JSON storage for questionnaires.

If you would like to learn more about how to optimize sync speed for Inspections, take a look at synchronization training provided by Resco Support Team Lead Timotej Leško.