Thread

Florin Cosmin

javascript developer

Nov 8, 2016

How to parse large amount of JSON data on mobile browsers?

I have this use-case where I work on a mobile app, that runs inside a browser or (if android) inside of a WebView. This app needs to work offline, so I need to download big amounts of data (25MB-100MB) at once, parse it, and save it in the local database (indexeddb or websql).

Especially on mobile, parsing huge amounts of data is very slow, and can potentially break the app. The amount of memory used is not enough and the app can crash.

What would be a good strategy/solution to fix this issue?

#android #javascript

Responses(4)

TheSheriff

Co-Founder, Founder, Entrepreneur & Problem Solver

Nov 8, 2016

Ok, so couple of points:

If you're working with mobile I would suggest using SQLite (optimised for mobile).
Why do you need to download 25-100MB of data?
That's a lot of data - are you sure you need it?
You have to be aware of the storage on a mobile device and asking for >100MB just for storage/cache is not a great idea.
You're working with a website inside a native app? Why not just optimise the website?
PSON is great for reducing JSON size by up to 50%.
Are you certain on the method of your offline storage?
Could you store only critical information 1-2MB then query a certral DB for the rest when the user comes back online?

It's difficult. Browsers manage storages differently, and mobile browsers do it extra differently even between mobile and desktop browsers of the same vendor.

Have a look at this great article about storages, quotas, and limits -> html5rocks.com/en/tutorials/offline/quota-research

If you managed to solve the storage problem, then another problem arises. Processing power and working RAM memory.

Browsers tend these days to notify the user about the current page being unresponsive and ask to kill the process or wait a little longer. Typically (desktop) browsers wait 60 seconds for the document ready event. If your app is busy processing some data and doesn't signal some readiness to the observer, then the user has to decide to kill your app or wait. Mobile browsers usually ask earlier because draining the batteries life is critical for the user.

If you managed to store the raw data and managed to cut the processing in little chunks not to raise unresponsive process alarms, then you are back again to storage problems. This time your database of choice needs to store the processed data. Again finger pointing to the article from above -> html5rocks.com/en/tutorials/offline/quota-research

If you managed both storage problems and the processing issues, then you need to deal with rendering all the data. Different strategies require different preparations. I'm not talking about the choice between React.js or Angular2.js; I'm talking about loading the processed from the mobile database and render it somehow. You need a data access strategy because the databases of mobile browsers are slow and with slow I mean slow. Here is a simple benchmark inserting only simple data and the test takes almost a minute to finish on a Mac Pro 64GB powered Chrome 54 desktop browser. -> http://nparashuram.com/IndexedDB/perf/index.html

Maybe tablets have enough power, and the right settings enabled to do somewhat well, but I have no hope.

I would try to keep the mobile device as a simple rendering device allowing users to interact with the drawings.

Use a backend system to process the JSON data, slice and cut it into the tiniest chunks of data possible. Know the use cases and selectively download only the data needed to render.

Or try ServiceWorkers.

littlepsylo

Software engineering

Nov 8, 2016

Without making a native app, have fun !

TheSheriff

Co-Founder, Founder, Entrepreneur & Problem Solver

Nov 8, 2016

Even with making a native app - storing that much data locally is bad practise

@hipkiss91 But it's easier than managing it with a webview.

@bboydflo I think I understand more now. But my thought process is that I don't understand what the data you're trying to store locally is. As you are aware, what you are doing causes the app/phone to crash if not enough memory is available to parse the data. This is a pretty big issue as you will never know what else the user has running in the background on the mobile device. Therefore your minimum required hardware of 512MB is ok, but things will still go wrong above this threshold.

Having an offline mode is becoming more and more important it doesn't mean that everything has to be stored offline, instead only critical information need be stored and updated once the user becomes online.
In my experience users tend to require more care and attention to the feedback instead of the core functionality. I.e. unsuring that if a user is offline, let them know and the reduced capabilities of an offline mode.

Search Hashnode

How to parse large amount of JSON data on mobile browsers?

Responses(4)

Recent threads