Comment by TheSheriff on "How to parse large amount of JSON data on mobile browsers?"

Ok, so couple of points:

If you're working with mobile I would suggest using SQLite (optimised for mobile).
Why do you need to download 25-100MB of data?
That's a lot of data - are you sure you need it?
You have to be aware of the storage on a mobile device and asking for >100MB just for storage/cache is not a great idea.
You're working with a website inside a native app? Why not just optimise the website?
PSON is great for reducing JSON size by up to 50%.
Are you certain on the method of your offline storage?
Could you store only critical information 1-2MB then query a certral DB for the rest when the user comes back online?

+1 for 2. and 3 :D 100MB of data is too much. The data should be reduced to a minumum for offline or distributed with the app installation (in the final form, for example SQLite)

@maruru Ty :D. If i recall correctly from my android days asking for anything over 20MB is basically considered 'WTF do you think you're doing' lol.

SQLite is not an option, because I need to store the data across browsers and OS (I am using IndexedDB with WebSQL fallback). I am using Dexie.js with Nparashuram's IndexedDB shim. 2. I need large amount of data because the user should be able to run offline for multiple days. 3. Yes. 4. True. 5. Website. So the user loads a webapp, but inside native app I am using Bluetooth capabilities from within my webapp (something like cordova, but made from scratch) 6. I will take a look, thanks! 7. Yes :). 8. No. The user will possibly need more data. The app crashes on older devices, where the user will request large amounts of data.

@bboydflo Couple formatting issues there i think! haha.
I think there's a couple of points I'm struggling to understand - what's the use case exactly? Why does the user need to be able to look at the website for in excess of a few days?

Also, your point 5. is quite confusing, does it mean that whilst you have a website, you also have a native app built in a manner similar too cordova but uses bluetooth? Why on Earth do you need to use bluetooth unless you have a product they have as well?

Finally, could you define "older devices"? If you're pitching for android v2/3 then there really is no point worrying about them. Additionally, the reason the app crashes on older generation mobile OS's is because your trying to process a lot of data.

@hipkiss91 So, the use case is following: The user needs to manually go in offline mode. That means, download all the data it needs to (it depends from user to user) and edit it for a couple of days while offline. When there is connection again, the user will attempt to sync all this data back to the server. The app is a webapp that should run cross browser. But we also have an Android webview wrapper, that loads the same webapp. From this wrapper the user is able to connect to bluetooth and use different bluetooth readers to read data into the webapp. This will only be available when the app is loaded in a android webview. Finally, when I say older devices I mean 512MB ram. I have couple of Android devices that run Android 4.x and have only 512 MB. When I try to JSON.parse a big string the app will crash most probably because of lacking memory. (Sorry for the formatting. I don't know how to start a new line)

@bboydflo Ok. For formatting, follow basic .md formatting - so for return use double space.

So, let me try and summarise - you have an android app that is a webview of your website and this app takes in data from a bluetooth source; the data is stored locally; then the data is uploaded to the website when the user returns online?

You need to be careful when you say older devices then - especially with mobile, as 90% of the time it refers to the operating system - not the hardware specs, these are two completely different things.

As for the use case, I understand what you're trying to do but I don't understand what data is being stored that is so critical and how there could possibly be so much.

@hipkiss91 You're almost there. The android app can connect to some bluetooth readers that would send some strings back to my web app. These strings represent Id's for my data. In online mode, I would require data directly from the server passing this id, but in offline mode I would get data from web storage. I think how the app works Is not so relevant in my case but I hope you understand how it works :)

@bboydflo I think I understand more now. But my thought process is that I don't understand what the data you're trying to store locally is. As you are aware, what you are doing causes the app/phone to crash if not enough memory is available to parse the data. This is a pretty big issue as you will never know what else the user has running in the background on the mobile device. Therefore your minimum required hardware of 512MB is ok, but things will still go wrong above this threshold.

Having an offline mode is becoming more and more important it doesn't mean that everything has to be stored offline, instead only critical information need be stored and updated once the user becomes online.
In my experience users tend to require more care and attention to the feedback instead of the core functionality. I.e. unsuring that if a user is offline, let them know and the reduced capabilities of an offline mode.

Search Hashnode