My FeedDiscussionsHeadless CMS
New
Sign in
Log inSign up
Learn more about Hashnode Headless CMSHashnode Headless CMS
Collaborate seamlessly with Hashnode Headless CMS for Enterprise.
Upgrade ✨Learn more
Jon Dixon

6 likes

·

4.3K reads

5 comments

Jonathan Kearney
Jonathan Kearney
Nov 4, 2022

Another great article, all very useful techniques to know from a practical point of view and a good lesson in theory too. Clearly explained as always, excellent

2
·
·1 reply
Jon Dixon
Jon Dixon
Author
·Nov 5, 2022

Thanks Jonathan

·
Plamen Mushkov
Plamen Mushkov
Nov 3, 2022

Great article! Really enjoyed it. Really impressive performance improvement with the last method.

Speaking about other methods, it could be again any of the current ones, but fetching only the deltas - records updated/inserted after the last sync. I have always used to have a specific package and table to keep the delta information (like last sync date, latest id, etc.). And then my query would only return the needed records which are then transferred using any of the methods you mention. Or directly in bulk using a DBLink if available.

·
·1 reply
Jon Dixon
Jon Dixon
Author
·Nov 5, 2022

I completely agree bringing just changed data is definitely the way to go for overall performance. This was more of a discussion on throughput. Thanks for your comment and idea.

1
·
Roger Scroce
Roger Scroce
Jul 28, 2023

Hi Jon, Just read your article. My preferred approach to this use case is PL/SQL.

To avoid the sequential approach and the need to parse the response twice to test for the next page of data, I use a 2 step approach:

1) call the rest api for a SINGLErecord to get the total record count.

2) build and execute ALL the api calls for individual pages in parallel. Parameterise the maximum concurrency in light of vendor rate limiting.

For my use case, Oracle CX implementations, this routinely achieves a 10X to 20X overall throughput.

100% Oracle and 100% free with Oracle ADB !

·