Recommended way / order to read data from a webservice, parse that data and insert it in a SQLite db

Question

I'm one to start mentioning that I'm totally new to Android, I've just finished reading a quick, introductory book and now I have to implement my very first app. This app is going to be use to take orders. Among all the data I'm going to store in a local db, two tables are the most important: Customers and Articles, being the latter the largest of all the tables (aprox 20000 records) One of the main process in my app, fetches all the data that my app need to work off-line when the user press a button that starts the daily operations on the device.

Well, the process consists of the following steps :

a. Read a restful service to retrieve the Customers Data
b. Parse the response to Json Objects
c. Insert those customers to the Customers Table
d. Read a restful service to retrieve the Articles Data
e. Parse the response to Json Objects
f. Insert those articles to the Articles Table

This is what I've planned to do:

1. Write a helper class that handles all the HTTP GET requests. Then call this class whenever I need to download all the Customers and Articles data
2. Since all this process might take a lot of time, I need to do it the background. So based on some suggestions I'm going to put all this code inside a Bound Service.
3. While all this long processing is taking place in the background I'll have to show some sort of indicator (a ProgressDialog) This is the reason I opted for using a Bound Service

Though I think I've got the general idea of how to do most of these thing separately, I think that putting the all together is quite a different story.

So these are the questions I've got now that I have to put the puzzle together:

1. Do you think the order in which I'm executing all the 6 steps of the process described is correct / efficient? If you had to make some changes, what would you change?
2. If the activity that started the service is explicitly cancelled or is hidden by another activity, the service has to have a way to let the user know that the process has finished. How could I implement that?
3. Is it possible/ recommended to write to the SQLite DB within the service? Is it the same as when I do so within an activity?
4. In J2ME I've done something similar, and when I put something like the 6 steps I mentioned above all of them are executed sequentially, that is , one after the other. Is it the same in Android?

I hope I'm not asking too many questions. I just put them together because they are all related.

Basically in this question I'm not asking for working code ( though it'd be OK if you could provide some sample code) What I'm really after is some suggestions, some guidance. Something like "Hey, I think this might help you with point number 3" or "You might find this article useful", "I think you'd better off using this instead of that". That kind of thing.

I decided to come to you because you're the experts and I really need someone to put me in the right direction.

Thank you very much.

P.S. Please do not close this question, if your think I need to change something just let me know and I'll do it.

Trait

Or you could do: a+d, b+e, c+f (so, no need to connect to the webservice and to the db twice). You can call the methods to get your objects first, do the appropriate processing and then copy them locally through transacted inserts (a transaction is needed, so that if one of the insert fails, both are cancelled and you don't get orphaned data). – Tepefy

Answer 1

I decided to come to you because you're the experts

first i am not expert and also i am not knowledgeable you can find more expert people than me but this is my opinion hope to give you some help.

first of all forget to use AsyncTask to download because it must be used for short background jobs not like yours i think the amount of file you want to download is pretty large.(i think so)

check downloadmanager of google to see how it works it may help you.

http://developer.android.com/reference/android/app/DownloadManager.html

http://blog.vogella.com/2011/06/14/android-downloadmanager-example/

if you want to use service use unbound service because you do not want the service to be destroyed by android or user when the user close the apps do you? i think you want to get your data any way.

in order to be efficient i recommend these steps:

a. Read a restful service to retrieve the Customers Data

b. when you get Customer data do :

• create another service or thread to Read a restful service to retrieve the Articles Data

• Parse the response to Json Objects on your old service or thread

now you have 2 services or threads that run concurrently so follow the steps that obvious insert parse and so, on each service or thread.

why do not i combine a and d? because i think user do not like to wait much time behind download progress bar.

in order to insert your data to database use transaction and i recommend you use:

http://greendao-orm.com/

it is more efficient ORM than others for database and you get free from db implementation.

If the activity that started the service is explicitly cancelled or is hidden by another activity, the service has to have a way to let the user know that the process has finished. How could I implement that?

use notification:

http://developer.android.com/training/notify-user/build-notification.html

http://www.tutorialspoint.com/android/android_notifications.htm

http://www.vogella.com/tutorials/AndroidNotifications/article.html

While all this long processing is taking place in the background I'll have to show some sort of indicator (a ProgressDialog) This is the reason I opted for using a Bound Service`

how can I update the UI from an Unbound Service?`

Use a LocalBroadCastManager, or in general BroadCastReciever

Android update activity UI from service

In J2ME I've done something similar, and when I put something like the 6 steps I mentioned above all of them are executed sequentially, that is , one after the other. Is it the same in Android?

this is depends on your steps, if you follow my idea you run concurrently and if you run your idea you will run sequentially.

Good Luck.

Stormystorting

Thank you @Stormystorting I really appreciate you took the time to answer. I've got one question about the way I parse the response to JSON Objects and then insert them to their respective tables. Wouldn't it be better if I insert the objects as I parse them. In the case of the Articles, wouldn't it affect the memory to have 2 or 3 thousand objects? – Trait

when you use downloadmanager you specify a destination folder so all of your json object is now on a DISK not on RAM ,so you can read chunk of data parse it and insert it to your database. how many json objects can you read? i think it is experimental you can check your object size and decide upon it. or you can create sophisticated parallel mechanism for example create multiple threads that read from multiple part of your folder and parse it and then send it to your database. in this scenario your parse time is reduced. – Stormystorting

mmlooloo what kind of thread can I use if first you say first of all forget to use AsyncTask to download and then you also say `when you get Customer data create another service or THREAD to Read a restful service to retrieve the Articles Data. So what kind of thread do you mean? a TimerTask, AsyncTask or a Thread? – Trait

you can use Service or Java Thread. Thread t = new Thread(Runnable r); extend the thread class. there are lots of example around you. or better than any of those is DownloadManager it also handles notification when the downloads are completed. it is designed to download large file. whats your file size? if it is short for example its around 1MB or less use 'Volley' if it is 10MB for example use 'DownloadManager` – Stormystorting

mmlooloo the Article data is around 4 or 5 MB, but the Customers is only 1 MB at most. So tell me,for the scenario I explained in my question I will need 2 DownloadManagers and in their respective "onReceive" insert to the Customers or Articles tables. Is that right?? – Trait

yes, use 2 of them in serial e.g first download 1MB file after you received it start another download and also start parsing the one you received. – Stormystorting

Answer 2

I did something like yours.

In first step I get data from webservice in HTTP GET of POST method using AsyncTask like this:

public class GetService extends AsyncTask<String, String, String> {

private String mRestUrl;
private ServiceCallback mCallback;
private final HttpClient Client = new DefaultHttpClient();
private String Content;
private String url;
private String Error;
private ProgressDialog barProgressDialog;
private ProgressDialog Dialog;

public GetService(String restUrl, ServiceCallback callback) {
    this.mRestUrl = restUrl;
    this.mCallback = callback;
    this.url = restUrl;
    Dialog = new ProgressDialog(AppContext.CurrentContext);
}

@Override
protected void onPreExecute() {
    super.onPreExecute();
}

@Override
protected String doInBackground(String... urls) {

    Content = null;

    BufferedReader reader = null;

    try {

        StringBuilder builder = new StringBuilder();

        HttpClient client = new DefaultHttpClient();

        HttpGet get = new HttpGet(this.url);

        HttpResponse response = client.execute(get);
        int status = response.getStatusLine().getStatusCode();

        if (status == 200) // sucess
        {
            HttpEntity e = response.getEntity();
            // String data = EntityUtils.toString(e);
            InputStream content = e.getContent();
            reader = new BufferedReader(new InputStreamReader(content));

            String line;
            while ((line = reader.readLine()) != null) {
                builder.append(line);
            }

            Content = builder.toString();
        } else if (status == 401) {
            return "-Auth Failed Error Code 400";
        } else {
            return "-Error Code: " + status;
        }

    } catch (Exception ex) {
        Error = ex.getMessage();
    } finally {
        Dialog.dismiss();
        try {
            reader.close();
        }

        catch (Exception ex) {
            ex.printStackTrace();
        }
    }

    return Content;

}

@Override
protected void onPostExecute(String result) {
    try {
        GetService.this.get(20000, TimeUnit.MILLISECONDS);
    } catch (InterruptedException e) {
        e.printStackTrace();
    } catch (ExecutionException e) {
        e.printStackTrace();
    } catch (TimeoutException e) {
        e.printStackTrace();
    }
    mCallback.onTaskComplete(result);
    super.onPostExecute(result);
}
}

and my callback class is:
public abstract class ServiceCallback{

public abstract void onTaskComplete(String result);
}

I call AsyncTask in my code everywhere I want to get data from webservice:

 new GetService(url, new ServiceCallback() {
                public void onTaskComplete(String response) {
                       // Parse Response of WebService
                }
            }).execute();

In second step I Parse response of WebService in onTaskComplete method using json helper Libraries like Gson or Jackson. for example in jackson:

List<YourClass> data = new ObjectMapper()
                .readValue(
                        response,
                        new TypeReference<List<YourClass>>() {
                        });

At the end I store Data in Database. for connecting to DB I Prefer to use GreenDao as my ORM. In this way storin data in DB can be done in one line code like this:

//after converting json to object
YourORMDaoClass.insertOrReplaceInTx(data);

To Use GreenDao ORM this link is very helpful;

Yuille

Thanks!! That sounds really interesting. The only problem is that as a I mentioned, the second table is going to be pretty large, and as far as I know, for long running tasks Services are preferred. Plus, I'd like this long running task to be as independent as possible from the activity that starts it, so that even if the activity is hidden or occidentally cancelled, he/she won't have to start downloading all over again – Trait

Your answer also brings me to another thing that I forget to ask. I've got a separate process that does almost everything you posted. But I need to do the same twice, one for the Customers table and one for the Articles table. Would it be OK/safe to call 2 times the AsyncTask that consumes the webservice?? – Trait

@Trait - Using a service is the right approach, I think. If you want it to be "as independent as possible from the activity that starts it" then I recommend using a started service rather than a bound service. Using two AsyncTasks is probably a good idea, provided there are no interactions between the two database update processes. – Whosoever

@TedHopp Thanks for taking the time to answer. Now regarding the database there's no interaction between the process that inserts into the Customers table and the process that inserts into the Articles table. The thing is , if I put theses 2 processes in 2 different AsyncTasks, how can I control that only when both of them have finished the user is somehow notify about the end of the whole process.This also brings me to another question, how can I update the UI from an Unbound Service? – Trait

@Trait - When the service is asked to update the data bases, it can reset a counter for the number of tasks completed. As each AsyncTask finishes, it calls a method in the Service from onPostExecute to increment the number of completed tasks. When the counter hits 2, the service can broadcast the result (probably using LocalBroadcastManager) and the relevant Activity can receive the broadcast and inform the user. – Whosoever

@TedHopp Sorry to bother you, but I need to ask you something. The documentation states that Starting with HONEYCOMB, tasks are executed on a single thread to avoid common application errors caused by parallel execution. That shouldn't be problem for me now, but in the future if I don't want this tasks to run serially what could I use instead of AsyncTasks? Thanks – Trait

@Trait - You can still use AsyncTask. The next line of the docs explain what to do: "If you truly want parallel execution, you can invoke executeOnExecutor(java.util.concurrent.Executor, Object[]) with THREAD_POOL_EXECUTOR." (That means that you simply need to set up an Executor and call the indicated method instead of calling execute().) – Whosoever

Answer 3

Do you think the order in which I'm executing all the 6 steps of the process described is correct / efficient? If you had to make some changes, what would you change?

It depends. If this data is related and cannot exists without each other then you should change the order like this:

a. Read a restful service to retrieve the Customers Data
b. Parse the response to Json Objects
d. Read a restful service to retrieve the Articles Data
e. Parse the response to Json Objects
c. Insert those customers to the Customers Table
f. Insert those articles to the Articles Table

Steps c and f should be combined in transaction.

Otherwise, the order does not matter. If data is not related then separating these processes and running them in sequence might be a good idea.

---

If the activity that started the service is explicitly cancelled or is hidden by another activity, the service has to have a way to let the user know that the process has finished. How could I implement that?

I suggest to start with implementation of the IntentService class. It handles for you background thread and works like a queue of events where single Intent delivers a data to process.

Actually you could implement one of the patterns presented by Google on one of their IO conferences. I have implemented an option A shown on the video. It works for me really well. The trick is that using ContentProvider from the background automatically updates UI which listen for changes thanks to CursorAdapter.

To update UI progress you can use LocalBroadcastManager or event bus libraries e.g. Otto.

You can also extend your tables and store status and progress, updating tables would automatically update UI as well, just keep in mind that these updates should be rare e.g. control in the service background how often table progress is updated calculating the progress first and checking with service local variable if it's changed.

In case your app is in the background, post status notification. User should be able to navigate back to your app clicking on the notification.

---

Is it possible/ recommended to write to the SQLite DB within the service? Is it the same as when I do so within an activity?

You can do it within the Service. Actually, if you follow the pattern I have mentioned above, you would do it in the Processor tier on the background service thread.

---

In J2ME I've done something similar, and when I put something like the 6 steps I mentioned above all of them are executed sequentially, that is , one after the other. Is it the same in Android?

It's completely up to you how communication with the server will work. If you decide to use IntentService class then it will work in a sequence which is not a bad idea on Android. On the other hand you may extend Service class directly and implement own thread executor with a thread pool. You can also have dedicated IntentService classes for unrelated operations.

---

I also recommend to read lessons:

• Transferring Data Without Draining the Battery
• Transferring Data Using Sync Adapters

If you don't want to play directly with HTTP connection implementation then consider using Retrofit or Volley

If you just need JSON parsers then these 2 are the best:

• GSON
• Jackson

Paucker

Thank you @Loop. I really appreciate you took the time to answer. I've got one question about the way I parse the response to JSON Objects and then insert them to their respective tables. Wouldn't it be better if I insert the objects as I parse them. In the case of the Articles, wouldn't it affect the memory to have 2 or 3 thousand objects? – Trait

If web service provides large responses that may extend memory to the limit then web service API should be improved. I suggest to try it out first as it is and check use of the memory. Too early full optimization is not a good practise because you may end up in the optimization endless loop and never finish your project. – Paucker