Need some clarification about beta/alpha testing on the developer console

Question

Background

The Android developer console has 3 tabs for publishing the app's apk file: alpha, beta and production, as shown here:

as I recall from one of Google IO lectures, one cool way to check how good is your app before making a 100% scale publishing, is to allow only a percentage of the users to download the app first. I think it's called "staged rollouts" , because you can rollout the publishing in case it had too many problems to be published to all.

My question

1. What is exactly the difference between them, especially between Alpha and Beta?

2. Only the production stage is available for people on the play store, right?

3. which one/s allow to publish only to specific people/percentage , and in which way do you do it?

4. which stage allows in-app billing, at least for testing ? I don't get why can't i test it out even before uploading the app.

5. in the percentage method, if I publish a new app version using the same way, will it first update for the people who were lucky enough to install the previous version?

Wat

As per the standards the main difference between Alpha and Beta testing is: Alpha testing is done by developers without any involvement of users, in the development environment before marketing the app. However Beta testing is done only by users in real environment. – Ribwort

Answer 1

See there are Four types of releases on play console for android developers;

1. Internal testing - It means you wanna share your app among the decided members. It means you have to put their Gmail into the testers list. This is usually for employees.

2. Alpha testing - Same as internal testing, but this time you can send your app to your friends, families, or on your smartphones.

3. Beta testing - This is a serious matter, this is public testing. But people cannot give reviews to your app. You can also limit how many users can install the app.

4. Production - This is the last release, now everybody can install your app, can give reviews, can report to google play if something bad is there. One more thing, now your app appears in play store search listing.

Now there is one more thing, open testing, closed testing, pausing tracks and stage rollouts, don't get confused by these terms.

1. Open testing - It means that you don't need to invite them, like no need to take their Gmail and add it to the user list. (Beta testing)

2. Closed testing - It means that you need to specifically add the user's Gmail into the list. (Alpha testing)

3. Staged rollout - This means we can provide the update to some percentage of users we currently have. Suppose 100 users are there and you rollout for 40% then only 40 users can update the app and after you confirm the app is working perfectly (using crash reports or reviews) then make it available for everyone.

4. Pause Track - See each testing has a track in play console. So if you click the pause track in beta testing then it will pause beta testing [changes occurs in few hours]. Same in internal and alpha. We usually pause test tracks when we release the app in production [usually I do].

Your app will not appear in search results until it is in production.

Viscountess

This sums things very nicely. Thanks. But the percentage one is only for release (production) , right? – Wat

yes, it is called staged rollout. Here is a link for this - support.google.com/googleplay/android-developer/answer/… – Viscountess

Sorry I keep forgetting about how it's called. "stage" has a context of theater or of video games :) – Wat

Sorry to deviate from the OP but since you have a decent grasp of the testing types/groups ... do you know how long it take for a updated build/release to reach "Internal Testers"? And how/where can I check the progress of this if it is indeed stuck in some kind of waiting/review pipeline? – Underwater

@Underwater There is no review of app in Internal testing, you can get the link from Google play console. Share that link to your testers after adding their account in the list of internal testers. Your app needs review from google team if it is published (beta/production) to people over the world (means anyone can download the app). – Viscountess

Thank you very much, @Vijay. That's what I thought but for some reason my internal testers can't see the latest APK and it is driving me insane! I will keep trying to figure it out. Thank you again for confirming that. 🙏🏻 – Underwater

Answer 2

To answer your questions:

1. What is exactly the difference between them, especially between Alpha and Beta?

There isn't much difference between the two aside from the fact that you just start with a small number of testers for alpha testing and switch to a bigger group for beta

2. Only the production stage is available for people on the play store, right?

By default, only production is available on the Play Store. However, you can now add an option in your Play Store page for users to opt into an open Beta program. Link

3. which one/s allow to publish only to specific people/percentage , and in which way do you do it?

You can do that for both. To do alpha,beta testing you need to send invites to people on their google+ accounts so that they can access your app and be able to download it. The invites is usually in the form of a link directing them to your app on the play store which is only visible to them after they accept the invite

4. which stage allows in-app billing, at least for testing ? I don't get why can't i test it out even before uploading the app.

You can do in-app billing for both alpha,beta testing. Check the link: http://developer.android.com/google/play/billing/billing_testing.html

5. in the percentage method, if I publish a new app version using the same way, will it first update for the people who were lucky enough to install the previous version?

As far as my experience goes, I have seen people who were testers get the update for the app first than everyone else. But I am not so sure about how does it work exactly.

Elkins

1.so we decide what's each? 3. why can't i set it by emails instead of google+ accounts? and how do i do it? all i've found is to use groups of people (which i don't know how to do either). 4. how do i reset the purchases for the testers, so that the could check the app again? should i put a button to reset them using the "consumption" of purchases, or is there an easier way? 2.+5. ok – Wat

yes you can either do alpha or beta doesn't matter. you actually add the testers using their email, but all i am saying is that they need a google+ account as well.. check out the link for the purchases that i posted in the answer. – Elkins

I see. can you please try to answer the other questions? – Wat

i didnt understand the last part to it.. 2.+5. ok.. what does that mean – Elkins

2+5 is that i say thanks about number 2 and number 5. what i ask now is why can't I use the emails of the testers instead of google+ accounts and how to use them (since all i've found is adding groups of people, which i also don't know how to do). I also asked how can I reset the purchases, so that the testers can test again the purchase process. – Wat

Please check my last comment. I answered the query – Elkins

where? i don't see anything related to resetting or adding people to beta testing... – Wat

developer.android.com/google/play/billing/billing_testing.html.. the link answers your query – Elkins

that's the same link you've already written about. anyway, i've found a way to create group here: groups.google.com/forum , and then I chose there to add more people, by choosing "members"->"Directly add members" . i also found out the apk must be signed in order to test it out. anyway, i still need to know how to reset the purchases. – Wat

I also found out that even though I have the test account showing the dialog of test-purchase, I still need to put credit card information. – Wat

OP, I suggest accepting this as the answer to the question. Seems to be a solid response. – Plattdeutsch

What do you define a bigger group as and when should I switch to BETA. What are the benefits of this switch? Thanks so much, please let me know. – Appraise

@Elkins I have a feeling this answer is not completely valid anymore. It seems you don't need to send invite to peoples' e-mail address to alpha or beta testing. Just depends on the method you chose - close testing or open testing - in android console 'thing'. – Middlebrooks

@Elkins Can we release two versions of the app - like for closed beta use version A and for internal tests use version B? By version, I mean I want two apps one pointing to staging environment and another to production. If we cant do that as I mentioned above, what is the other best approach to do that? thanks – Dyarchy

Open Beta also go to the Play Store. "If you set up an open test, users can find your test app on Google Play." support.google.com/googleplay/android-developer/answer/… – Oblique

Answer 3

Difference between Alpha and Beta testing (Software/Apps Testing)

Alpha Testing is conducted within the organization and tested by an individual developer or a team of developers or testers. This testing is closed for public.

Beta testing is conducted by the end users who are not programmers, software engineers or testers. This testing may be open for public.

Open or Closed Testing

Closed Alpha Testing: This testing is conducted within the organization and restricted by email or group invitations only. Those who are added onto the list of alpha testers on Google Play Store have access to test app.

Open Alpha Testing: Those who have Opt-in link of app have access to test the app. This testing is not restricted by email or group invitations. You can limit the number of testers on Google Play Store.

Closed Beta Testing: This testing is conducted outside the organization and restricted by email or group invitations. Those who are added onto the list of beta testers on Google Play Store have access to test app.

Open Beta Testing: This testing is conducted outside the organization and it is not restricted by email or group invitations. This testing is conducted in real time as the app will be made open for public to install. You can limit the number of testers on Google Play Store.

Generally, alpha testing is first conducted by developers or testers within the organization. Beta testing comes after and is conducted by non-technical or end users outside the organisation. Finally, after completing the testing, the app goes to production.

In app purchase

Licensed users may purchase your in-app products without any resulting charges to the user. Test purchases can be only used in both alpha and beta releases.

To add user license: Google play Console -> Settings -> Developer Account -> Account details -> License Testing

Learn more about testing

Equi

Currently no closed beta testing available in android console – Giddy

Answer 4

@user2511882's answer is great, but there are more information to add.
First of all, alpha and beta channels are designed to test multiple version of app. You can test only one version (identified by versionCode) on each channel, so alpha and beta channels provide two test tracks.

However they have a little difference on test type. You can run closed alpha and open beta test simultaneously, but it's not possible to run two open alpha/beta test nor open alpha and closed beta test.

Another difference is that alpha test APK's version code should be higher than beta test one. If you upload higher version of APK on beta test channel, alpha test will automatically be closed. (This principle applies for production APK too. APK version code should be 'Production < Beta < Alpha'.)

For more details, check this help.

Salcido

Just to clarify, those apps with multiple apk support can test both versions for each architecture (ARM and x86) at the same time on the same channel (alpha or beta) even though they have different versionCodes – Depth