1. Measuring Network Quality:
Strengths and Weaknesses of different Evaluation Methods
Astrid Wallisch| MI | Market Research & User Experience
Harald Schwab| MI | Market Research & User Experience
April 2009 General Online Research 09 in Vienna]

2. Table of Content 1. Survey Method 2. Demographics 3. Results 4.
Management Summary

3. Survey Method and Database #1
Goals of this Survey
Evaluation of different survey methods (SMS, WAP, CAWI and CATI) with the objective of finding an appropriate method for measuring network quality continuously
The study design allows a comparison of the methods with regard to
Response rates
Answer patterns
Demography of the sample
Satisfaction with the survey method
Satisfaction of mobilkom austria customers with network quality in the categories data and voice

4. Survey Method and Database #2
Online-Survey
Field time: to
Sample: n=634 (before reminder: n=422);
Response rate: 15,6% (before reminder: 10,4%)
Duration of survey: 02:55 minutes (median)
WAP-Survey
Field time: to
Sample: n=451;
Response rate: 2,5%
Duration of survey: 03:46 minutes (median)
Information about the cited benchmarks:
Customer Satisfaction Survey 2008: Cati, interviews with residential customers, June 2008
Mobile Broadband Profiling 2008: Cati, 300 interviews with residential customers, May2008
A1.net Survey 2008: CAWI, 5500 A1 Broadband customers, March2008
SMS-Survey
Field time: to
Sample: n=485;
Response rate: 3,0%
Duration of survey: 01:59 minutes (median)

5. Survey Method and Database #3 [Matrix of Methods]
Three independet samples of the same size are taken from the A1-customer base („Only-Voice“, „Only-Data“ and „Voice- & Data“)
A1 Mobile Customers
(voice telephony)
A1 Customers with Datacard and Mobile
A1 Datacard Customers
(mobile broadband)
SMS Survey
We asked for
satisfaction…
with netquality of voice telephony
with the survey method
satisfacion…
with netquality of mobile broadband
response rate too low
[SMS are displayed in the access software of the datacard but users hardly ever read or reply to them]
WAP (MMR) Survey
Not possible /
doesn‘t make sense
Online Survey
[already covered by other studies]
We asked for satisfaction…
Cati Survey
used as a benchmark (and based on already conducted studies))

6. Realisation of SMS and WAP-Survey [Screenshots of the Network Quality Survey]
MMR Flow
SMS Flow
Basics on customer equipment:
SMS-Usage: 76% of the mobile phone user SEND, and 87% RECEIVE SMS (among the up to 19-year-old almost 100%)
MMS-Usage: 56% of mobile phone user own a MMS-capable mobile phone (among the up to 19-year-old almost 80%)
Source: AIMq408, mobile phone owners representative for austrian population aged 14+

7. The SMS-Survey is rated best among the three methods.
Question: I like this kind of survey.
The SMS-Survey is rated best among the three methods. MW
1,96
1,94
1,84
2,11
in %; n(Total)=1.570; n(Web)=634, n(SMS)=485, n(WAP)=451; Basis: all respondents;

8. Demographics: Age
Compared to the online-survey the SMS- as well as the WAP-survey reach younger target groups.
in %; Angaben zum Alter aus Adressmaterial gezogen;

9. Demographics: Province
Better response in Styria and Lower Austria for WAP-surveys (due to A1 customer base).
in %; Angaben zum Bundesland aus Adressmaterial gezogen;

10. Demographics: Education
Better reception of WAP-surveys among people up to educational level „apprenticeship“.
Information: the question about education was not asked in the SMS survey
in %; Basis: all respondents (question was not part of the SMS-questionnaire);

11. Demographics: Occupation
More pupils, partly students and blue-collar workers among WAP survey respondents.
Information: the question about occupation was not asked in the SMS survey.
in %; Basis: all respondents (question was not part of the SMS-questionnaire);

12. Question: In which places do you mainly use the A1 mobile broadband
Except of WAP outdoor usage the survey results show little differences between the survey methods regarding places of use.
* Benchmark: MBB Profiling Private/SOHO 2008;
in %; n(Total)=834; n(Web)=634, n(WAP)=200; Basis: respondents who own an A1-Data-SIM (Question was not asked in the SMS-questionaire)

13. Question: In which places do you mainly use the A1 mobile broadband
* Benchmark: MBB Profiling Private/SOHO 2008;
in %; n(Total)=834; n(Web)=634, n(WAP)=200; Basis: respondents who own an A1-Data-SIM (Question was not asked in the SMS-questionaire)

14. Question: How often do you use your mobile datacard/your mobile USB-modem from A1?
There are only small differences between the survey methods concerning the frequency of use.
in %; n(Total)=834; n(Web)=634, n(WAP)=200; Basis: respondents, who own an A1-Data-SIM (Question was not asked in the SMS-questionnaire) );

15. Management Summary Differences between the survey methods?
Regarding response rates the online-survey was the most successful method.
The SMS- and the WAP-survey reached younger target groups than the online-survey.
Multiple response questions brought similar results, regardless of the survey method.
The respondents of the WAP-Survey were slightly more satisfied with the network quality (voice and data) than the respondents of online- and SMS-survey (and WAP results were close to the CATI-results).
Respondents of the online-survey have a higher degree of education and occupation than the respondents of the WAP-survey.
From the three conducted survey methods, the SMS-survey was rated best.

16. Learnings from the comparison of methods #1
Based on the comparison of survey methods, we can draw the following learnings for mobilkom austria
SMS Surveys
MMR Surveys
Perceived Advantages
[beside this method comparison, SMS-surveys are already used for customer satisfaction surveys each time the customer leaves an A1 shop – with very good response rate and feedback]
Very „fast“ method – survey mostly filled out the very same day
Possibile to reach a high number of customers
 see SMS, plus additional:
Even multimedia content can be displayed
Open questions are well accepted
Perceived Disadvantages
[for example, MMR surveys were applied for studies dealing with Vodafone live! and „Mobile Gaming“ ]
Lower response-rates than with „classic“ methods like CATI and CAWI (despite use of incentives)
Questionnaire: Limitations concerning number and complexity of questions.
SMS could be perceived as spam
SMS has to be free of charge for the respondent.
WAP-Push (SMS with link to WAP-survey) is partially unknown
Mobile devices sometimes store received WAP-Push unfavorable, so respondents might have problems finding the WAP-Push.
Data-traffic has to be free of charge for the respondent

17. Learnings from the comparison of methods #2
Based on the comparison of survey methods, we can draw the following learnings for mobilkom austria
SMS Surveys
MMR Surveys
(possible) future fields of implementation
Research on network quality in addition to the existing method portfolio, if results are needed very fast and for a reduced set of items).
Surveys that are linked to special incidents (such as the customer satisfaction survey after a visit in an A1 shop)
Studies among young target groups (passion for SMS and internet access via mobile phone)
Location based surveys (with prior approval of the customer)

18. Contact: Astrid Wallisch Harald Schwab Market Intelligence
Head of Market Research &
User Experience
mobilkom austria AG Obere Donaustraße 33 A-1020 Wien Mobil: Fax:
Harald Schwab
Market Intelligence
Market Research &
User Experience
mobilkom austria AG Obere Donaustraße 33 A-1020 Wien Mobil: Fax: