TextNow Welcome Screen Comprehension Testing

Evaluating V4 of the mobile welcome experience.

TextNow UXR Co-Op

Overview

TextNow is a free phone service provider with unlimited talk & text, free data for select apps, and nationwide 5G coverage—all in one convenient app. As TextNow’s service offerings, business priorities, and understanding of users have evolved, the onboarding experience has undergone multiple iterations. I led this UXR project in close collaboration with PMs and product designers to evaluate the V4 Welcome page—the first touchpoint users encounter in the onboarding flow. The goal was to assess whether the page clearly communicated TextNow’s value proposition while remaining visually appealing and trustworthy.

Problem

Currently, comprehension and consideration of TextNow as an alternative to traditional phone service is fairly low. By improving comprehension during the new user experience, we should see growth in the number of users who depend on TextNow for their phone service. We have historically run unmoderated and moderated testing on past onboarding experiences. Using these insights to guide our research, we evaluated the latest version of the Welcome page for comprehension as well as efficacy of design and appeal.

The V4 Welcome page…

Takes in consideration multiple rounds of testing
Completely updated phone plans
Updated brand, visuals, tone
Sleeker, more modern, dynamic

Process

Method: unmoderated testing through UserTesting

Participant Profile: 6 prospects who have never heard of TextNow, 3 iOS users and 3 Android users

Working closely with my team, I developed a structured set of test questions grouped by theme. The primary challenge was crafting questions that were clear, comprehensive, and non-leading. We iterated extensively on wording and order, checking in with stakeholders to ensure the questions captured both what users needed to understand and what they naturally took away from the page.

16 Comprehension Questions

Total Possible Score: 26

Participants reviewed the Welcome screen once, then answered uniquely weighted comprehension questions from memory without referring back to the page.

What are the two different ways people can use TextNow? & In your own words, please briefly explain the main differences between "Get wireless phone service” and “Add a free second line”.
Besides your mobile device and the TextNow app, what else do you need to activate wireless service?
What is the total price of getting a TextNow SIM?
What does having a TextNow SIM enable you to do that you wouldn’t be able to do otherwise?
What are the different data options that TextNow offers for wireless users?
Describe your overall wireless package with the Free Flex Plan. What does it include, how much does it cost, etc. Please be specific.
With TextNow’s Free Flex Plan, you get free data for select apps. What types of apps does this include?
What data options does the Free Flex Plan offer?
How much does the On-Demand Data Pass cost?
How much data do you get with On-Demand Data Passes? How long does the data pass last?
In one sentence, what is the main difference between free data for select apps vs. data passes ?
In your own words, how is TextNow able to provide you phone service for free?
In one sentence, describe how the Unlimited Plan differs from the Free Flex Plan.
Describe the different pricing options for TextNow’s Unlimited Plan.
If you decide to end your Unlimited Plan, what would be the state of your service ?
In your own words, please explain what TextNow Rewards are.

4 Trust/Wrap-Up Questions

After our graded questions, we asked more generic questions about the experience as a whole

From a scale of 1 – 5 where 1 = not at all confident and 5 = extremely confident, how confident are you that TextNow is a reliable and credible service based on this page?
Overall, if you had to describe what TextNow is to a friend or family member, what would you say?
Thinking back to this onboarding process, was there anything confusing or surprising about the experience overall? Please be specific.
After having gone through the onboarding experience, what unanswered questions do you have, if any?

3 Design/Brand Perception Questions

Lastly, we wanted to know how the new look and feel resonated with users.

What are 3 words you would use to describe the look and feel of this page? Why did you choose these words?
What 3 words would you use to describe the language and tone used on this page? Why did you choose these words?
What 3 words would you use to describe your impression of TextNow as a company? Why did you choose these words

Findings & Recommendations

Comprehension Scores table

Although the comprehension scores were obscenely low, several contextual factors were critical to interpreting the results…

Participants viewed the page only once and answered questions entirely from memory
The questions were more detailed and cognitively demanding than in previous studies
The goal was not high scores, but identifying where comprehension consistently broke down

We positioned these results as a diagnostic tool rather than a performance metric. Clearly communicating this distinction was essential to prevent misinterpretation and concern from stakeholders.

Each key finding was mapped directly to a recommendation. Several high-impact opportunities emerged around clarifying plan differentiation, data options, and the mechanics of “free” service.

This study surfaced several clear next steps:

Continue close collaboration between Product, Design, and UXR to ensure insights are prioritized appropriately
Test the next iteration of the Welcome screen in-app to validate improvements in a real onboarding context
Address gaps across the rest of the onboarding flow to ensure a cohesive end-to-end experience
Regularly update the Welcome screen as new features launch (eSIM, loyalty programs, etc.)

Retrospective

This project reinforced how much nuance goes into crafting effective research questions. Small changes in wording or order can significantly influence participant understanding and response quality. I also deepened my skills in cross-functional collaboration, experiencing how ongoing communication with PMs and designers directly improves both the efficiency of a project and the quality of its outcomes.

Rogers Beyond The Seat ›

IF YOU LIKE WHAT YOU SEE,