First-time Configuration
When installing and opening Talking Eyes for the first time, the app displays an 8-step onboarding guide ("Welcome to Talking Eyes"). Please complete this setup with the assistance of a family member or caregiver.
If you choose not to enable AI suggestions in Step 3 by selecting "Not Now", the app automatically skips user profile, audio, and location permission steps, proceeding to Step 7 (Expression Content), then Step 8 (Device Placement).
The 8-Step Onboarding Flow
Step 1: Express Needs with Your Gaze (Overview)
- Introduces the core concepts of quadrant communication: high-frequency expressions are placed in large, clear screen zones, selected progressively via gaze dwell or touch, assembled into sentences, and spoken aloud.
- Direct touchscreen tapping is fully supported, allowing caregivers to demonstrate, assist, or operate without camera tracking.
- Tap "Next" in the bottom-right corner to proceed.
Step 2: Make Your Gaze an Input Method (Camera Permission)
- Explains that eye-tracking interaction relies on the front camera and ARKit face tracking (on devices supporting TrueDepth). The device processes facial meshes, eye positions, and gaze vectors in real-time on-device.
- Privacy Protection: Facial and gaze data exist solely in volatile device memory during active usage. They are never written to disk, never uploaded to servers, and never shared with third parties.
- Tapping "Next" triggers the system permission dialog:
- Tap "Allow" to enable camera eye tracking;
- If denied or if the device lacks ARKit face tracking, tap "Next" to continue using Touch Mode.
Step 3: Enable AI Suggestions? (Data Sharing Consent)
- Details the role of AI suggestions: combining assembled sentence text, user preferences, daily keywords, and environmental summaries to generate contextual candidates, minimizing gaze fatigue and screen switching.
- Explicitly discloses the third-party AI provider (Volcano Engine / Volcengine Ark - Beijing Volcano Engine Technology Co., Ltd.) and the exact data fields transmitted (current sentence, user settings, authorized location and weather summaries, speech transcripts; never camera feeds, facial geometry, or gaze points).
- Two Choices:
- Agree to Data Sharing and Enable: Tapping "Next" advances to Step 4 to configure user profile and caregiving contexts.
- Not Now: Tapping "Next" proceeds to Step 7 (Expression Content), then Step 8 (Device Placement) (skipping Steps 4, 5, and 6). The fourth quadrant on the home screen defaults to on-device "Recents", and all core communication features remain fully accessible.
Step 4: Configure User Profile (Only When AI is Enabled)
- Tap "Configure User Profile" to open the form and configure the user's background and nursing limits:
- Age group and gender;
- Condition or diagnostic tags (e.g., stroke, ALS, TBI);
- Intake route (oral/tube), fluid restrictions, respiratory support, mobility/positioning limits, hemiplegia side, and other clinical tags;
- Caregiver address names, regular medications, favorite foods, habits, primary environment (home/hospital/nursing home), and custom keywords.
- This information is stored locally on the device to assist AI in generating realistic, clinically safe suggestions (e.g., forbidding drink recommendations under fluid restrictions).
- Save the form and tap "Next".
Step 5: Understand Ambient Sounds (Microphone & Speech Recognition, Only When AI is Enabled)
- Optional feature: With permission, the app transcribes caregiver speech and classifies ambient room sounds on-device to provide real-time conversational context for AI predictions.
- Privacy Protection: Raw audio recordings are never saved locally or uploaded to servers.
- Tapping "Next" requests Microphone and Speech Recognition permissions sequentially. These may be granted or declined without affecting quadrant selection or speech synthesis.
Step 6: Express Within Context (Location Permission, Only When AI is Enabled)
- Optional feature: Location helps determine general venue settings and fetches current local weather via Apple WeatherKit to enrich AI context.
- Privacy Protection: No travel tracks or historical movement logs are recorded.
- Tapping "Next" requests Location permission. This can be granted or declined without impacting core communication.
Step 7: Personalize Expression Content
- Both home-screen expression libraries can be edited: categorized expressions use categories → groups → phrases (up to 64 phrases); frequent phrases use categories → phrases (up to 16).
- Customize names, icons, and phrases under "Settings → Expression Content" after onboarding. Deleting an item leaves its position empty; other items do not move. Individual positions can be restored to defaults.
- Tap "Next" to continue to device placement.
Step 8: Secure Device, Keep Gaze Stable (Device Placement Assistant)
- Guides securing the iPad in landscape orientation on a stable stand directly facing the user, avoiding glare and wobbling.
- If Camera permission was granted in Step 2, a "Go to Placement Assistant" button is available:
- Opens a full-screen live view;
- Verifies that the forehead, chin, and facial contours fit within the dashed guides;
- Confirms the face is centered, maintaining an optimal distance of 40–70 cm.
- Device placement verification is optional and can be accessed at any time in Settings.
- When ready, tap "Finish Setup" in the bottom-right corner.
Post-Onboarding Main Flow
Tapping "Finish Setup" dismisses the onboarding guide and enters the main application flow:
1. Gaze Calibration (Camera Authorized)
If the device supports face tracking and Camera permission is granted, the app automatically begins detection:
- Finding Face: A dimmed full-screen camera preview displays "Looking for face";
- Preparing Calibration: Once the face is detected, a prompt announces calibration is starting;
- 6-Point Sampling: The screen sequentially presents 6 calibration targets (Center → Top-Left → Top-Right → Bottom-Right → Bottom-Left → Center). Each point pauses 1 second for gaze acquisition followed by 1.5 seconds of sample collection. If lighting becomes insufficient, calibration pauses until lighting normalizes;
- Once completed, the app transitions immediately to the four-quadrant home screen, ready for gaze interaction.
2. Touch Mode (Camera Denied or Device Unsupported)
If the device lacks face tracking or camera access is denied, completing the onboarding tutorial transitions directly into the four-quadrant home screen. Users communicate by tapping targets directly, with no calibration required.
Permissions Checklist & Summary
| Permission | Onboarding Step | Purpose | If Denied |
|---|---|---|---|
| Camera | Step 2 | Gaze tracking, blink detection, 6-point calibration & placement assistant | Touch Mode operates normally |
| Microphone & Speech Recognition | Step 5 | Optional on-device caregiver speech transcription and ambient sound detection | Does not affect quadrant selection, Pinyin, or speech |
| Location | Step 6 | Reverse geocoding and WeatherKit weather for AI suggestions | Does not affect core communication |
Re-configuring & Replaying Tutorials
- Replay Onboarding Guide: Go to "Settings → Tutorials & Support" and tap "Watch Tutorial" to replay the 8-step guide at any time (in replay mode, tap "Close" in the top-right corner to exit).
- Re-calibrate Gaze: Tap "Calibrate" in the top-left corner of the home screen, or select "Re-calibrate Gaze Now" in "Settings → Advanced".
- Edit Profile & Permissions: Update information at any time in "Settings → User Profile", "Settings → General", and "Settings → Device Status".
For next steps, see Interface Overview and Quadrant Navigation.