Journal of AI, Data Science and Cyber Systems

INTRODUCTION

With technological advancement, screen interfaces have become the central medium of contemporary visual culture. However, the forms and functions are rapidly changing as user interaction is transitioning from the flat displays toward spatial and immersive mediums like mixed and augmented reality [1]. This “post‑screen” practices noticeably shift visual experience of merely looking at images within the boundary to an embodied, multisensory practice of seeing which is accompanied by touching, and moving within the image [2,3]. Concurrently, AI‑driven visuals and algorithm-based curation increasingly shape screen content that influence perception, attention, and visual culture at scale [4].

The studies conducted by Human Computer Interaction (HCI) and Mixed Reality (MR) have developed powerful techniques to track gestures, gaze, and motion. However, limited studies have been performed that connect technical developments to semiotics and visual studies. Semiotics is a study to examine how images construct meanings [5]. and how ways of seeing are cultural constructions [6]. Particularly, we lack conceptual tools for understanding how non‑verbal modalities (e.g., gesture, embodied navigation) function as visual grammar within mixed reality [7].

Therefore, this research addresses the gap by proposing a heuristic and semiotic framework for visual perception in post‑screen cultures by reviewing literature and conducting a qualitative analysis. It examines emerging technologies transforming urban spatial experience through AR/VR visualizations and how AI generated visuals shapes perception through algorithmic bias [4].

Research questions: With the research questions below, the study asks how mixed reality systems deliver visual experience and how future screen interactions can be culturally meaningful, accessible, and ethically transparent.

How mixed reality interfaces construct new visual perception of the post screen culture through multisensory interaction?

How Mixed Reality (AR/VR) interfaces function similar to screens interface and not just an embedded perceptual media platform that shape the act of seeing?

LITERATURE REVIEW

Mixed Reality, Visions of the City and Spatial Visual Experience

Mixed Reality (MR), Augmented Reality (AR), and Virtual Reality (VR) transform seeing from a distant, optical activity into an embodied practice of moving, living, and acting within the projected digital‑physical environments [8].MR interfaces depend on non‑verbal modalities like gesture, gaze, body posture, and locomotion which turn user’s body into an input device [9]. In MR, the depth cues, field of view, latency, and spatial audio impact on the sense of presence which shapes user interpretation of visual information [10]. However, cultural and artistic MR projects portray that such environments also act as sites of visual culture, where images are no longer bound to frames [11]. The visual appears as surfaces and ambient fields integrated into architecture and urban space [12,13].

Despite being a post-screen interface of visual media, visual studies has engaged with MR as a visual medium partially. It is often treated as a technical extension rather than a transformation of visual grammar itself. A few researchers argue that MR presents a “hybrid form of looking,” where users simultaneously attend to physical surroundings and virtual space. Within this hybrid paradigm, scholars highlight the role of AR and MR in urban visual representation to map cities, narrate, and imagine through visual simulations [14]. According to Graham, Zook, and Boulton (2013) augmented urban spaces are the digital and algorithmic representations of the Visions of the City. MR applications for architecture, heritage frequently use smart surfaces, AR lenses, and/or large projections to visualize proposed buildings, historical reconstructions, or other environmental scenarios [15]. These visualizations serve act as technical tools as well as cultural data to frame an understanding of the city and its futures appearances [16].

Additionally, cultural and critical urban studies indicate that MR projected “visions of the city” are not always neutral. They often showcase certain narratives like creative city, smart city, heritage city that privilege particular user groups. For example, heritage‑oriented MR projection may emphasize monumental histories while sidelining informal or marginalized spaces [17]. Similarly, smart‑city dashboards systems often presume a normative and able‑bodied, and technologically equipped user [18]. This often leaves the accessibility needs under‑addressed [19].

AI Visuals, Algorithmic Bias, and Influence on Act of Looking

Its foundation of Human-Centered AI is that there should not be a trade-off between automation and human control. Shneiderman (2020) argues that well-designed AI systems can be automated leaving control in human hands. For example, smartphone cameras have auto-focus and exposure while offering manual access with user’s control . Shneiderman (2022) proposes that AI should be designed to amplify human intelligence, not replace it. This indicates that AI systems are not just computational benchmarks to detect accuracy but also contribute to human capability and wellbeing.

Wearables and augmented reality extend digital twin research as it adds embodied interaction in workplace simulation. VR and AR have been used in industry settings to support operators in training, robot programming, and ergonomic evaluation.This is crucial since workers do not experience collaborative AI only through what they see. They also use movement, gestures, space, and sensor feedback to decide whether to trust and follow the AI. Elmqvist et al. (2025) advocate for “Participatory AI”, building on the tradition of Participatory Design in Scandinavia from the 1970s. It was originally a reaction to industrial automation but now extended to AI design. The authors offer a case study where workers in quality assurance at a manufacturing company participated in the design process of an augmented reality system based on AI that would guide measurement procedures at work. The case clearly shows that participatory techniques that helped workers have their own say during previous waves of automation are relevant now. Perhaps it is making sure that AI-enabled fieldwork tools are made for the users, not just for them.

Visual Grammar and Ways of Looking in Screen Cultures

For a long time, visual studies have examined how images create perception through framing, composition, and spectatorship. It shows, “ways of seeing” are not neutral and they are constructed historically and culturally. With the rise of interactive and networked screens, users don’t just look, they rather operate, navigate, scroll, and select. This action blends the act of looking with decision.

Interaction design, interface layouts, icons, and visual cues guide user attention, interpretation, and response.“Micro‑aesthetics,” in interfaces such as animation and visual feedback structure temporal flow of looking and acting that subtly shape user perception.However, most of the work still assumes 2D based screen interfaces focus on visual composition more than embodied, multisensory engagement. Recent work on visual platforms suggests that attention is now distributed across multiple devices and windows. This forms a networked and combined gaze where users constantly reallocate focus.Consequently, this has implications for visual accessibility. Because if visual grammar is optimized for normative vision, speed, multitasking and attention, users with non-normative profiles may be marginalized by design.These insights present the necessity to rethink the visual grammar with a heuristic approach of post‑screen cultures that will particularly consider embodiment, accessibility, and the cultural aspects.

METHODOLOGY

The research plans to employ qualitative and design‑oriented methodology to critically analyze user studies and explore prototyping. This approach is structured in three phases to address the research questions.

Phase 1: Semiotic Analysis of Mixed Reality Cases

First, an existing MR/AR application in cultural, educational, or urban contexts will be selected (example: museum AR guides, urban visualization apps). These cases will be analyzed using visual and semiotic methodologies to identify how gestures, gaze and spatial interface elements encode visual grammar. It will also examine user attention, movement, and interpretation. Analysis will be categorized into sections including spatial framing, interaction cues, sensor, and visual narrative structure of space. This phase will generate an initial conceptual model of post‑screen visual in practical systems.

Phase 2: Studying of Embodied Perception in Mixed Reality

Based on the first phase, a qualitative study will be conducted with participants experiencing MR prototypes in a lab or controlled setting. Data collection will combine:

Interviews focusing on user perception, attention, and sense‑making.

Recording observation and interaction including their gesture, gaze direction and navigation patterns.

Questionnaires on users’ presence, comfort, preference and perception.

Data will be analyzed thematically to understand how users experience visual grammar, how they interpret system cues, and how predictive/adaptive behaviors affect their sense of control and understanding.

Phase 3: Heuristic Framework and Design Prototyping

Insights from the previous phases will be merged to develop a heuristic framework for Human-Computer Interaction. This will test the prototype through user evaluation in a controlled or semi-controlled setting. Participants will be asked to use the prototype to compare the work settings. Data collection may include task-based observation, interviews, and short questionnaires. It will benefit to understand the participants’ confidence, trust, perceived independence, and anxiety. The analysis will be thematic and interpretive as participants will be asked to provide descriptive comparisons across the two conditions. All participant fieldworkers will participate voluntarily with informed consent, and assurance that their participation, critique, and feedback regarding the AI prototype will have no bearing on their employment status. Special precautions will be taken to guarantee psychological safety during workshops and interviews to allow fieldworkers to raise any concerns that they may have regarding the use of AI in their place of work.

CONCLUSION

This research proposal establishes a heuristic and semiotic framework to offer design principles that balance technological innovation with cultural aspects. The idea is to understand user perception through emerging interfaces. Moreover, the study highlights how algorithmic input and human influence in shaping visual culture. Furthermore, future screen interactions must prioritize inclusive design to include a diverse user group through inclusive design. The findings of this research will advance visual studies, HCI, and media theory by providing actionable guidelines for developers and policy makers.