Augmented Reality-Based Digital Twin
Role
AR/MR Researcher and Developer
Duration
September 2023 - June 2024
AR Design and Development Tools
Figma, Lucidchart, Unity Engine, Microsoft Mixed Reality Toolkit, PUN2, Git
Skills
Problem Framing, Problem-Solving, Iterative Design, UX Design for AR, Data Visualization in AR, Digital Prototyping, Competitive Analysis, System Design, Programming in C#
Team
Yelnaz Rysbek, Kemel Shomenov, Nursultan Jyeniskhan, Dr. Essam Shehab, Dr. Md. Hazrat Ali
How can HoloLens2 be used to monitor and control the additive manufacturing process?
Overview
As a designer and developer, I led the creation of an exploratory Augmented Reality (AR) system to enhance the monitoring and control of additive manufacturing processes. This project addresses limitations in traditional 3D printing interfaces, particularly the lack of real-time visibility and remote control capabilities. Focused on Fused Deposition Modeling (FDM) printers, our proof-of-concept solution leverages the Microsoft HoloLens 2 to create an immersive, interactive experience, enabling users to oversee and adjust the printing process remotely.
As a result of the work, a conference paper titled “Development of Augmented Reality System for Additive Manufacturing“ was published.
Research
Methods
Literature Review
Stakeholder Interviews
Competitive Analysis
Key Insights
User Needs: Real-time data visualization, minimal latency, secure access, and simplified remote control.
Existing Solutions: While applications such as OctoPrint offer web-based monitoring, AR integration remains limited.
Competitive Analysis: OctoPrint, MagicHand, and MIT Media Lab AR for AM offer basic 3D printer monitoring, IoT control via AR, and AM visualization with tablets, respectively, but lack immersive interaction, specific AM integration, or broad device compatibility.
Key Functionality Requirements
Authentication: Restricted access to authorized users.
Real-Time Monitoring: Displays parameters like temperature, time, and filament usage.
Simulation: Visualizes the AM process in real time.
Control Interface: Enables actions like start, pause, and stop remotely.
Development
System Architecture Design
A four-layer architecture was designed, including the physical, data acquisition, data processing, and application layers.
Network Integration
Data is gathered from the FDM printer using OctoPrint and Raspberry Pi, transferred to a Unity desktop server, and finally sent to the HoloLens 2 through Photon Unity Networking (PUN2) for real-time synchronization.
User Interface Design
The interface was developed using the Mixed Reality Toolkit (MRTK) and Microsoft’s design language for AR, prioritizing simplicity, visibility, and learnability.
Features: Digital twin visualization of the printer, interactive control buttons, and parameter display.
Final Result
The resulting AR system facilitates seamless remote monitoring and control of AM, addressing the critical industry needs for improved visibility and interaction in AM environments. Key functionalities, such as real-time simulation and remote control, are now accessible through the HoloLens 2 interface. The system also has the potential for scalability across different AM devices and industries, paving the way for a future digital twin model. The transmission time from OctoPrint to the Unity Server is 0.01569 ms with a standard deviation of 0.006199 ms, while the Unity Server to HoloLens 2 transmission time is 936 ms, which ensures seamless interaction with the physical printer through AR glasses
General Control Menu
Temperature Visualization Menu