GIBSON in Marunouchi

GIBSON, a project to create new communication experiences that fuse real and cyber space, is being conducted in collaboration with Hakuhodo DY Holdings, Inc. The demonstration test was conducted in collaboration with Mitsubishi Estate Co.

GIBSON" is a "cyber space" built using a "digital twin," a 3D copy of the real space (physical space), which enables VR users in remote locations (experience space) and AR users in the real space (Marunouchi tour area) to log in and experience communication as if they are sharing the same space and place. The system enables a communication experience as if they are sharing the same space.

In recent years, there has been an increasing number of attempts to construct VR spaces using digital copies (digital twins) of actual cities. However, even if a 3D model can reproduce the shape of a city, the real space keeps changing from moment to moment, and its appearance changes drastically depending on the weather and time of day, so the real space and the cyber space that reproduces it become very different as time goes by. In a system like GIBSON, which enables communication between remote locations by aligning the environmental context observed by AR and VR users as much as possible, this discrepancy becomes an obstacle to communication and a sense of immersion, and the VR system user's sense of being in the location (locality) (It is difficult for the user of a VR system to feel as if he or she is in the same place.

In response to these issues, we thought that by sequentially distributing the real-time situation of the real space to the VR space side using multiple expression formats such as RGB images, 360-degree images, and 3D images by LiDAR, and by incorporating technologies such as spatial acoustics, VR users participating from remote locations can experience the ever-changing situation of the city with a sense of realism. We thought that by incorporating technologies such as spatial sound into the real-time situation being delivered sequentially to the VR space, VR users from remote locations would be able to experience the ever-changing situation of the city with a sense of presence. In addition, in order to combine and deliver a variety of video delivery technologies, we utilized local 5G lines in this experiment and evaluated their contribution to improving the sense of realism.

Two pairs of subjects (11 pairs in total) were given a tour of the Marunouchi area using GIBSON's system, followed by a questionnaire and interviews about the experience. MESON and Hakuhodo members on the AR side, who were on site in the Marunouchi area, used AR glasses: NrealLight and acted as guides, accessing the VR space from a remote indoor office using Oculus Quest 2. The participants walked around the virtual Marunouchi area with a pair of subjects.

The two subjects received and viewed video feeds from the AR staff on site, and conversed with each other while viewing images and other objects placed in the virtual space. The participants walked around the city in a virtual space. The experience included the use of 360° video, point cloud video, 2D video, and other technologies to transmit images of the city, and the inclusion of items related to sightseeing, events, and commerce. The value of the experience in the currently envisioned use cases was evaluated.

In addition to the feeling of being in the actual location (in this case, Marunouchi), the form of communication realized by the GIBSON concept also brings the feeling of "actually being in a new space where cyber and physical are fused together. In addition to the sense of being in the actual location (in this case, Marunouchi), this demonstration experiment also revealed that the communication mode realized by the GIBSON concept brings about the sensation of "actually being in a new space that fuses cyber and physical.

In addition to this sense of presence, the system also scored well in terms of observing dynamic changes in the surrounding environment, and was used in the previously envisioned use cases of sightseeing, events, and commerce. In addition, we were able to confirm that the system is promising for use in tourism, events, and commerce, the use cases we had previously envisioned, as well as for further use cases such as hybrid online/offline use in the office. We were also able to receive suggestions from the participants about further use cases, such as hybrid online/offline use in the office.