(언리얼 온라인 러닝) Oculus와 언리얼 엔진을 사용한 VR 개발

가상현실 개발을 위한 언리얼 엔진 준비하기 

학습 과정 소개

1. 오큘러스 vr을 위한 언리얼 엔진 설정방법

2. VR에 필요한 그래픽과 렌더링 관련 내용

3. Locomotion을 구현하는 방법

4. 손 모델과 인터랙션을 구현하는 방법

5. UI를 구현하는 방법

6. 레벨 로드의 체계를 정리하는 최적화 방법

7. 공간 오디오를 추가하는 방법

8. 네트워킹 기능을 구성하는 방법

9. 증강현실

10. 최적화도구

11. 를 사용하는 방법

12. Virtual Reality Checks

아웃라인과 Oculus 소프트웨어 

dashboard.oculus.com

www.oculus.com/setup/

 

언리얼 엔진 설치 및 프로젝트 생성

 

프로젝트 세팅으로 언리얼 엔진 최적화

1. 플러그인의 오큘러스를 제외한 모든 Subsystem을 해제

2. 프로젝트 셋팅

- Forward Shading 체크. VR프로젝트에서 '보통' 사용되지 않는 기능이 많은 디퍼드를 꺼서 성능 향상

- Motion Blur 체크 해제 (포워드 렌더링는 모션블러를 지원하지 않음))

- Instance Stereo 체크 (VR은 카메라가 두대인데 이 때 겹치는 렌더정보를 공유해서 드로우 작업Draw thread duration을 줄임)

- Anti-Aliasing 을 MSAA로 변경 (포워드에서만 가능한 좋은 퀼리티의 안티앨리어싱)

3. Project-> Supported Platform 설정

Windows, 퀘스트용은 Android

4. Description category -> Start in VR로 자동적으로 VR로 실행되게 설정

5. Smooth Frame Rate 기본 비활성. FPS가 가변적인 환경에서 도움이 되는 옵션. VR환경에선 고정프레임이 중요하므로 

6. Use Fixed Framerate 기본 비활성. 고정된 FPS가 되지못하면 게임의 실행속도를 늦춰서 프레임을 맞추는 기능.

7. Custom Timestep -> 기본 None. 프레임의 타임스텝이 중요한 환경에서 키는 기능.

8. (Quest : Mobile HDR 비활성, Mobile Multi-View 활성)

 

9. DefaultEngine.ini 에서 다음 내용을 추가

 

[OnlineSubsystemOculus]
bEnabled=true
OculusAppId=****
RiftAppId=****

ID는 대쉬보드의 API탭에서 확인가능. 둘곳에 붙여넣기 하면 됨.

 

퀘스트용으로는 Engine/Config/Android/AndroidEngine.ini

 

 

VR에서 데모 레벨 탐색

핵심 에셋 생성

여러가지 기본 블루프린트들을 생성한다.

 

GameMode : 

Specify all low-level blueprints for the map we use.

Contains various rules for the game.

Identifies which pawn the controller should possess.

 

GameInstance : 영구적인 프로퍼티가 있으므로 레벨 변경시에도 유지될 데이터를 저장.

Handling Sessions.

Moving Map.

Retrieving data for Leaderboard.

Holding our most important variables.

 

GameState :

Keeps track of properties that change during gameplay.

Keeps track of the Oculus ID of the player in our game.

 

PlayerState:

Oversees holding any data relevant to us

Accessible by anyone

Player's score, The Color of our hands, Our name

 

이후 다양한 열거형과 구조체 타입을 설정한다.

 

폰 생성

 

핵심 에셋의 작동

 

 

---

VR 그래픽 및 렌더링 고려 사항 확인하기 

개요 및 라이팅 렌더링

Consideration for Rendering VR:

- How many ms we are willing to spend on each frame and what resolution ?

- How many lights we need per pixel and our preferred shadow method?

- How complex the materials are and which Anti-Aliasing method to use?

 

Quality VR Experience Utilize

- High Resolution

- High stable framerate

 

Classic Forward Rendering

- 초기 3D 그래픽카드들이 사용하던 방식. 

- Geometry is rendered with triangles and each covered pixel the GPU has to compute final RGB color based on textures, material attributes and lighting data.

- The performance per object was dominated by how many lights touched it.

- attenuation radius를 제한하여 영향을 주는 라이팅의 갯수를 조절할 수 있게 하는게 좋다.

- 하나의 물체가 하나의 빛에서만 렌더링 되게 하는게 좋다.

 

Deferred Rendering Techniques

- All rely on storing data in a screen-space aligned texture or render target

- Each Pixel on the screen has a Texture Containing : Material Color, Roughtness, Surface Normals.

- Some deferred techniques only require a few channels : Deferred shadows and fog - z-buffer

- Read from attributes in a following pass, decoupling itself from the material shaders

- Allow for simpler, more elegant code which can result in better performance with scenes of high complexity.

- The downside comes with the need to read and write more momery, this has more of an impact at higher resolution, even larger impact on low bandwidth hardware like mobile VR.

 

Classic Deferred shading passes

- First, all objects get rendered with a pixel shader outputting material attributes.

- Then, shading is computed using the G Buffer and light information

 

Deferred Shading

- offers simplicity and elegance

- Good performance when using complex materials with many lights.

 

Note for Anti-Aliasing with deferred rendering

- It is possible to render the GBuffer using MSAA (Multi Sample Anti-Aliasing), but it adds complexity and adds even more to how much memory we will need, reducing performance, and it is not currently supported by Unreal Engine.

 

Temporal Anti Aliasing

- Requires a few frames to converge to a good result

- Content that wasn't on the screen before will appear blurrier briefly until it becomes sharp.

- Fast movements or motion over noisy content can result in smearing.

- Works with deferred shading fixes more aliasing issues than MSAA, such as specular aliasing.

- Relies on feedback with full screen buffers, giving it a higher performance impact for VR

 

Fast Approximate Anti-Aliasing (FXAA)

- Does not have the restriction of delayed convergence.

 

TAA와 FXAA는 렌더타깃을 읽어오는것에 의존하기때문에 메모리 대역폭이 한정된 모바일에서는 힘들다.

 

Deferred, Pros and Cons

- Offers great benefits when using many features.

- Need to be selective when targeting high resolution and high frame rate.

 

Cluster Forward Rendering

- modern twist on the classic method, that is used in Unreal Engine 4.

- Utilizing dynamic branching and a per-frame light hierachy, Clustered Forward Rendering allows for efficient light culling without a large performance impact from the intermediate buffer, such as the G Buffer, directly outputting an RGB Color. 

- Avoids wasting memory bandwidth, allows the use of the GPU built-in hardware Anti-Aliasing, MSAA

 

MSAA

- Anti-aliasing can be achieved by rendering an image in a higher resolution.  

- MSAA is more efficient as it avoids the 4x fragment shader cost and duplicates the color values into 4 sub pixel regions, depending on whether a primitive covers the sub pixel. the additional sub pixel saves the fragment shader execution. but it still uses three times more memory.

- MSAA only solves primitive aliasing, but has no effect on aliasing from textures or shading.

- Texture aliasing can be solved by using texture mip maps and high quality filtering.

- 텍스쳐 크기가 2의 제곱이 되게 하자.

 

Specular Shimmering and Compoiste Textures.

- A form of shading aliasing that MSAA cannot solve, but UE4 implements techniques that fix specular shimmers with the Composite texture feature.

docs.unrealengine.com/en-US/RenderingAndGraphics/Textures/Composite/index.html

 

- 복잡한 씬에서는 디퍼드에서 포워드로 전환해도 퍼포먼스 이득을 보기 어렵다.

- 기능을 제약하면 효과가 확실히 들어난다. :

ex) 리플렉션은 거울같은 오브젝트에만 사용하기, 스카이박스의 그림자를 끄기 등

 

Design Best Practices

- if you target very high-end PCs and complex visuals you might want to consider deferred with FXAA.

- UE4's forward renderer has a limited feature set. some of theose features have a high performance impact in a forward rendere and shouldn't be used.

 

Shader Permutations

Generated automatically and can save programmer time, provides efficient shaders, but note that it results in long compile times and more memory required for the shader variants.

 

- 포워드 셰이딩은 단일 셰이더 패스에 Shading, Light types, Meterial types, Tone mapping을 처리하기 때문에 유연성이 떨어진다. 디퍼드는 서로 다른 패스를 사용해서 기능별 토글이 유리하다.

 

To Toggle Features

- Use dynamic branching in the shader.

- compile specialized shaders.

 

Materials Best Practices

- Using fewer materials and using them more often. This makes optimization more effective and allows the GPU to render more efficiently since it does not need to switch shaders as often to render many materials.

 

For 90 FPS

Only have 11ms CPU time before you miss a frame.

 

Lighting 

라이팅은 VR 렌더링에서 퍼포먼스에 가장 큰 영향을 주는 요소중 하나. 

다이나믹 라이트의 수를 줄여서 신 최적화를 하는게 좋다.

 

Static Lights

The least computationally expensive method for lighting a scene.

라이트맵으로 구워지기 때문에 순회시간과 공간량으로만 영향을 끼친다.

Good lighting artist can make a very convincing scene with only static lights.

High performance, low quality, don't affect dynamic object.

 

Dynamic Lights

가능한 한 최소화 하는게 최선. 손정등, 횃불 등

다이나믹 그림자는 섀도맵 기법을 사용하는데 그림자를 드리우는 모든 요소당 렌더링 패스가 필요하므로 최대한 피하는게 상책..

 

Stationary Lights

Have behaviors of both dynamic and static lights.

- Whenever stationary lights or dymaic lights overlap, the renderer needs to do a lot of extra work.

- 스테이셔너리 라이트는 4개까지만 오버랩 가능하며, 이후로는 다이나믹 라이트로 변환됨.

- 다이나믹 라이트의 최대 렌더링 거리를 정의해 두고 가능한한 줄이자.

 

Reflection Captures

Take a 360 snapshot of your scene and aplly the lighting details of that snapshot to dynamic and static objects.

Make sure the reflection captures are not set to capture every frame.

 

GPU Time

렌더링 될 대상을 제외시키는 걸로 CPU, GPU 시간을 절약할 수 있다.

퀘스트같은경우 배터리 절약도 기여할 수 있다.

 

View Frustum Culling

View frustum culling tests a tight pre-computed bounding box around each object against the view frustum.

- the view frustum is defined by the screen edges and the camera location.

- for VR the engine can compute a single frustum for both eyes.

 

Note

- Splitting up the geomoetry into too many meshes can increase the draw call count and CPU time which reduces performance. 각 메쉬에 300+의 삼각형을 두는게 권장됨.

- You can batch together smaller objects that are used multiple times in order to render them in a single draw call.

-> All objects are drawn together even if some of these objects are outside of the viewing frustum (which could impact performance)

 

Occlusion Culling

A real time check of whether an object is potentially occluded by other objects so our app knows not to render the objects we can't see, increasing performance.

- 오클루전이 적은 씬에서는 효과를 보기 힘듬

 

Note

- Profile your content

- Pay attention to the performance impact over mltiple frames.

 Project settings-> Engine - Rendering - > Occlusion Culling

 

PreComputed Visibility Volume

Stores the visibility state of Actor's locations in the world based on where the player or camera is.

 

머티리얼 렌더링

퍼포먼스와 관련해선, 불투명한 머티리얼을 쓰는게 좋지만 헤어, 나무 등 불가피한 상황이 있다.

 

Opaque Rendering

A Blend Mode for solid objects and the most performant material. with rough front-to-back sorting it can avoid PS computation for occluded pixels.

 

Masked Rendering

A Blend Mode for objects that have transparency through a mask that produces pixels that are either 100% opaque or 100% translucent,  with no in-between transparency.

언리얼4에서 MSAA가 활성화되어있으면 픽셀 셰이더에서 단순한 텍스쳐 제거나 인스트럭션 클립만 하는게 아니라 

픽셀 디시전별 바이너리 결과가 나온다. MSAA 수준에 따라 셰이드가 더 많이 보이며 알파값을 기반으로 MSAA 샘플을 비활성하기때문에 살짝 더 부드러운 트랜지션이 이루어진다. 4X MSAA는 셰이드를 세 개만 제공해서 가장자리가 큰 화면 영역을 커버할 때 벤딩이 일어난다.

 

Translucent Rendering

A Blend Mode that can be used for objects that exist anywhere between fully opaque or fully transparent, but this will have the most performance impact, and is thus the least recommended for optimizing a game.

 

Alpha Transparency, Additive and Modulate

Blend Modes that are the more performant alternatives to Translucent. It's important to remeber that Additive can only brighten the material, while Modulate can only darken the material.

 

 

VR 고려 사항

일반적인 3D 렌더링은 화면이 평평한 2D지만, VR용으로는 거대한 FOV에 맞춰서 이미지를 왜곡시켜야 한다.

이 때 화면의 바깥 영역에서 계산 낭비가 생기는데, Quest는 FFR로 이를 해결하려한다.

 

Fixed Foveated Rendering

allow us to save pixel shader computations and memory bandwidth in the outer screen areas.

블루프린트 노드로 해당 옵션을 사용할 수 있다. Set Fixed Foveated Rendering Level

The higher the FFR level, the better performance savings, but at the expense of peripheral resolution.

FFR High Top : 화면 아래쪽이 디테일해지고 위쪽 해상도가 낮아진다.

FFR이 활성화되면 이미지 중앙에서 멀리 떨어진 영역의 해상도가 낮아지기 때문에 디더링이 더 부각되는걸 볼 수 있다.

Try to avoid dithering with FFR. 

 

Multiview Render Feature

Addresses the inefficiency of sequential multiview rendering by adding a means to render to multiple elements of a 2D texture array simultaneously.

Mobile HDR을 꺼야 Mobile Multi-View, Mobile Multi-View Direct를 활성화 할 수 있다.

 

파티클

Where Possible, consider using more material effects and a few layered particle effects.

빌보드 효과는 스트레오로 보는 VR에서 평평하다고 느끼게하기 쉽다.

Avoid rendering large sprites close to the camera, or inetersecting the camera.

Use smaller particles where possible.

spawn particles in a volume, read particularly well to, since there is depth between particles.

 

---

이동 및 인체 공학 이해와 구현 

VR 속 이동 디자인

Motion Sickness

Also known as Simulation Sickness. More common in Virtual Reality than traditional gaming experiences since the game camera is our player head.

 

Note

Test Regularly to see if there are any unintended issues like simulation sickness for users who are less familiar with being inside a headset.

 

Avoid

- Cinematic Cameras.

- Anything that moves the position or orientation of the Point Of View

 

Field of View

Don't override or mainpulate. This value needs to match the physical geometry of the headset and lens.

 

Walking Bob

Camera effect to mimic walking, this effect exaggerates movement in VR, possibly causing motion sickness for users.

 

Shaking the Camera

in VR games, this artificial movement can trigger Simulation Sickness very quickly.

 

Strong an Vibrant Lighting

These can cause Simulation Sickness in VR. Avoid this by using cooler shades and dimmer lights than you normally would. 

 

Bright Scenes and Strobing Effects

These can be harmful for users with specific photo-sensitivities.

 

Avoid Stairs

When the user in moving quickly, especially up and down stairs, it can be very disorienting. Use elevators instead if you want to include spaces with elevation.

 

Acceleration

Should be instantaneous and not gradual

users should have precise control over the speed.

 

Depth of Field & Motion Blur

These generate Simulation Sickness because they interfere with how our eyes behave.

코드 교환 방법
이동 유형과 고려 사항

 

Stationary

The simplest form of locomotion, directly driven by the world space location of a user.

 

Rail Locomotion

Puts users on "rails", moving them through the environment automatically.

 

Gaze teleportation

Functions similarly to general teleportation, but the point that users will teleport to is determinded by their gaze.

 

Standard Controller Movement

The user moves around by using the analog sticks on the controllers in their hands.

컨트롤러의 값을 바로 이동속도에 맵핑하는게 좋다. 서서히 움직이는건 멀미 유발.

 

 Vignette

Creates a tunnel vision effect in the user's eyes.

표준 컨트롤러 이동 구현

Oculus Guardian
Oculus Guardian 디버그 뷰
리센터링
남은 에셋 수집

 

 

---

핸드 프레전스 및 인터랙션 통합하기 

VR에서 핸드 구성

 

핸드 커뮤니케이션: 능동 및 수동

- 직접 호출을 통한 능동, 인터페이스를 통한 수동

 

 

VR 핸드 인터랙션

 

Broadcasting Events

Gives you a simple way to notify other systems of what the player is doing and respond without needing to explicitly find references to other objects. 

 

손에 물건을 잡을 때 다양한 변화(ex:양손 잡기)가 있을 수 있으므로 이벤트-드리븐으로 유연하게 코드를 짜는게 좋다.

 

Fist collider, Finger collider 

These colliders are attached to the hands, but not used for grabbing.

 

Colliders Best Practices

- grabbing collider should completely overlap with your fist collider. this ensures you can always grab objects you intend to instead of pushing them away from you

- If there are physics objects that overlap with the collider at the moment you enable the collider, you may need to add a custom collision resolution to ensure the objects do not fly away.

 

Late Update

Used to update the hand positions at the last possible moment before rendering.

 

오브젝트 잡기

이벤트 디스패쳐로 리스폰하는 방식 좋아보임

 

VR 핸드에 생동감 부여

애니메이션 블루프린트 커뮤니케이션

다트판 콜리전

다트 게임 빌드

 

---

언리얼 모션 그래픽으로 UI 구현하기

UI 및 그래픽 개요

텍스트를 많이 안쓰는게 좋다.

 

UMG Consideration

- Use efficiently as possible so that it doesn't bog down the framerate

- Even if you only have a small amount of UI, you should run it efficiently to save on power and extend battery life

- With every widget comes an impact to render performance and memory

 

Rendering Conventional UI

- Use isolated elements that float around in the world.

- Rener the UI to a texture which will then get drawn onto elements on the world.

ex) if text is only going to be a few words or sentences, separated from other pieces of text, the better choice would probably be to use an isolated text render actor. 

However as the amount of words, boxes, and elements increases, it's more efficent to use the render target, which is where UMG comes in.

 

Note

Draw calls on the Quest need to stay below 150 per frame.

 

Stereo Layer

Render in parallel to the world of your game engine, allowing you to send a separate Texture to the VR Head Mounted Display(HMD) and have it re-projected in a separate rendering pass than the rest of the project.

- Can be difficult to manage and mix with the rest of your 3D content.

- Can quickly cause eye strain.

 

HUD Elements

When nested within the camera's hierarchy they automatically inherit the camera's transforms, keeping it welded in the HMD's screen space.

- 중심에서 멀어질수록 텍스트를 읽기 힘들어진다. 화면 가장자리에 렌더링하는걸 피하자.

- Applying a Render Target to a masked material, on a simple plane attache to the camera.

- Creating a custom mesh that takes advantage of Stereo Rendering, giving the UI three-dimensional depth.

- Keep your HUD materials as simple as possible, unlit, with the fewest number of instructions possible.

 

텍스트 기반 UI

텍스트 렌더에서 알아야할것 - 사인드 디스턴스 필드(SDF)

Signed Distance Fonts

Fonts that are encoded into their texture sheets in such a way that they take adavange of the texture sampling in the GPU to smooth out the edges.

- Texture minification

- The distance field math works well for edges but doesn't work well when multiple edges are within a pixel. You might see very harsh aliasing.

 

For MSAA

modify your content to use a masked material. This way you still get full shadow casting receiving, antialiasing, high quality lighting and per pixel occlusion.

 

UMG

Creates a hierarchy of all your UI elements starting with the canvases at the top of those hierarchies.

- Tri-linear sampling makes transitions smooth.

- Antisotropic filtering make this even more readable when seen from an angle.

- Check the mipmap box to encourage automatic mipmap generation.

 

Stereo Layer Considerations

You'll need to specify whether the layers exist in a fixed virtual world location or if they are going to be a locked layer that moves with the user's head.

언리얼 모션 그래픽

 
UX 및 인터랙션

3D 위젯

UMG와 블루프린트 간 커뮤니케이션

유저 세팅 구성

다트 게임에 UMG 추가

점수판에 애니메이션 추가

 

---

레벨 로드 최적화 시연 

레벨 나누기
화면 로딩

Options for Load screen

- Fade to solid color

- Show VR Splash screen

- Show one or more stereo layers

 

PSO 캐시 시스템

 

---

가상현실용 사운드 디자인 

공간 오디오와 VR에서의 그 중요성
코드 교환 방법
Oculus Spatializer를 활용한 구현
공간 오디오 테스트 및 디버깅

 

 

---

소셜 VR 경험 만들기 

Oculus 대시보드
코드 교환 방법
맵 간 이동
업적
리더보드
블루프린트에서 C++로
Destinations 및 Rich Presence

 

---

Oculus 혼합현실 캡처 

Oculus MRC
코드 교환 방법
MRC 블루프린트

 

---

프로젝트 최적화를 위한 툴 및 원리 

콘솔 명령
코드 교환 방법
시각화 옵션
퍼포먼스 분석 및 일반 최적화 적용 
33%
퍼포먼스 문제 파악 및 대처
코드 교환 방법
단계별 최적화 적용

 

---

Oculus VR 앱 제출 준비 

100%
VRC - 가상현실 체크
코드 교환 방법