Debugging/profiling tools in AAA game industry

Introduction : This post was initially about a CPPCon 2015 presentation which is about memory debugging at Electronic Arts. After noticing more CPPCon presentations from Ubisoft I`ve decided to add them to this post as well. And finally added links to tools of Unreal engine and CryEngine.

Memory debugging at Electronic Arts : 

There is a really nice 1 hour CPPCon presentation in which Scott Wardle talks about memory debugging at EA. You can watch in on :

And the PowerPoint presentation can be downloaded from :

https://github.com/CppCon/CppCon2015/blob/master/Presentations/Memory%20and%20C++%20debugging%20at%20EA/Memory%20and%20C++%20debugging%20at%20EA%20-%20Scott%20Wardle%20-%20CppCon%202015.pptx

Initially, the presentation starts with how the memory management in the game industry was. Before Xbox360 and PS3, it was more like embedded systems since there is neither OS
or virtual memory and there is mostly a single CPU core. As with Xbox360 and PS3, both have light OSs called as system software and both are multicore. Xbox360 PowerPC processor had 3 cores each with hyperthreading , on the other hand, PS3 had 1 PPU which is the main core and 6 programmable 128 bit SPUs. And finally, with PS4 and XboxOne there is 64 bit address space and HDD capability for swaps. As he talks about this changes in target game hardware, he shows how their debugging utilities evolved/improved.

Since game projects are quite large projects, obviously the memory leaks are not only the main problem. He mentioned that they had to attack other problems such as fragmentation and heap corruption. Fragmentation will slow things and also heap corruption can cause people , for example , to blame graphics programmers if there is something wrong with shadows, however later you might find out that actually, it is another team that exceeded their own buffer and wrote into shadows buffer. On the other hand, fragmentation is also important as you can find yourself easily in an out of memory case even if there is enough memory.

Notes
– In their custom heaps , they have the concept of “arena”s to particularly avoid lock contention as in jeMalloc. You can find out from which arena a pointer is. In their terminology, a heap is an arena and an allocator for that arena.

ea_arenas

– One of the interesting things is as they pass a const char* argument to new operator which is allocation category. And later they also added concept of scopes to allocators

– As a debug utility they log all allocations and frees in debug mode , therefore they have tools processing those logs That tool shows all allocations and frees by time , all shows memory areas for different categories and it looks particularly useful for fragmentation as it shows coloured visualisation of the memory.

ea_memory_snapshots
ea_mem_tool_visualization

– Mentions that memory leaks are not the hardest to find , however, the more painful thing is tracking down circular references. This can be sorted with a garbage collector, however. this time the problem is you end up with consuming more memory. Obviously, you want to lower your memory consumption as it helps to suffer from cache misses less. He also mentions that in the previous generation consoles , they did everything to reduce memory consumption including reducing the code size. This is also something mentioned
about EASTL , specifically they want to avoid code bloat that comes with templates. Regarding the memory usage, an example is in EA STL documentation on http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2271.html#Appendix_16 In temporary and permanent memory section, it mentions that some games consume system memory so they definitely needed memory usage optimisation.

– One of the problems they had to solve is polymorphism in order to be able to use one allocator for all different classes.

– Their  DeltaViewer in their TurboTuner which stores various data , not only memory allocations and also threads, IO and others , in tables and is able to provide different views. Also it has a memory leak investigator. As for memory leak handling , they capture allocations and also frees however they are not only interested in that all allocations are freed , but also when they are freed. So an allocation happened for a particular level should be freed at the end of that level , not at the beginning or middle of next level as higher memory usage
leads to low performance.

ea_leaks

– Also they can categorize memory usage. This feature is quite useful as they can see the memory usage level by level.

ea_memory_categorisation

Performance cost of these debug mode features : He mentioned he is surprised about the performance cost of it as it was like 3-4 frames in a second.

 

Ubisoft tools : Ubisoft Montreal delivered some quite nice talks. Like EA, they also do memory tagging / memory allocations with names. Additionally, they have their “break”s which triggers breakpoints in VS debugger on certain events :

Here is CPPCon 2014 presentation :

 

And in CPPCon 2016 , they have made another presentation in which they have showed their profiling tools which also help issues such as deadlocks and memory leaks. Here is the Youtube video “Rainbow Six : Quest for performance”  :  https://www.youtube.com/watch?v=tD4xRNB0M_Q

  • They mention similar about cost of memory allocations and finding issues in games. Another common thing with EA presentation is the tagging events which obviously greatly helping them in identifiying issues. It is mentioned that Rainbow Six Siege is 70 FPS and with memory allocation profiling turned on,  it only drops to 20 FPS.
  • Differently from EA`s presentation, they showed their array analyzer tool which gives statistics about their array implementation such as number of instances, reserve sizes, peak buffer sizes , linear traversals and copy counts :ubisoft_array_analyzer
  • Another nice looking tool is  their lock analyzer which shows them lock contention :ubisoft_lock_analyzer

Unreal engine : Unreal engine 4 is one most used game engines in the world. And of course it comes with its own toolchain. You can actually use Unreal Engine freely and explore some of these tools :

https://docs.unrealengine.com/latest/INT/Engine/UI/ClassViewer/index.html Let`s you explore all classes in the Unreal engine

https://docs.unrealengine.com/latest/INT/Gameplay/Tools/VisualLogger/ That is the log watching utility. The feature I really like in that is its being timelined based which is always like that in game industry tools.

https://docs.unrealengine.com/latest/INT/Engine/Performance/ Performance and profiling tools

https://docs.unrealengine.com/latest/INT/Gameplay/Tools/NetworkProfiler/ Network profiler

Cryengine : Another common AAA game engine which also you can try for free. Some of tools it has are :

http://docs.cryengine.com/display/SDKDOC2/Debugging+and+Profiling+Tools  Profiling tools , here is the screenshot of memory profiling :

meminfo201

RadGameTools Telemetry  : RADGameTools is a game middleware company which is famous for their Bink video codecs. They also offer a profiling tool designed for games which is similar to the tool mentioned :

http://www.radgametools.com/telemetry.htm

It can also show thread utilization , lock contention and even context switches. You can watch is video on :

Advertisements
Posted in c++

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s