Introduction : This post was initially about a CPPCon 2015 presentation which is about memory debugging at Electronic Arts. After noticing more CPPCon presentations from Ubisoft I`ve decided to add them to this post as well. And finally added links to tools of Unreal engine and CryEngine.
Memory debugging at Electronic Arts :
There is a really nice 1 hour CPPCon presentation in which Scott Wardle talks about memory debugging at EA. You can watch in on :
And the PowerPoint presentation can be downloaded from :
Initially, the presentation starts with how the memory management in the game industry was. Before Xbox360 and PS3, it was more like embedded systems since there is neither OS
or virtual memory and there is mostly a single CPU core. As with Xbox360 and PS3, both have light OSs called as system software and both are multicore. Xbox360 PowerPC processor had 3 cores each with hyperthreading , on the other hand, PS3 had 1 PPU which is the main core and 6 programmable 128 bit SPUs. And finally, with PS4 and XboxOne there is 64 bit address space and HDD capability for swaps. As he talks about this changes in target game hardware, he shows how their debugging utilities evolved/improved.
Since game projects are quite large projects, obviously the memory leaks are not only the main problem. He mentioned that they had to attack other problems such as fragmentation and heap corruption. Fragmentation will slow things and also heap corruption can cause people , for example , to blame graphics programmers if there is something wrong with shadows, however later you might find out that actually, it is another team that exceeded their own buffer and wrote into shadows buffer. On the other hand, fragmentation is also important as you can find yourself easily in an out of memory case even if there is enough memory.
– In their custom heaps , they have the concept of “arena”s to particularly avoid lock contention as in jeMalloc. You can find out from which arena a pointer is. In their terminology, a heap is an arena and an allocator for that arena.
– One of the interesting things is as they pass a const char* argument to new operator which is allocation category. And later they also added concept of scopes to allocators
– As a debug utility they log all allocations and frees in debug mode , therefore they have tools processing those logs That tool shows all allocations and frees by time , all shows memory areas for different categories and it looks particularly useful for fragmentation as it shows coloured visualisation of the memory.
– Mentions that memory leaks are not the hardest to find , however, the more painful thing is tracking down circular references. This can be sorted with a garbage collector, however. this time the problem is you end up with consuming more memory. Obviously, you want to lower your memory consumption as it helps to suffer from cache misses less. He also mentions that in the previous generation consoles , they did everything to reduce memory consumption including reducing the code size. This is also something mentioned
about EASTL , specifically they want to avoid code bloat that comes with templates. Regarding the memory usage, an example is in EA STL documentation on http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2271.html#Appendix_16 In temporary and permanent memory section, it mentions that some games consume system memory so they definitely needed memory usage optimisation.
– One of the problems they had to solve is polymorphism in order to be able to use one allocator for all different classes.
– Their DeltaViewer in their TurboTuner which stores various data , not only memory allocations and also threads, IO and others , in tables and is able to provide different views. Also it has a memory leak investigator. As for memory leak handling , they capture allocations and also frees however they are not only interested in that all allocations are freed , but also when they are freed. So an allocation happened for a particular level should be freed at the end of that level , not at the beginning or middle of next level as higher memory usage
leads to low performance.
– Also they can categorize memory usage. This feature is quite useful as they can see the memory usage level by level.
Performance cost of these debug mode features : He mentioned he is surprised about the performance cost of it as it was like 3-4 frames in a second.
Ubisoft tools : Ubisoft Montreal delivered some quite nice talks. Like EA, they also do memory tagging / memory allocations with names. Additionally, they have their “break”s which triggers breakpoints in VS debugger on certain events :
Here is CPPCon 2014 presentation :
And in CPPCon 2016 , they have made another presentation in which they have showed their profiling tools which also help issues such as deadlocks and memory leaks. Here is the Youtube video “Rainbow Six : Quest for performance” : https://www.youtube.com/watch?v=tD4xRNB0M_Q
- They mention similar about cost of memory allocations and finding issues in games. Another common thing with EA presentation is the tagging events which obviously greatly helping them in identifiying issues. It is mentioned that Rainbow Six Siege is 70 FPS and with memory allocation profiling turned on, it only drops to 20 FPS.
- Differently from EA`s presentation, they showed their array analyzer tool which gives statistics about their array implementation such as number of instances, reserve sizes, peak buffer sizes , linear traversals and copy counts :
- Another nice looking tool is their lock analyzer which shows them lock contention :
Unreal engine : Unreal engine 4 is one most used game engines in the world. And of course it comes with its own toolchain. You can actually use Unreal Engine freely and explore some of these tools :
https://docs.unrealengine.com/latest/INT/Engine/UI/ClassViewer/index.html Let`s you explore all classes in the Unreal engine
https://docs.unrealengine.com/latest/INT/Gameplay/Tools/VisualLogger/ That is the log watching utility. The feature I really like in that is its being timelined based which is always like that in game industry tools.
https://docs.unrealengine.com/latest/INT/Engine/Performance/ Performance and profiling tools
Cryengine : Another common AAA game engine which also you can try for free. Some of tools it has are :
http://docs.cryengine.com/display/SDKDOC2/Debugging+and+Profiling+Tools Profiling tools , here is the screenshot of memory profiling :
RadGameTools Telemetry : RADGameTools is a game middleware company which is famous for their Bink video codecs. They also offer a profiling tool designed for games which is similar to the tool mentioned :
It can also show thread utilization , lock contention and even context switches. You can watch is video on :