Preamble
This article has been written based on my personal experience, if you think you are offended in some ways because of my personal opinions about programming and / or my coding style, please stop reading now. I am not a guru programmer and I don’t want to impose anything on anyone.
Introduction
During the development of my engine(s), I have always had the need for better handling of game assets, even if in the past it was sufficient to hard code the needed resources at startup. Now with the added speed of modern computers it is possible to create something more advanced, dynamic and general-purpose.
The most important points I needed for a resource manager were:
- reference counted resources.
- If a resource is already present in the database, expose it as raw pointer, else load it.
- know automatically when to free the resource.
- fast access using string to retrieve resources
- clean everything on exit without manually deleting the resources.
- mapping resource using a referenced structure.
One of this was that if I loaded the resource for the first time, its count was set to 1 and when the same resource was asked for its count was increased to 2. So I still had the problem to ignore the first reference, which was again solvable using a custom deleter and tracking how many resources where effectively still left. Further more, shared pointers are thread safe and according to the standard they are synced even if there is no strict necessity.
I am not saying that smart pointer are useless, they are used quite extensively, but my approach needed something different; I needed to have total control of reference counting. Basically what I needed was a wrapper class to store the pointer to the loaded resource, and its reference counting. Note that the code I copied from my engine uses some other utility functions - they are not necessary, because they handle error messaging and string ‘normalisation’ (eliminating white spacing and lowering the string down), so you can easily ignore those functions and substitute them with yours. I have also removed all debugging printing during the execution, to keep things clearer.
Let’s have a look at the base class for resources.
class CResourceBase { template < class T > friend class CResourceManager; private: int References; void IncReferences() { References++; } void DecReferences() { References--; } protected: // copy constructor and = operator are kept private CResourceBase(const CResourceBase& object) { } CResourceBase& operator=(const CResourceBase& object) { return *this; } // resource filename std::string ResourceFileName; public: const std::string &GetResourceFileName() const { return ResourceFileName; } const int GetReferencesCount() const { return References; } //////////////////////////////////////////////// // ctor / dtor CResourceBase( const std::string& resourcefilename ,void *args ) { // exit with an error if filename is empty if ( resourcefilename.empty() ) CMessage::Error("Empty filename not allowed"); // init data members References = 0; ResourceFileName=CStringFormatter::TrimAndLower( resourcefilename ); } virtual ~CResourceBase() { } };
The class is self-explanatory - the interesting point here is the constructor, which needs a resource filename, including the full path of your resource and a void pointer to an argument class in case you wanted to load the resource with some initial default parameters. The args pointer comes in handy when you want to instantiate assets during runtime and don’t want to load them. There are some cases where this is useful, for every other case the constructor will serve our purposes well. All of our assets will inherit from this class.
There is, obviously, the reference counter and some functions for accessing it.
The Resource Manager
The resource manager basically is a wrapper for an unordered map. It uses the string as a key and maps it to a raw pointer. I have decided to use an unordered map because I don’t need a sorted arrangement of assets, I really do care at retrieving them in the fastest possible way, in fact accessing a map is O(log N), while for an unordered map is O(N). In addition just because the unordered map is constant speed (O(1)) doesn't mean that it's faster than a map (of order log(N)). Anyway, in my test cases the N value wasn’t so huge, so the unordered map has always been faster than map, thus I decided to use this particular data structure as the base of my resource manager.
The most important functions in the resource map are Load and Unload.
The Load function tries to retrieve the assets from the unordered map. If the asset is present, its reference count is increased and the wrapped pointer is returned. If it's not found in the database, the function creates a new asset class, increases its reference count, stores it in the map and returns its wrapped pointer. Note that its the programmer’s responsibility to create an asset class with a proper constructor. The base class, inherited from CResourceBase class, must provide a string containing the full path from where the asset needs to be loaded and an argument class if any - this will be clearer when the example is provided.
The Unload function does exactly the opposite: looks for the requested asset given its file name, if the resource is found, its reference counter is decreased, and if it reaches zero the associated memory is released.
Since I think that a good programmer understands better 1000 lines of code rather than 1000 lines of words, here you have the entire resource manager:
template < class T > class CResourceManager { private: // data members std::unordered_map< std::string, T* > Map; std::string Name; // copy constructor and = operator are kept private CResourceManager(const CResourceManager&) { }; CResourceManager &operator = (const CResourceManager& ) { return *this; } // force removal for each node void ReleaseAll() { std::unordered_map< std::string, T* >::iterator it=Map.begin(); while ( it!=Map.end() ) { delete (*it).second; it=Map.erase( it ); } } public: /////////////////////////////////////////////////// // add an asset to the database T *Load( const std::string &filename, void *args ) { // check if filename is not empty if ( filename.empty() ) CMessage::Error("filename cannot be null"); // normalize it std::string FileName=CStringFormatter::TrimAndLower( filename ); // looks in the map to see if the // resource is already loaded std::unordered_map< std::string, T* >::iterator it = Map.find( FileName ); if (it != Map.end()) { (*it).second->IncReferences(); return (*it).second; } // if we get here the resource must be loaded // allocate new resource using the raii paradigm // you must supply the class with a proper constructor // see header for details T *resource= new T( FileName, args ); // increase references , this sets the references count to 1 resource->IncReferences(); // insert into the map Map.insert( std::pair< std::string, T* > ( FileName, resource ) ); return resource; } /////////////////////////////////////////////////////////// // deleting an item bool Unload ( const std::string &filename ) { // check if filename is not empty if ( filename.empty() ) CMessage::Error("filename cannot be null"); // normalize it std::string FileName=CStringFormatter::TrimAndLower( filename ); // find the item to delete std::unordered_map< std::string, T* >::iterator it = Map.find( FileName ); if (it != Map.end()) { // decrease references (*it).second->DecReferences(); // if item has 0 references, means // the item isn't more used so , // delete from main database if ( (*it).second->GetReferencesCount()==0 ) { // call the destructor delete( (*it).second ); Map.erase( it ); } return true; } CMessage::Error("cannot find %s\n",FileName.c_str()); return false; } ////////////////////////////////////////////////////////////////////// // initialise void Initialise( const std::string &name ) { // check if name is not empty if ( name.empty() ) CMessage::Error("Null name is not allowed"); // normalize it Name=CStringFormatter::TrimAndLower( name ); } //////////////////////////////////////////////// // get name for database const std::string &GetName() const { return Name; } const int Size() const { return Map.size(); } /////////////////////////////////////////////// // ctor / dtor CResourceManager() { } ~CResourceManager() { ReleaseAll(); } };
Mapping Resources
The resource manager presented here is fully functional of its own, but we want to be able to use assets inside a game object represented by a class.
Think about a 3D object, which is made of different 3D meshes, combined together in a sort of hierarchial structure, like a simple robot arm, makes the idea clearer. The object is composed of simple building blocks, like cubes and cylinders. we want to reuse every object as much as possible and also we want to access them quickly, in case we want to rotate a single joint.
The engine must fetch the object quickly, without any brute force approach, also we want a name for the asset so we can address it using human readable names, which are easier to remember and to organize.
The idea is to write a resource mapper which uses another unordered map using strings as keys and addresses from the resource database as the mapped value. We need also to specify if we want to allow the asset to be present multiple times or not. The reason behind this is simple - think again at the 3D robot arm. We need to use multiple times a cube for example, but if we use the same resource mapper for a shader, we need to keep each of the shaders only once. Everything will become clearer as the code for the mapper unfolds further ahead.
template < class T > class CResourceMap { private: ///////////////////////////////////////////////////////// // find in all the map the value requested bool IsValNonUnique( const std::string &filename ) { // if duplicates are allowed , then return alwasy true if ( Duplicates ) return true; // else , check if element by value is already present // if it is found, then rturn treu, else exit with false std::unordered_map< std::string, T* >::iterator it= Map.begin(); while( it != Map.end() ) { if ( ( it->second->GetResourceFileName() == filename ) ) return false; ++it; } return true; } ////////////////////////////////////////////////////////////////////////////// // private data std::string Name; // name for this resource mapper int Verbose; // flag for debugging messages int Duplicates; // allows or disallwos duplicated filenames for resources CResourceManager<T> *ResourceManager; // attached resource manager std::unordered_map< std::string, T* > Map; // resource mapper // copy constructor and = operator are kept private CResourceMap(const CResourceMap&) { }; CResourceMap &operator = (const CResourceMap& ) { return *this; } public: ////////////////////////////////////////////////////////////////////////////////////// // adds a new element T *Add( const std::string &resourcename,const std::string &filename,void *args=0 ) { if ( ResourceManager==NULL ) CMessage::Error("DataBase cannot be NULL (5)" ); if ( filename.empty() ) CMessage::Error("%s : filename cannot be null",Name.c_str()); if ( resourcename.empty() ) CMessage::Error("%s : resourcename cannot be null",Name.c_str()); std::string ResourceName=CStringFormatter::TrimAndLower( resourcename ); // looks in the hashmap to see if the // resource is already loaded std::unordered_map< std::string, T* >::iterator it = Map.find( ResourceName ); if ( it==Map.end() ) { std::string FileName=CStringFormatter::TrimAndLower( filename ); // if duplicates flag is set to true , duplicated mapped values // are allowed, if duplicates flas is set to false, duplicates won't be allowed if ( IsValNonUnique( FileName ) ) { T *resource=ResourceManager->Load( FileName,args ); // allocate new resource using the raii paradigm Map.insert( std::pair< std::string, T* > ( ResourceName, resource ) ); return resource; } else { // if we get here and duplicates flag is set to false // the filename id duplicated CMessage::Error("Filename name %s must be unique\n",FileName.c_str() ); } } // if we get here means that resource name is duplicated CMessage::Error("Resource name %s must be unique\n",ResourceName.c_str() ); return nullptr; } ///////////////////////////////////////////////////////// // delete element using resourcename bool Remove( const std::string &resourcename ) { if ( ResourceManager==NULL ) CMessage::Error("DataBase cannot be NULL (4)"); if ( resourcename.empty() ) CMessage::Error("%s : resourcename cannot be null",Name.c_str()); std::string ResourceName=CStringFormatter::TrimAndLower( resourcename ); if ( Verbose ) CMessage::Trace("%-64s: Removal proposal for : %s\n",Name.c_str(),ResourceName.c_str() ); // do we have this item ? std::unordered_map< std::string, T* >::iterator it = Map.find( ResourceName ); // yes, delete element, since it is a reference counted pointer, // the reference count will be decreased if ( it != Map.end() ) { // save resource name std::string filename=(*it).second->GetResourceFileName(); // erase from this map Map.erase ( it ); // check if it is unique and erase it eventually ResourceManager->Unload( filename ); return true; } // if we get here , node couldn't be found // so , exit with an error CMessage::Error("%s : couldn't delete %s\n",Name.c_str(), ResourceName.c_str() ); return false; } ////////////////////////////////////////////////////////// // clear all elements from map void Clear() { std::unordered_map< std::string, T* >::iterator it=Map.begin(); // walk trhough all the map while ( it!=Map.end() ) { // save resource name std::string filename=(*it).second->GetResourceFileName(); // clear from this map it=Map.erase ( it ); // check if it is unique and erase it eventually ResourceManager->Unload( filename ); } } ////////////////////////////////////////////////////////// // dummps database content to a string std::string Dump() { if ( ResourceManager==NULL ) CMessage::Error("DataBase cannot be NULL (3)"); std::string str=CStringFormatter::Format("\nDumping database %s\n\n",Name.c_str() ); for ( std::unordered_map< std::string, T* >::iterator it = Map.begin(); it != Map.end(); ++it ) { str+=CStringFormatter::Format("resourcename : %s , %s\n", (*it).first.c_str(), (*it).second->GetResourceFileName().c_str() ); } return str; } ///////////////////////////////////////////////////////// // getters ///////////////////////////////////////////////////////// // gets arrays name const std::string &GetName() const { return Name; } const int Size() const { return Map.size(); } ////////////////////////////////////////////////////////// // gets const reference to resource manager const CResourceManager<T> *GetResourceManager() { return ResourceManager; } ///////////////////////////////////////////////////////// // gets element using resourcename, you should use this // as a debug feature or to get shared pointer and later // use it , using it in a section where performance is // needed might slow down things a bit T *Get( const std::string &resourcename ) { if ( ResourceManager==NULL ) CMessage::Error("DataBase cannot be NULL (2)"); if ( resourcename.empty() ) CMessage::Error("%s : resourcename cannot be null",Name.c_str()); std::string ResourceName=CStringFormatter::TrimAndLower( resourcename ); std::unordered_map< std::string, T* >::iterator it; if ( Verbose ) { CMessage::Trace("%-64s: %s\n",Name.c_str(),CStringFormatter::Format("Looking for %s",ResourceName.c_str() ).c_str()); } // do we have this item ? it = Map.find( ResourceName ); // yes, return pointer to element if ( it != Map.end() ) return it->second; // if we get here , node couldn't be found thus , exit with a throw CMessage::Error("%s : couldn't find %s",Name.c_str(), ResourceName.c_str() ); // this point is never reached in case of failure return nullptr; } ///////////////////////////////////////////////////////// // setters void AllowDuplicates() { Duplicates=true; } void DisallowDuplicates() { Duplicates=false; } void SetVerbose() { Verbose=true; } void SetQuiet() { Verbose=false; } //////////////////////////////////////////////////////////// // initialise resource mapper void Initialise( const std::string &name, CResourceManager<T> *resourcemanager, bool verbose,bool duplicates ) { if ( resourcemanager==NULL ) CMessage::Error("DataBase cannot be NULL 1"); if ( name.empty() ) CMessage::Error("Array name cannot be null"); Name=CStringFormatter::TrimAndLower( name ); // normalized name string ResourceManager=resourcemanager; // copy manager pointer // setting up verbose or quiet mode Verbose=verbose; // setting up allowing or disallowing duplicates Duplicates=duplicates; // emit debug info if ( Verbose ) { if ( Duplicates ) CMessage::Trace("%-64s: Allows duplicates\n",Name.c_str() ); else if ( !Duplicates ) CMessage::Trace("%-64s: Disallows duplicates\n",Name.c_str() ); } } ///////////////////////////////////////////////////////// // ctor / dtor CResourceMap() { Verbose=-1; // undetermined state Duplicates=-1; // undetermined state ResourceManager=NULL; // no resource manager assigned } ~CResourceMap() { if ( Verbose ) CMessage::Trace("%-64s: Releasing\n",Name.c_str() ); Clear(); // remove elements if unique } }; }
Basically, the class is a wrapper for the resource database operations. Among the private data, as you can see, its present an unordered map where the first key is a string and the mapped value is the pointer directly mapped from the resource database. Let's have a look at the function members now.
The Add function performs many tasks. First, checks if the name for the asset is already present, since duplicated names for the assets are not allowed. If name is not present, it performs the attempt to upload the assets from the resource database, then it checks if the filename is unique and if the duplicates flag is not set to true. Here I have used a brute force approach, the reason behind it is that if I wanted to have a sort of bidirectional mapping, I should have used a more complex data structure, but I wanted to keep things simple and stupid. At this point the resource database uploads the asset, and if it's present, it hands back immediately the address for the required resource. If not it loads it, making the process completely transparent for the resource mapper and it gets stored in the unsorted map data structure. Note again, that all the error checking are just wrappers for a throw, you may want to replace with your error checking code, without compromising the Add function itself.
The Remove function is a little bit more interesting, basically the safety checks are the same used in Add, the resource is erased from the map, the resource database removal function is invoked, but the resource database doesn’t destroy it if it is still shared in some other places. By ‘some other places’ I mean that the asset may be still be present in the same resource mapper or in another resource mapper instantiated somewhere else in your game. This will be clearer with a practical example further ahead.
The Clear function basically performs the erasure of the entire resource map, using the same counted reference mechanism from the resource database.
The Get function retrieves the named resource by specifing its resource name and gives back the resource pointer.
The Initialise function attaches the resource mapper to the resource database.
Example of Usage
First of all, we need a base class, which could be a game object. Let's call it foo just for example
class CFoo : public vml::CResourceBase { //////////////////////////////////////////////////// // copy constructor is private // no copies allowed since classes // are referenced CFoo( const CFoo &foo ) : CResourceBase ( foo ) { } //////////////////////////////////////////////////// // overload operator is private, // no copies allowed since classes // are referenced CFoo &operator =( CFoo &foo ) { if ( this==&foo ) return *this; return *this; } public: //////////////////////////////////////////////// // ctor / dtor // this constructor must be present CFoo(const std::string &resourcefilename, void *args ) : CResourceBase( resourcefilename,args ) { } // regular base constructor and destructor Cfoo() {} ~CFoo() { } };
Now we can instantiate our resource database and resource mappers.
CResourceManager<CFoo> rm; CResourceMap<CFoo> mymap1; CResourceMap<CFoo> mymap2;
I have createed a resource manager and two resource mappers here.
// create a resource database rm.Initialise("FooDatabase", vml::CResourceManager<CFoo>::VERBOSE ); // attach this database to the resource mappers, bot of them allowd duplicates mymap1.Initialise( "foolist1",&rm, true,true ); mymap2.Initialise( "foolist2",&rm, true,true ); // populate first resoruce mapper // the '0' argument means that the resource 'a' , whose filename is foo1.txt // doesn't take any additional values at construction time mymap1.Add( "a","foo1.txt",0 ); mymap1.Add( "b","foo1.txt",0 ); mymap1.Add( "c","foo2.txt",0 ); mymap1.Add( "d","foo2.txt",0 ); mymap1.Add( "e","foo1.txt",0 ); mymap1.Add( "f","foo1.txt",0 ); mymap1.Add( "g","foo3.txt",0 ); // populate second resource mapper mymap2.Add( "a","foo3.txt",0 ); mymap2.Add( "b","foo1.txt",0 ); mymap2.Add( "c","foo3.txt",0 ); mymap2.Add( "d","foo1.txt",0 ); mymap2.Add( "e","foo2.txt",0 ); mymap2.Add( "f","foo1.txt",0 ); mymap2.Add( "g","foo2.txt",0 ); // dump content into a stl string which can be printed as you like std::string text=rm.Dump();
Running this example and printing the text content gives:
Dumping database foodatabase Filename : foo1.txt , references : 7 Filename : foo2.txt , references : 4 Filename : foo3.txt , references : 3
This concludes the article. I hope it will be useful for you, thanks for reading.