AvtFileFormat plugin objects and ST/SD type plugins
One of the really cool things about VisIt is that the same basic I/O infrastructure supports database file formats that either do or do not support multiple mesh pieces (e.g. domains) and do or do not support multiple timesteps.
There are four base classes for file format objects
- avtSTSDFileFormat for files that support only single timestep and only a single domain
- avtMTSDFileFormat for files that support multiple timestep but only a single domain
- avtSTMDFileFormat for files that support only a single timestep but multiple domains
- avtMTMDFileFormat for files that support multiple timesteps and multiple domains
VisIt sees all databases as being multiple timestep and multiple domain in nature. The avtFileFormat classes help provide the abstractions when the underlying database itself does not.
The way VisIt does this, particularly in parallel, and some of the possible issues the ensue is the main subject of this page.
In parallel, each processor creates all avtFileFormat objects for the database in XXXCommonPluginInfo::SetupDatabase over all times and all domains.
For a really nifty MTMD plugin, that is only one avtFileFormat object that gets instanced on each processor.
But for a STSD plugin with 1000 domains and 1000 timesteps (this isn't necessarily a very likely scenario as STSD formats are typically unwieldly for such large databases), each processor will instance 1,000,000 file format objects. As an aside, this is one reason doing any work in a avtFileFormat constructor other than initializing data members is probably a bad idea. If you open a file in a constructor, then that will be happening 1,000,000 times and on each processor. This represents a scalability problem that I think we'll have to address in the not too distant future.
Queries from VisIt for Database Metadata occur on a per timestep basis. VisIt believes any domain will do to serve up database metadata and so only ever calls PopulateDatabaseMetadata for domain 0.
So, that means only that avtFileFormat instance for domain 0 can assume anything that it gains knowledge of via PopulateDatabaseMetadata can be saved and later used in future queries from VisIt such as GetMesh or GetVar.
All other avtFileFormat instances cannot make that assumption because the PopulateDatabaseMetadata methods in their instances are never called.
Queries from VisIt for GetMesh or GetVar occur on a per domain basis, not per timestep. Worse as load balancer varies assignment of domains to processors, the particular set of avtFileFormat instances on a given processor used for one pipeline execution may vary to the next.
This means that GetMesh and GetVar can be called on SD plugins who have not yet themselves executed PopulateDatabaseMetadata. If you write a plugin assuming whatever is learned in PopMD can be used in GetMesh/GetVar, you'll be seriously confused.
I ran into this in my re-write of the Exodus plugin. The older version of the plugin had what amounted to a call to PopulateDatabaseMetadata as the first thing it did inside of GetMesh or GetVar if it had not already been called. But, that means each processor is hitting up the file for information it very likely already has inside of some other instance on that same processor.
And, it may do this many times as different instances get triggered as domain assignment to processors varies. If PopMD makes a habit of storing up a bunch of information read from the file (it has to at least create a fully populated avtDatabaseMetadata object) then that results in a potential accumulation of large redundant objects on each processor.