View source for Documentation:CacheInvalidation

<message>Write the content here to display this box</message>
Read the background: [https://blog.mdriven.net/cache-invalidation-a-real-problem-for-us-all/ Cache Invalidation–a real problem for us all]

==== Some gotchas documented after debugging ====
I noticed that some obvious things used by the ViewModel was missing from the manifest - and thus the changes on this were not detected as an invalidation. I suspected a major problem - but debugging made me realize my mistake. Things changed during Cache refresh will not be in the manifest (to avoid pumping loops where a cache module invalidates itself). 

I had a use case were an action in the UI nulled out an Attribute TimeRefreshed - and the cache module set this time to DateTime.Now - me thinking that nulling it out from the UI would refresh the module - but as this is an attribute written by the module itself it is not part of the manifest. 

The solution is to have one attribute that the UI sets and cache module reads but does not change... Like if the UI sets TimeWhenUserWantsThisRefreshed - and Cache just sees this like self.TimeWhenUserWantsThisRefreshed.notnull or any other access except a changing one- then TimeWhenUserWantsThisRefreshed will be in the manifest, the CacheModule may then write an attribute TimeRefreshed , but external changes to this will not trigger cache invalidation.

Also: members on the default superclass will also be ignored. Since these are so abstract and broad you should not rely on them for surgical cache invalidation triggering (would be costly).

=== Update 2025-01-27: Debugger updates ===
[[File:Documentation CacheInvalidation 1737965684653.png|alt=Cache invalidation in debugger|frameless|556x556px]]

Describing video: https://youtu.be/W6xYAw5gBQQ

=== Update 2024-01-11: Integer is not big enough ===
Some tables involved in MDrivenCache that have a lot of transactions will suffer from the 2+billion limit of the int Identity column. The tables concerned primarily are these:

* MDrivenCache_ManifestRow
* MDrivenCache_RecentlyUpdated
* MDrivenCache_RecentlyUpdatedWorkPart

Since we do not actually care for the primary key on these tables we can drop them and re-create them:
 -- the below will fail due to index dependency - drop those indexes and rerun
 ALTER TABLE MDrivenCache_ManifestRow DROP COLUMN rowid
 ALTER TABLE MDrivenCache_ManifestRow ADD rowid BIGINT primary Key IDENTITY(1,1)

 -- the below will fail due to index dependency - drop those indexes and rerun
 ALTER TABLE MDrivenCache_RecentlyUpdated DROP COLUMN rowid
 ALTER TABLE MDrivenCache_RecentlyUpdated ADD rowid BIGINT primary Key IDENTITY(1,1)

 -- the below will fail due to index dependency - drop those indexes and rerun
 ALTER TABLE MDrivenCache_RecentlyUpdatedWorkPart DROP COLUMN rowid
 ALTER TABLE MDrivenCache_RecentlyUpdatedWorkPart ADD rowid BIGINT primary Key  --- note this is NOT a identity column
MDrivenServer will need to be updated in order to be able to move from MDrivenCache_RecentlyUpdated to MDrivenCache_RecentlyUpdatedWorkPart as it makes use of the rowid key - and new versions handle the value as a Int64 rather than as an Int32. 

'''Note''': An acute short-term fix for a system halting for the above 2+billion reason can simply drop the rowid and re-introduce it as Int32 since the recreation will shrink the max identity to the actual used values. This is much less than the Int32 limit.

=== Update 2022-11-29: To Clear Cache Invalidation Data and Start Over ===
To clear every cache module, you need to delete in the database accordingly:
 delete from MDrivenCache_ManifestHead
 delete from MDrivenCache_ManifestRow
Deleting the heads will make the objects of relevant classes re-create the cache modules still marked for cache invalidation.

=== Admin and Update: Update January 2021 ===
CacheInvalidation maintenance view in MDrivenServer:
[[File:2021-01-27 14h04 14.png|none|thumb|991x991px]]
In this UI, you can find the Latest or oldest invalidated and updated cache modules, list all the manifest rows for one module, and delete a module if needed.

=== Update to Strategy 2020.03.12 ===
The introduction video for cache invalidation is here: <nowiki>https://youtu.be/Pj2CK6Xkufs</nowiki>

'''Changes to Cache Invalidation'''

Save writes to the recentlyupdatedtable, and then this table was joined with a cache manifest that could take time - and this locked the recentlyupdatedtable that in turn limited other saves for the complete system.

To fix this, a new table has been added to recentlyupdatedtableWorkPart. We quickly move seen rows in the recentlyupdatedtable to recentlyupdatedtableWorkPart to minimize the time recentlyupdatedtable is in a transaction. Then we do the potential heavy work from the recentlyupdatedtableWorkPart table.

The EnsureCacheInvalidationTables function adds the recentlyupdatedtableWorkPart table, even if it is the only one missing.

=== Original Strategy ===
''MDrivenCacheInvalidationAdmin''
* LastInvallidationLoop
* LastUpdateLoop  
''MDrivenCacheInvalidationRecentlyUpdated''
* id
* classname
* membername
* time  
''MDrivenCacheManifestHead''
* ViewModel
* RootId
* Class
* ViewmodelMetaChecksum // set on update
* priority // set by ViewModel property - this way information can control importance!
* Updated     // set on update
* Invalidated  
''MDrivenCacheManifestRow''
* Id 
* Class 
* MemberName  

==== Upon Normal Save ====
New rows indicating member level change are inserted in MDrivenCacheInvalidationRecentlyUpdated.

Special treatment of newly created and deleted objects - they get the **created and **deleted flag

For changed embedded links (single end), we also add the dirty of the other end.

==== Upon InvalidationLoop ====
We look for direct hits on id and attribute and invalidate.

We look for any objects of a class that have been changed recently that should now be part of the set.

(i.e. the other end of <code>OptionalAssociationEndNameForBeingIncluded</code>, does it point to id -> invalidate).

We look for manifest rows indicating the use of allinstances and match to **created and **deleted and invalidate.

We look for meta checksum changes in the ViewModel. 

==== Upon UpdateLoop ====
We look for the Invalidated which has the diff between Now and Invalidated higher than allow oldness and we sort these by priority.

For each cache found, we instantiate the named ViewModel with the given root object.

We execute all actions on the ViewModel.

We save all the used and hence dependant model usage during cache - allowing us to set persisted calculated fields and create persisted derived objects.

We ignore the use of common superclass members to avoid invalidating caches by un-precise things i.e. changetime.

=== Things to Think About ===
''How do we ensure the initial creation of the cache when a new cache is created that already has existing root objects?''

''Are the current polled criteria-based SSVM-Run helpful? Is it the criteria-based expression that discovers new cache needs or is it based on type alone?!'' Type-alone criteria will be hidden in UI and ignored in RT.

''Is the cache type-wise complete - or only on matching the polling criteria?!'' Type complete - all objects for type will get a cache manifest

If we want to involve the polling criteria (OCL-PS) with a creation need - ''do we also want it with a refresh need?'' No!

- If we involve the polling criteria in the refresh decision, we may end up with invalidated caches that never get updated and that is not good.

- It may be easier for the user to completely separate the polled criteria ViewModels from the auto-invalidated. Mixing the two reasons for running may get confusing.

- We still want the polling periodicity for update discovery to be controllable by the user and also the chunking! Yes - chunking and periodicity still apply.

Any way we look at it, we need some way to join in the cache head to make use of the invalidated value in order to select ViewModels to create/refresh.
 select d.a0_detailid from a0_detail d where CAST(d.a0_detailid AS nvarchar(30)) not in (select rootid from a0_mdrivencache_manifesthead where viewmodel='ViewModelServerSideDetail')
This limits the possible RootObjects in cached ViewModels to having a single key column.

The discovery of missed cached roots only creates invalidated heads - then the update process discovers, orders, and executes updates.

==== Special ViewModel Considerations ====
The ViewModel used in the cache has the option to use the following attributes with special meanings:

'''CacheIgnoreObjects''' - you may have many  columns starting with this. Return object list. The list of objects will NOT be added to the manifest

'''CacheIgnoreClasses''' - set of strings that correspond to the name of classes to ignore in the manifest and hence in invalidation. Example: Set{User.asstring,Singleton.asstring,SysSingleton.asstring,AcoSuper.asstring}

'''Priority''' - integer written to the head and used with an order by next update loop - 1 is a higher priority than 1000 (default on the update is 1000, on new cache need found priority is 1).

=== Practical Queries ===
''What has changed but not yet made it?'' To check if it will invalidate anything: 
 select count(*) from MDrivenCache_RecentlyUpdated
Same as above, grouped on class and member:
 select count(*),classname,membername from MDrivenCache_RecentlyUpdated group by classname,membername
 order by classname,membername
Insights into what cachemodules you have and how many rows in each
 select h.viewmodel,count(*) from MDrivenCache_ManifestHead h, MDrivenCache_ManifestRow r where r.HeadID=h.Rowid  group by h.viewmodel
Insights into cachemodules manifest size average and maximum row count
 SELECT viewmodel, AVG(row_count) as averagecount ,Max(row_count) AS maxcount
 FROM (
 select h.viewmodel,r.headid,count(r.rowid) as row_count from MDrivenCache_ManifestHead h, MDrivenCache_ManifestRow r where r.HeadID=h.Rowid  group by h.viewmodel,r.headid
 ) AS subquery
 GROUP BY viewmodel;

{{Edited|July|12|2025}}


[[Category:Advanced-Friendly]]
[[Category:Caching and pre materialized views in the model layer]]
[[Category:Performance and Optimization]]
[[Category:Priority 70]]