Read the background: https://blog.mdriven.net/cache-invalidation-a-real-problem-for-us-all/

Update 2022-11-29 - to clear cache invalidation data and start over

To clear every cache module you need to delete in the database accordingly:

delete from MDrivenCache_ManifestHead
delete from MDrivenCache_ManifestRow

Deleting the heads will make objects of relevant classes re-create the cache modules still marked for cache invalidation

Admin and Update - Update 2021 January

CacheInvalidation maintaince view in MDrivenServer:

In this UI you can find out the Latest or oldest Invalidated, and updated cache modules. You can list all the manifest rows for one module. You can delete a module if needed.

Update to strategy 20200312

Changes to cache invalidation. Save writes to recentlyupdatedtable, and then this table was joined with cache manifest that could take time - and this locked the recentlyupdatedtable table that in turn limited other saves for the complete system.

To fix this a new table has been added recentlyupdatedtableWorkPart. We quickly move seen rows in recentlyupdatedtable to recentlyupdatedtableWorkPart to minimize the time recentlyupdatedtable is in a transaction. Then we do the potential heavy work from the recentlyupdatedtableWorkPart table.

The EnsureCacheInvalidationTables functions adds the recentlyupdatedtableWorkPart table even if it is the only one missing.

Orginal strategy

MDrivenCacheInvalidationAdmin

• LastInvallidationLoop
• LastUpdateLoop

MDrivenCacheInvalidationRecentlyUpdated

• id
• classname
• membername
• time

MDrivenCacheManifestHead

• ViewModel
• RootId
• Class
• viewmodelMetaChecksum // set on update
• priority // set by viewmodel property - this way information can control importance!
• Updated     // set on update
• Invalidated

MDrivenCacheManifestRow

• Id
• Class
• MemberName

Upon normal save

new rows indicating member level change are inserted in MDrivenCacheInvalidationRecentlyUpdated

Special treatment of newly created and deleted objects - they get the **created and **deleted flag

For changed embedded links (single end) we also add dirty of other end

Upon InvalidationLoop

we look for direct hits on id and attribute and invalidate

we look for any objects of class that has been changed recently that now should be part of the set

(ie other end of OptionalAssociationEndNameForBeingIncluded , does it point to id -> invalidate)

we look for manifest rows indicating use of allinstances and match to **created and **deleted and invalidate

we look for meta checksum change of viewmodel

Upon UpdateLoop

We look for Invalidated that has the diff between Now and Invalidated higher than allow oldness, we sort these by priority.

For each cache found we instansiate the named ViewModel with for the given root object.

We execute all actions on the ViewModel

We save all the used and hence dependant model usage during cache - allowing us to set persisted calculated fields and create persisted derived objects.

We ignore use of common super class members to avoid having caches be invalidate by un-precise things ie changetime

Things to think about

How do we ensure initial creating of cache - when a new cache is created that already have existing root objects?

Is the current polled criteria based SSVM-Run helpful? Is it the criteria based expression that discovers new cache needs or is it based on type alone?  ! Type alone - criteria will be hidden in UI and ignored in RT

Is the cache type-wise complete - or only on matching the polling criteria? ! Type complete - all objects for type will get a cache manifest

If we want to involve the polling criteria (ocl-ps) with creation need - do we also want it with refresh need? ! No

- If we involve the polling criteria in refresh decision we may end up with a invalidated caches that never gets updated and that is not good.

- It may be easier for user to completely separate the polled criteria ViewModels from the auto invalidated, mixing the two reasons for running may get confusing

- We still want the polling periodicity for update discovery to be controllable by user and also the chunking ! Yes - chunking and periodicity still applies

Anyway we look at it - we need someway to join in the cachehead to make use of the invalidated value in order to select viewmodels to create/refresh

select d.a0_detailid from a0_detail d where CAST(d.a0_detailid AS nvarchar(30)) not in (select rootid from a0_mdrivencache_manifesthead where viewmodel='ViewModelServerSideDetail')

This limits the possible RootObjects in cached viewmodels to having a single key column.

The discover of missed cached roots only creates invalidated heads - then the update process discovers, orders and executes updates.

Special ViewModel considerations

The ViewModel used in cache has the option to use the following attributes with special meaning:

CacheIgnoreClasses - set of strings that corresponds to name of classes to ignore in manifest and hence in invalidation. Example: Set{User.asstring,Singleton.asstring,SysSingleton.asstring,AcoSuper.asstring}

Priority - integer written to head and used with order by by next update loop 1 is higher priority than 1000 (default on update is 1000, on new cacheneed found prioritu is 1)

Practical queries

What has changed but not yet made it to check if it will invalidate anything

select count(*) from MDrivenCache_RecentlyUpdated

Same as above grouped on class and member:

select count(*),classname,membername from MDrivenCache_RecentlyUpdated group by classname,membername
order by classname,membername