Read the background: Cache Invalidation–a real problem for us all

Update 2024-01-11: Integer is not big enough

Some tables involved in MDrivenCache that have a lot of transactions will suffer from the 2+billion limit of the int Identity column. The tables concerned primarily are these:

• MDrivenCache_ManifestRow
• MDrivenCache_RecentlyUpdated
• MDrivenCache_RecentlyUpdatedWorkPart

Since we do not actually care for the primary key on these tables we can drop them and re-create them:

-- the below will fail due to index dependency - drop those indexes and rerun
ALTER TABLE MDrivenCache_ManifestRow DROP COLUMN rowid
ALTER TABLE MDrivenCache_ManifestRow ADD rowid BIGINT primary Key IDENTITY(1,1)

-- the below will fail due to index dependency - drop those indexes and rerun
ALTER TABLE MDrivenCache_RecentlyUpdated DROP COLUMN rowid
ALTER TABLE MDrivenCache_RecentlyUpdated ADD rowid BIGINT primary Key IDENTITY(1,1)

-- the below will fail due to index dependency - drop those indexes and rerun
ALTER TABLE MDrivenCache_RecentlyUpdatedWorkPart DROP COLUMN rowid
ALTER TABLE MDrivenCache_RecentlyUpdatedWorkPart ADD rowid BIGINT primary Key  --- note this is NOT a identity column

MDrivenServer will need to be updated in order to be able to move from MDrivenCache_RecentlyUpdated to MDrivenCache_RecentlyUpdatedWorkPart as it makes use of the rowid key - and new versions handle the value as a Int64 rather than as an Int32.

Note: An acute short-term fix for a system halting for the above 2+billion reason can simply drop the rowid and re-introduce it as Int32 since the recreation will shrink the max identity to the actual used values. This is much less than the Int32 limit.

Update 2022-11-29: To Clear Cache Invalidation Data and Start Over

To clear every cache module, you need to delete in the database accordingly:

delete from MDrivenCache_ManifestHead
delete from MDrivenCache_ManifestRow

Deleting the heads will make the objects of relevant classes re-create the cache modules still marked for cache invalidation.

Admin and Update: Update January 2021

CacheInvalidation maintenance view in MDrivenServer:

In this UI, you can find the Latest or oldest invalidated and updated cache modules, list all the manifest rows for one module, and delete a module if needed.

Update to Strategy 2020.03.12

Changes to Cache Invalidation

Save writes to the recentlyupdatedtable, and then this table was joined with a cache manifest that could take time - and this locked the recentlyupdatedtable that in turn limited other saves for the complete system.

To fix this, a new table has been added to recentlyupdatedtableWorkPart. We quickly move seen rows in the recentlyupdatedtable to recentlyupdatedtableWorkPart to minimize the time recentlyupdatedtable is in a transaction. Then we do the potential heavy work from the recentlyupdatedtableWorkPart table.

The EnsureCacheInvalidationTables function adds the recentlyupdatedtableWorkPart table, even if it is the only one missing.

Original Strategy

MDrivenCacheInvalidationAdmin

• LastInvallidationLoop
• LastUpdateLoop

MDrivenCacheInvalidationRecentlyUpdated

• id
• classname
• membername
• time

MDrivenCacheManifestHead

• ViewModel
• RootId
• Class
• ViewmodelMetaChecksum // set on update
• priority // set by ViewModel property - this way information can control importance!
• Updated     // set on update
• Invalidated

MDrivenCacheManifestRow

• Id
• Class
• MemberName

Upon Normal Save

New rows indicating member level change are inserted in MDrivenCacheInvalidationRecentlyUpdated.

Special treatment of newly created and deleted objects - they get the **created and **deleted flag

For changed embedded links (single end), we also add the dirty of the other end.

Upon InvalidationLoop

We look for direct hits on id and attribute and invalidate.

We look for any objects of a class that have been changed recently that should now be part of the set.

(i.e. the other end of OptionalAssociationEndNameForBeingIncluded, does it point to id -> invalidate).

We look for manifest rows indicating the use of allinstances and match to **created and **deleted and invalidate.

We look for meta checksum changes in the ViewModel.

Upon UpdateLoop

We look for the Invalidated which has the diff between Now and Invalidated higher than allow oldness and we sort these by priority.

For each cache found, we instantiate the named ViewModel with the given root object.

We execute all actions on the ViewModel.

We save all the used and hence dependant model usage during cache - allowing us to set persisted calculated fields and create persisted derived objects.

We ignore the use of common superclass members to avoid invalidating caches by un-precise things i.e. changetime.

Things to Think About

How do we ensure the initial creation of the cache when a new cache is created that already has existing root objects?

Are the current polled criteria-based SSVM-Run helpful? Is it the criteria-based expression that discovers new cache needs or is it based on type alone?! Type-alone criteria will be hidden in UI and ignored in RT.

Is the cache type-wise complete - or only on matching the polling criteria?! Type complete - all objects for type will get a cache manifest

If we want to involve the polling criteria (OCL-PS) with a creation need - do we also want it with a refresh need? No!

- If we involve the polling criteria in the refresh decision, we may end up with invalidated caches that never get updated and that is not good.

- It may be easier for the user to completely separate the polled criteria ViewModels from the auto-invalidated. Mixing the two reasons for running may get confusing.

- We still want the polling periodicity for update discovery to be controllable by the user and also the chunking! Yes - chunking and periodicity still apply.

Any way we look at it, we need some way to join in the cache head to make use of the invalidated value in order to select ViewModels to create/refresh.

select d.a0_detailid from a0_detail d where CAST(d.a0_detailid AS nvarchar(30)) not in (select rootid from a0_mdrivencache_manifesthead where viewmodel='ViewModelServerSideDetail')

This limits the possible RootObjects in cached ViewModels to having a single key column.

The discovery of missed cached roots only creates invalidated heads - then the update process discovers, orders, and executes updates.

Special ViewModel Considerations

The ViewModel used in the cache has the option to use the following attributes with special meanings:

CacheIgnoreClasses - set of strings that correspond to the name of classes to ignore in the manifest and hence in invalidation. Example: Set{User.asstring,Singleton.asstring,SysSingleton.asstring,AcoSuper.asstring}

Priority - integer written to the head and used with an order by next update loop - 1 is a higher priority than 1000 (default on the update is 1000, on new cache need found priority is 1).

Practical Queries

What has changed but not yet made it? To check if it will invalidate anything:

select count(*) from MDrivenCache_RecentlyUpdated

Same as above, grouped on class and member:

select count(*),classname,membername from MDrivenCache_RecentlyUpdated group by classname,membername
order by classname,membername