We havent started to use automated tests with XbyK but like you it is high on my list and I was looking through the documentation just last week for what is available within the platform. Our human QC test plans for dev/staging just hit 400 tests but our smoke plans for production I have kept to under 20 tests and we run a smoke test at any PROD code change.
Operational problems - we are building honeypot fields both into forms and shopping carts and IP usage restrictions in. i.e. 3 uses in 5 mintues and you are blocked for the month, you use this hidden field and you are blocked for 3 hours with an onscreen message to say call us if you are a human to resolve this issue. This is saving us the most time. Stopping nuisance that gets through the WAF and captcha and fills databases with rubbish.
I would love to know how you do API reachability and latency - because my devops always tell me it isnt possible to alert on an API outage - usually because of firewall constraints. I do get a response email when it is up, or when the nightly/weekly batch succeeded and all is well. But not when it the API is down or the process failed and I really struggle to notice the absence of the success messages when it does occur. I would love to learn more.
What you have said is a really good list but as the production/ project manager I would add these:
5 min cadence on your uptime tool of choice for these three
Footer sentence load - footers are below the page and scheduled to load late, they change infrequently (so dont falsely alert often) it is also important to monitor them on home but functional pages (with forms or ecommerce) for example. The sentence load time gives you better confidence than a 200 return in my opinion because it proved the whole page loaded reasonable time.
Form Submit button text - complex business critical forms 5 min monitoring of the save button. It also helps load time for the forms keeping them in cache.
Search with a complex query - i.e. on a site where search has had faults we not only probe search loads but undertake a complex search and seek a sentence return on a page we know is permanant. This helped us troubleshoot problems with the search indexing in K13 and we have stuck with this as a good monitoring practice in XbyK, especially as we are delivering filterable search now with XbyK and taxonomies.
I am thinking that monthly is the right cadence for these
Widget use - pick your 2 most complicated widgets - we are finding the widgets the most fragile part of our layer. They are breaking most often on refresh from Kentico. Dont just do the code test, the test needs to be create a page and add the widget in the page builder with configuration end to end (a computer use agent can be trained to do that for you as a user would do it but we only use computer use agents in dev environments without user data in them - see my other contribution to a chat today - https://community.kentico.com/q-and-a/q/anyone-running-coding-agents-headlessly-claude-code-routines-etc-on-xperience-projects-736a260d )
Ecommerce successful payments and failed payment with email verification. We have hidden low cost products to achieve this and the client knows if it sees a submission from Leroy Llama that is us doing our testing. I work the test costs into our retainer. Dont forget a failed card - you want to see the onscreen messages.
Form fills - especially as we have custom field types and functions like draft save, I have playwright scripts for business forms. Again Leroy Llama completes the form - they know it is a test and to ignore but to check the CRM for successful transfer.
AI screenshot comparisons on sentinel pages - I have built and am productising right now an agent that sets the baseline for the site after first release and checks periodically for issues. I have been working on the agent for about 6 months to replace the human QC but I am still working out how much token use is worth it and where to deploy it- https://community.kentico.com/q-and-a/q/anyone-running-coding-agents-headlessly-claude-code-routines-etc-on-xperience-projects-736a260d )
