DashVault 'My Investments' or CAFS reported "down"

DashVault 'My Investments' or CAFS reported "down"

In a situation where we receive a report for a 'My Investments' portal outage (aka CAFS or something along the lines that DashVault can't receive account infomration), there are a series of components which should be checked for integrity.

AWS - specifically Lambda, S3, CloudFront, RDS, EC2 (us-east-2 / Ohio)
Keycloak - a pair of EC2 instances running in the CACU prod AWS account, combined with a HA postgres RDS
DashVault Proxy Function - a Lambda function in the us-east-2 region / CACU AWS prod account which performs calls to CACU's Mulesoft enterprise service bus via HTTPS.  Look for a sharp increase in the error count metrics or peruse the function invocation logs looking for authorization errors or connectivity errors to Mulesoft.

Downstream systems upon which the solution relies:
- CACU Mulesoft cloud endpoint  (rarely goes down, but possible) [contact Eldon Rousch]
- Salesforce (less reliance on this today, but is still critical to look up accounts and their inter-dependencies) [contact Rachael Aitkens]
- Orion (primary source for account information) [contact Rachel A, Rob S]

Key Contacts:

[Technical]
Eldon Rousch  (Mulesoft administrator, Technical architect)
Rachael Aitkens (Salesforce Administrator, key process knowledge)
Mark Allen (Enterprise Architect)

[Business]
Mike Haggerty (President, CAFS)
Justin Steitz (VP, CAFS)
Phil Bess (Director, Ops - CAFS)
Rob Swirbul (Project Management)
Anita Newton (CIO, CACU/CUSO1)

Key Links:


    • Related Articles

    • CloudWatch Alert: Cacu-Keycloak-Db-Prep-Exception

      Keycloak has been observed to reach an invalid internal database driver state which prevents the software from working normally.  It may show up in CloudWatch Logs for /srv/keycloak-3.4.3.Final/standalong/log/server.log as an error for "could not ...