Package com.ssgllc.fish.service.util.published

Class AnomalyDetectionUtil

java.lang.Object
com.ssgllc.fish.service.util.published.AnomalyDetectionUtil
@Component public class AnomalyDetectionUtil extends Object
Anomaly detection for scripts: time-series aberration detection (the EWMA, adaptive regression and regression/EWMA switch detectors) and the space-time permutation scan for clusters.

Called as the anomalyDetectionUtil bean. Results are maps and lists, so a script can read them and save them with entityUtil directly. The input is rows. To use the rows of a report, get them with reportUtil.runReport:

def points = anomalyDetectionUtil.detectTimeSeries(reportUtil.runReport("DailyCaseCounts", [DISEASE: diseaseId]), "SWITCH", [:])

There is no REST endpoint for these functions. To give a result to a dashboard widget, call the function from the post-process script of the report of the widget (report config postProcessScript). The script gets the rows of the query as rows and gives the rows that the widget shows:

return anomalyDetectionUtil.detectTimeSeries(rows, input.params.DETECTOR, [period: input.params.PERIOD])

  • Constructor Details

    • AnomalyDetectionUtil

      public AnomalyDetectionUtil(com.ssgllc.fish.service.anomaly.AnomalyDetectionService anomalyDetectionService)

  • Method Details

    • detectTimeSeries

      public List<Map<String,Object>> detectTimeSeries(List<Map<String,Object>> rows, String detector, Map<String,Object> options)
      Tests each day of a daily time series for a count that is higher than its recent baseline predicts.

      Each result row has: series, date (yyyy-MM-dd), count, expected, threshold (the smallest count that gives a high alert), lowThreshold (the largest count that gives a low alert), direction (the side of the result: HIGH or LOW), testStatistic, pValue, alert (ALERT for p below 0.01, WARNING for p below 0.05, NONE, or NO_BASELINE for the first days) and detector (the detector used for that day). Days with no row get a count of zero.

      Option direction: HIGH (default) finds counts above the baseline, LOW finds counts below it (for example a site that stops sending data), and BOTH runs the two tests. With BOTH, chance alone gives two times the alerts of one side.

      A length of time is a count and a unit: text such as 3 DAY, 7 WEEK, 6 MONTH or 1 YEAR, a map such as [count: 6, unit: "MONTH"], or a number of days.

      Option period: the length of time of one tested point, default 1 DAY. With a longer period the daily counts are summed into periods that end on the end date, and each row is one period: date is its first day and endDate its last. Periods of months and years go back from the end date (a series that ends on November 11 has the period October 12 to November 11). They do not have the same number of days, so the test (and its testStatistic and pValue) uses the count for each day, and the expected count and the thresholds of a row are for the days of that period. A period longer than a day uses the EWMA detector.

      Options baseline and guardBand: the history that a point is compared with, and the gap between that history and the point, as lengths of time. baselineDays and guardBandDays are the same options as numbers of days. For a period of one day the defaults are 28 days and 2 days. For a longer period the defaults are 12 weeks (and 8 periods or more) and 1 period, and the baseline must be from 7 to 520 periods.

      Groovy example:
      def rows = [[date: "2026-10-01", count: 3], [date: "2026-10-02", count: 5]] // and so on: 31 days or more
      def points = anomalyDetectionUtil.detectTimeSeries(rows, "SWITCH", [baselineDays: 28, guardBandDays: 2])
      def last = points.last()
      if (last.alert == "ALERT") { log.info("Alert on ${last.date}: ${last.count} cases, expected ${last.expected}") }
      def drops = anomalyDetectionUtil.detectTimeSeries(rows, "EWMA", [direction: "LOW"]).findAll { it.alert == "ALERT" }
      def weeks = anomalyDetectionUtil.detectTimeSeries(rows, null, [period: "1 WEEK", baseline: "12 WEEK"])
      def months = anomalyDetectionUtil.detectTimeSeries(rows, null, [period: "1 MONTH"])

      Parameters:
      rows - one row for each day, with date (yyyy-MM-dd or a date) and count; optionally series (each value is an independent series) and total
      detector - SWITCH (default when null), EWMA or REGRESSION
      options - optional: baselineDays (default 28; 7 or more for EWMA; 14 or more and a multiple of 7 for SWITCH and REGRESSION), guardBandDays (default 2), w1 (default 0.4) and w2 (default 0.9) for EWMA, endDate (default: the last date in the rows), direction, period, baseline and guardBand. May be null. A series has at most 3660 days, and all series together at most 100000 days.
      Returns:
      one row for each day of each series, in date order

    • spaceTimeScan

      public Map<String,Object> spaceTimeScan(List<Map<String,Object>> rows, Map<String,Object> options)
      Runs a prospective space-time permutation scan: it finds groups of cases that are close together and in the most recent days, and are more numerous than the usual counts of those locations and days predict. It needs no population data.

      The result has startDate, endDate, totalCases, locationCount, replications, skippedRows (rows with no location, no date, or no valid coordinates) and clusters. Each cluster has rank, centerLocationId, centerLatitude, centerLongitude, radiusKm, locationIds, startDate, endDate, observed, expected, logLikelihoodRatio, pValue, pValueMethod (MONTE_CARLO or GUMBEL), recurrenceIntervalDays (1 / pValue: with one scan each day, chance alone gives a cluster this strong once in this number of days) and caseIds.

      Groovy example:
      def rows = [[caseId: "a1", locationId: "25025010103", date: "2026-11-08", latitude: 42.36, longitude: -71.06]]
      def result = anomalyDetectionUtil.spaceTimeScan(rows, [maxTemporalDays: 30, replications: 999])
      def signals = result.clusters.findAll { it.recurrenceIntervalDays >= 100 }

      Parameters:
      rows - one row for each case, with caseId, locationId, date (yyyy-MM-dd or a date), latitude and longitude. All rows of one location must have the same coordinates. A row with no coordinates gets them from the option locations.
      options - optional: endDate (default: the latest case date), studyPeriodDays (365), minTemporalDays (2), maxTemporalDays (30), maxSpatialPercent (50), maxRadiusKm (no limit), minCases (2), replications (999), seed (random), dayOfWeekAdjustment (false), maxClusters (10), locations (a location table: rows with locationId, latitude and longitude, for example one row for each ZIP code or facility, or the rows of a report: locations: reportUtil.runReport("ZipCentroids", [:])). May be null. Each of the three options in days has a second form with no Days on its name that takes a length of time: studyPeriod: "1 YEAR", minTemporal: "2 DAY", maxTemporal: "1 MONTH". Months and years are counted back from endDate, and have their mean length (30 and 365 days) when the options give no endDate. Limits: replications 9999, studyPeriodDays 3660, maxTemporalDays 366, 5000 locations.
      Returns:
      the scan result with its clusters, most likely first