NethVoice: Monitor service availability #8093

Description

Add the first set of built-in alerts for NethVoice and NethVoice proxy, as part of #8066, so administrators know about problems before users report them.

Each module exports its own metrics and publishes its own alert rules. The node runs the export-metrics command of each module every 5 minutes (NethServer/ns8-core#1365). The metrics app evaluates the rules stored by the module (NethServer/ns8-metrics#91) and sends notifications with its existing portal and email settings.

The full list of alerts, with severity and conditions, is in this comment.

Scope

• Services: ServiceDown for the NethVoice services (FreePBX/Asterisk, MariaDB, Janus, CTI, Phonebook, Tancredi, Satellite, Reports, SFTP, SAML2/OAuth2 proxies) and the proxy services (kamailio, postgres, redis, rtpengine). Service state is read from systemd, not from podman.
• Trunks: NethVoiceTrunkUnreachable. NethVoice checks trunks that bypass the proxy; NethVoice proxy checks UDP trunks that go through it. TCP and TLS trunks through the proxy are not checked yet.
• DNS: NethVoiceDnsFailure and NethVoiceDnsSlow for the NethVoice, CTI and proxy names, checked on the DNS resolver of the node (/etc/resolv.conf).
• AI: NethVoiceAiKeyInvalid and NethVoiceAiCreditExhausted for the Deepgram and OpenAI keys used by transcription.
• Alerts are off by default and are turned on per module from the "Alerts" card of the Settings page (set-alerts action). The list of alerts and their thresholds cannot be changed.

Call quality and configurable thresholds belong to the parent issue.

Acceptance criteria

• With alerts off (default), no rules are published, but metrics are still exported. Turning alerts on publishes the rules; turning them off removes them.
• Rules are published at module create, update, restore and clone, and removed when the module is removed. Rules of other modules are not touched.
• ServiceDown fires with reason=stopped when a service is stopped for 10 minutes, or with reason=crash_loop when systemd restarts it at least 3 times in 10 minutes. Disabled services and services never started are not watched.
• NethVoiceTrunkUnreachable fires when a trunk stays unregistered or its provider does not answer for 10 minutes.
• NethVoiceDnsFailure fires when a name does not resolve (reason=not_resolved) or the proxy name does not point to the proxy IP (reason=ip_mismatch) for 15 minutes. NethVoiceDnsSlow fires when the resolver is slower than 3 seconds or does not answer for 15 minutes.
• NethVoiceAiKeyInvalid fires when transcription is on and the provider rejects the API key for 10 minutes. NethVoiceAiCreditExhausted fires when Satellite reports a call refused for lack of credit, and clears after the next successful call.
• A check that cannot run (for example AI provider unreachable or FreePBX stopped) is reported with nethvoice_exporter_collector_up = 0 (nethvoice_proxy_exporter_collector_up for the proxy) and raises no alert.
• Alerts show the service or trunk, module instance and node, in English and Italian.
• Requires core 3.24.0 or later and metrics app 1.7.0 or later. With an older metrics app the rules are ignored.

Verification

• Unit tests for the metric collectors and rule publishing, and promtool tests for the alert rules, in both repositories.
• Robot tests in CI for export-metrics, set-alerts and rule publishing.
• Manual QA on a test cluster: services stopped and in crash loop, trunk failures, DNS not_resolved and ip_mismatch, invalid AI key, and rule cleanup on module removal.

References

• NethVoice: feat(alerts): add first NethVoice alerts nethesis/ns8-nethvoice#1067
• NethVoice proxy: feat(alerts): add ServiceDown and trunk alerts nethesis/ns8-nethvoice-proxy#231
• Core, periodic export-metrics: feat(metrics): publish module metric files ns8-core#1365
• Metrics, module alert rules: feat(alerts): allow module-provided rules ns8-metrics#91
• Module-provided alert rules: Allow applications to provide their own alert rules #8100
• Parent issue: NethVoice: Configurable Alerting for Critical NethVoice Events #8066
• Alert list: NethVoice: Monitor service availability #8093 (comment)
• Superseded (systemd-exporter approach): Add systemd Prometheus exporter nethesis/ns8-nethvoice#907, feat(metrics): add systemd exporter nethesis/ns8-nethvoice-proxy#194