Database structure: Difference between revisions

From VoIPmonitor.org
No edit summary
(VS-1789: Document nth_occurrence column in cdr_custom_headers table)
 
(24 intermediate revisions by the same user not shown)
Line 1: Line 1:
{{DISPLAYTITLE:CDR Database Table Schema}}
{{DISPLAYTITLE:CDR Database Table Schema}}


'''This document provides an overview of the most important tables and columns in the VoIPmonitor database schema. It is intended as a reference for administrators, developers, or anyone directly querying the database.'''
'''Database schema reference for VoIPmonitor. The <code>cdr</code> table stores one record per call; related tables store RTP statistics, custom headers, and metadata.'''


The core of the VoIPmonitor database is the `cdr` (Call Detail Record) table, which stores one primary record for each monitored call. Many other tables are linked to `cdr` to store additional, more detailed information.
= Schema Overview =


== Understanding Column Naming Conventions ==
<syntaxhighlight lang="sql">
Before diving into the columns, it's important to understand these common naming patterns:
-- List all CDR-related tables
*'''`a_` prefix:''' Refers to the "A-leg" or the '''caller's''' media stream (RTP sent from the caller).
SHOW TABLES LIKE 'cdr%';
*'''`b_` prefix:''' Refers to the "B-leg" or the '''callee's''' media stream (RTP sent from the callee).
</syntaxhighlight>
*'''`_mult10` / `_mult100` / `_mult1000`:''' Indicates that the stored integer value must be '''divided''' by this number to get the real decimal value. For example, a `jitter_mult10` value of `15` represents a real jitter of `1.5`.


== The `cdr` Table ==
<kroki lang="plantuml">
This is the main table containing the primary information for every call.
@startuml
skinparam shadowing false
skinparam defaultFontName Arial
skinparam linetype ortho


=== General Call Information ===
entity "cdr" as cdr {
;`ID`
  * ID : bigint <<PK>>
:The unique primary key for the record.
  --
;`id_sensor`
  id_sensor, calldate, callend
:The ID of the sensor that processed this call, corresponding to `id_sensor` in `voipmonitor.conf`.
  caller, called, duration
;`calldate`
  sipcallerip, sipcalledip
:The start time of the call, typically measured from the first `INVITE` packet.
  a_lost, b_lost, a_mos_f1_mult10
;`callend`
  lastSIPresponse_id <<FK>>
:The end time of the call, measured from the last packet associated with the call.
  a_ua_id, b_ua_id <<FK>>
;`bye`
}
:An indicator of how the call was terminated. `1` typically means a normal termination with a `BYE` request.
;`whohanged`
:Indicates which party sent the `BYE` message first (caller or callee).
;`lastSIPresponse_id`
:A reference to the `cdr_sip_response` table, linking to the last final SIP response for the call.


=== Call Timing & Duration ===
entity "cdr_next" as cdr_next {
;`duration`
  * cdr_ID <<PK,FK>>
:The total length of the call in seconds (`callend` - `calldate`).
  --
;`connect_duration`
  fbasename, match_header
:The length of the connected portion of the call in seconds, measured from the first `200 OK` to the `callend`.
}
;`progress_time`
:The time in seconds from the initial `INVITE` to the first provisional response (e.g., `180 Ringing`). This is often referred to as [[Glossary#Post_Dial_Delay_.28PDD.29|Post-Dial Delay (PDD)]].
;`first_rtp_time`
:The time in seconds from the start of the call to the detection of the very first RTP packet.


=== Participant Information ===
entity "cdr_rtp" as cdr_rtp {
;`caller` / `called`
  cdr_ID <<FK>>
:The user part of the number from the `From` and `To` SIP headers, respectively.
  --
;`caller_domain` / `called_domain`
  saddr, daddr, ssrc
:The domain part from the `From` and `To` SIP headers.
  received, lost, payload
;`caller_reverse` / `called_reverse`
}
:Reversed versions of the `caller` and `called` fields, used internally for efficient `LIKE '%search'` queries in the database.
;`callername`
:The display name part from the `From` SIP header.
;`sipcallerip` / `sipcalledip`
:The source and destination IP addresses from the SIP signaling. These are stored as integers; use the `INET_NTOA()` or `INET6_NTOA()` SQL function to convert them to a string representation.
;`a_ua_id` / `b_ua_id`
:A reference to the `cdr_ua` table, linking to the User-Agent string of the caller's and callee's device.


=== Network Quality Metrics ===
entity "cdr_proxy" as cdr_proxy {
==== Packet Loss ====
  cdr_ID <<FK>>
;`a_lost` / `b_lost`
  --
:The total number of RTP packets lost, detected by sequence number gaps.
  dst (proxy IP)
;`lost`
}
:The higher value between `a_lost` and `b_lost`.
;`packet_loss_perc_mult1000`
:The maximum packet loss percentage from either stream, multiplied by 1000.
;`a_sl1` - `a_sl10` / `b_sl1` - `b_sl10`
:A distribution of consecutive packet loss events. `a_sl1` counts the number of times a single packet was lost, `a_sl5` counts the number of times 5 packets were lost in a row, etc.


==== Jitter & Delay (PDV) ====
entity "cdr_sdp" as cdr_sdp {
;`a_avgjitter_mult10` / `b_avgjitter_mult10`
  cdr_id <<FK>>
:The average jitter calculated according to RFC3550, multiplied by 10.
  --
;`a_maxjitter` / `b_maxjitter`
  ip, port, is_caller
:The maximum jitter value observed during the call.
}
;`jitter_mult10`
:The higher `maxjitter` value between the two streams, multiplied by 10.
;`a_d50` - `a_d300` / `b_d50` - `b_d300`
:A distribution of Packet Delay Variation (PDV). `a_d50` counts packets with a delay between 0-50ms, `a_d70` counts packets with a delay between 50-70ms, etc.


==== MOS (Mean Opinion Score) ====
entity "cdr_siphistory" as cdr_siphistory {
;`a_mos_f1_mult10` / `b_mos_f1_mult10`
  cdr_ID <<FK>>
:The calculated [[Glossary#VoIPmonitor.27s_Parametric_MOS|parametric MOS score]] based on a simulated 50ms fixed jitter buffer, multiplied by 10.
  --
;`a_mos_f2_mult10` / `b_mos_f2_mult10`
  method, calldate, cseq
:MOS score based on a simulated 200ms fixed jitter buffer, multiplied by 10.
}
;`a_mos_adapt_mult10` / `b_mos_adapt_mult10`
:MOS score based on a simulated adaptive jitter buffer, multiplied by 10.
;`mos_min_mult10`
:The lowest MOS score recorded between the A and B streams, multiplied by 10.


==== RTCP Reported Metrics ====
entity "cdr_ua" as cdr_ua {
These metrics are sourced from RTCP reports sent by the endpoints themselves, as opposed to being calculated by the sniffer.
  * ID <<PK>>
;`a_rtcp_loss` / `b_rtcp_loss`
  --
:Total packets lost as reported by RTCP.
  ua (User-Agent string)
;`a_rtcp_avgfr_mult10` / `b_rtcp_avgfr_mult10`
}
:Average fraction loss reported by RTCP, multiplied by 10.
;`a_rtcp_avgjitter_mult10` / `b_rtcp_avgjitter_mult10`
:Average jitter reported by RTCP, multiplied by 10.


=== Other Information ===
entity "cdr_sip_response" as cdr_sip_response {
;`dscp`
  * ID <<PK>>
:Stores the DSCP values from SIP and RTP packets for Quality of Service (QoS) analysis.
  --
;`payload`
  lastSIPresponse
:The codec payload type number used during the call.
}


== Important Related Tables ==
cdr ||--|| cdr_next : "1:1"
cdr ||--o{ cdr_rtp : "1:N"
cdr ||--o{ cdr_proxy : "1:N"
cdr ||--o{ cdr_sdp : "1:N"
cdr ||--o{ cdr_siphistory : "1:N"
cdr }o--|| cdr_ua : "N:1"
cdr }o--|| cdr_sip_response : "N:1"
@enduml
</kroki>


; `cdr_next`
== Column Naming Conventions ==
:A 1-to-1 extension of the `cdr` table that stores additional data to keep the main `cdr` table smaller and faster.
:*'''fbasename:''' A string derived from the SIP `Call-ID`, used to link PCAP files to the CDR.
:*'''match_header:''' Stores the content of a custom header (defined by `matchheader` in `voipmonitor.conf`) used for linking call legs.


; `cdr_rtp`
{| class="wikitable"
:Stores detailed statistics for each individual RTP stream within a call, used by the GUI to render stream information in the call history view.
|-
! Pattern !! Meaning !! Example
|-
| <code>a_</code> || Caller's stream (A-leg) || <code>a_lost</code>, <code>a_mos_f1_mult10</code>
|-
| <code>b_</code> || Callee's stream (B-leg) || <code>b_lost</code>, <code>b_mos_f1_mult10</code>
|-
| <code>_mult10</code> || Divide by 10 || <code>jitter_mult10 = 15</code> → jitter = 1.5
|-
| <code>_mult100</code> || Divide by 100 || Percentage values
|-
| <code>_mult1000</code> || Divide by 1000 || <code>packet_loss_perc_mult1000</code>
|}


; `cdr_dtmf`
= The <code>cdr</code> Table =
:Stores detected DTMF key presses (if `dtmf2db` is enabled).


; `cdr_proxy`
== Call Information & Timing ==
:Stores the IP addresses of all SIP proxies that the call traversed.


; `cdr_sip_response`
{| class="wikitable"
:A lookup table storing the text of unique SIP responses (e.g., "404 Not Found"). The `cdr` table references this table by ID to save space.
|-
! Column !! Description
|-
| <code>ID</code> || Primary key
|-
| <code>id_sensor</code> || Sensor ID (from <code>voipmonitor.conf</code>)
|-
| <code>calldate</code> / <code>callend</code> || Start/end timestamps
|-
| <code>duration</code> || Total length in seconds
|-
| <code>connect_duration</code> || Connected time (after 200 OK)
|-
| <code>progress_time</code> || Time to first provisional response ([[Glossary#Post_Dial_Delay_.28PDD.29|PDD]])
|-
| <code>bye</code> || <code>1</code> = normal termination with BYE
|-
| <code>whohanged</code> || Which party sent BYE (caller/callee)
|-
| <code>lastSIPresponse_id</code> || FK to <code>cdr_sip_response</code>
|}


; `cdr_ua`
== Participant Information ==
:A lookup table storing the `User-Agent` strings of SIP devices.
 
{| class="wikitable"
|-
! Column !! Description
|-
| <code>caller</code> / <code>called</code> || Numbers from From/To headers
|-
| <code>caller_domain</code> / <code>called_domain</code> || Domain parts
|-
| <code>sipcallerip</code> / <code>sipcalledip</code> || IPs stored as integers (use <code>INET_NTOA()</code>)
|-
| <code>a_ua_id</code> / <code>b_ua_id</code> || FK to <code>cdr_ua</code> (User-Agent)
|}
 
== Quality Metrics ==
 
=== Packet Loss ===
{| class="wikitable"
|-
! Column !! Description
|-
| <code>a_lost</code> / <code>b_lost</code> || Total RTP packets lost
|-
| <code>packet_loss_perc_mult1000</code> || Max loss percentage × 1000
|-
| <code>a_sl1</code> - <code>a_sl10</code> || Distribution of consecutive loss events
|}
 
=== Jitter & Delay ===
{| class="wikitable"
|-
! Column !! Description
|-
| <code>a_avgjitter_mult10</code> || Average jitter × 10
|-
| <code>a_maxjitter</code> || Maximum jitter observed
|-
| <code>a_d50</code> - <code>a_d300</code> || PDV distribution buckets (0-50ms, 50-70ms, etc.)
|}
 
=== MOS (Mean Opinion Score) ===
{| class="wikitable"
|-
! Column !! Description
|-
| <code>a_mos_f1_mult10</code> || [[Comprehensive_Guide_to_VoIP_Voice_Quality#MOS_Calculation_in_VoIPmonitor|MOS]] with 50ms fixed buffer × 10
|-
| <code>a_mos_f2_mult10</code> || MOS with 200ms fixed buffer × 10
|-
| <code>a_mos_adapt_mult10</code> || MOS with adaptive buffer × 10
|-
| <code>mos_min_mult10</code> || Lowest MOS between A/B streams × 10
|}
 
=== RTCP-Reported Metrics ===
These come from endpoint RTCP reports, not calculated by the sniffer. See [[Call_Detail_Record_-_CDR#Understanding_Sniffer_vs_RTCP_Packet_Loss|Sniffer vs RTCP Packet Loss]].
 
{| class="wikitable"
|-
! Column !! Description
|-
| <code>a_rtcp_loss</code> || Packets lost (reported by endpoint)
|-
| <code>a_rtcp_avgfr_mult10</code> || Average fraction loss × 10
|-
| <code>a_rtcp_avgjitter_mult10</code> || Average jitter × 10
|-
| <code>a_rtcp_avgrtd_mult10</code> || '''Roundtrip delay (latency)''' in ms × 10
|}
 
= Related Tables =
 
== <code>cdr_next</code> (1:1 with cdr) ==
 
Extension table linked via <code>cdr_ID</code>.
 
{| class="wikitable"
|-
! Column !! Description
|-
| <code>fbasename</code> || Derived from SIP Call-ID; used to link PCAP files. Special characters replaced with underscores (<code>convertchar</code> in config)
|-
| <code>match_header</code> || Content of custom header for linking call legs (<code>matchheader</code> in config)
|}
 
== <code>cdr_rtp</code> (Per-Stream RTP Stats) ==
 
One row per RTP stream. Multiple streams per call.
 
{| class="wikitable"
|-
! Column !! Description
|-
| <code>cdr_ID</code> || FK to <code>cdr</code>
|-
| <code>ssrc</code> || RTP Synchronization Source identifier
|-
| <code>saddr</code> / <code>daddr</code> || Source/destination IP (as integer)
|-
| <code>sport</code> / <code>dport</code> || Source/destination port
|-
| <code>payload</code> || Codec type. '''Video streams: payload ≥ 10000''' (offset applied)
|-
| <code>received</code> / <code>lost</code> || Packet counts
|}
 
{{Tip|To query video streams only: <code>WHERE payload >= 10000</code>}}
 
== <code>cdr_sdp</code> (SDP-Negotiated Ports) ==
 
Stores media endpoints declared in SDP. Requires <code>save_sdp_ipport = yes</code>.
 
{| class="wikitable"
|-
! Column !! Description
|-
| <code>ip</code> / <code>port</code> || Media IP and port from SDP
|-
| <code>is_caller</code> || 1 = caller side, 0 = callee side
|}
 
{{Note|<code>cdr_sdp</code> stores what endpoints '''agreed to use''' in SDP; <code>cdr_rtp</code> stores what was '''actually sent'''.}}
 
== <code>cdr_proxy</code> (Proxy Chain) ==
 
IP addresses of SIP proxies the call traversed.
 
== <code>cdr_siphistory</code> (SIP Method History) ==
 
Stores SIP requests/responses including REFER, BYE, CANCEL, OPTIONS, etc. Requires <code>save_sip_history = all</code>.
 
== <code>cdr_ua</code> / <code>cdr_sip_response</code> (Lookup Tables) ==
 
Lookup tables for User-Agent strings and SIP response texts.
 
= Common SQL Queries =
 
== Basic CDR Query with IP Conversion ==
 
<syntaxhighlight lang="sql">
SELECT
    ID, caller, called,
    INET_NTOA(sipcallerip) AS caller_ip,
    INET_NTOA(sipcalledip) AS called_ip,
    duration
FROM cdr
WHERE calldate >= '2024-01-01'
  AND sipcallerip = INET_ATON('192.168.1.50')
LIMIT 10;
</syntaxhighlight>
 
== Get PCAP Filename (fbasename) ==
 
<syntaxhighlight lang="sql">
SELECT c.ID, c.caller, c.called, cn.fbasename
FROM cdr c
JOIN cdr_next cn ON c.ID = cn.cdr_ID
WHERE c.ID = 12345;
</syntaxhighlight>
 
== Get User-Agent Strings ==
 
<syntaxhighlight lang="sql">
SELECT c.ID, ua_a.ua AS caller_ua, ua_b.ua AS callee_ua
FROM cdr c
LEFT JOIN cdr_ua ua_a ON c.a_ua_id = ua_a.ID
LEFT JOIN cdr_ua ua_b ON c.b_ua_id = ua_b.ID
WHERE c.calldate >= CURDATE() - INTERVAL 1 DAY
LIMIT 10;
</syntaxhighlight>
 
== Find Calls Through Specific Proxy ==
 
<syntaxhighlight lang="sql">
SELECT c.ID, c.calldate, c.caller, c.called,
      INET_NTOA(cp.dst) AS proxy_ip
FROM cdr c
JOIN cdr_proxy cp ON c.ID = cp.cdr_ID
WHERE cp.dst = INET_ATON('192.168.1.100')
LIMIT 100;
</syntaxhighlight>
 
== Query Video Stream Quality ==
 
<syntaxhighlight lang="sql">
SELECT cdr_ID, payload,
      INET_NTOA(saddr) AS src_ip,
      received, lost,
      (lost / (received + lost)) * 100 AS loss_pct
FROM cdr_rtp
WHERE payload >= 10000  -- Video streams only
  AND cdr_ID = 12345;
</syntaxhighlight>
 
== Find Calls with Specific SIP Method (REFER) ==
 
<syntaxhighlight lang="sql">
SELECT cdr_ID, callid, calldate
FROM cdr_siphistory
WHERE method = 'REFER'
  AND calldate > '2024-01-01'
GROUP BY cdr_ID;
</syntaxhighlight>
 
= Custom Headers =
 
== CDR-Level Custom Headers ==
 
CDR custom headers are stored in <code>cdr_next_X</code> tables. Use <code>cdr_custom_headers</code> to find the location:
 
<syntaxhighlight lang="sql">
-- Step 1: Find table/column for header
SELECT dynamic_table, dynamic_column
FROM cdr_custom_headers
WHERE header_field = 'X-Customer-ID';
-- Result: dynamic_table=2, dynamic_column=1 → cdr_next_2.custom_header_1
 
-- Step 2: Query the data
SELECT c.ID, c.calldate, cn.custom_header_1 AS customer_id
FROM cdr c
JOIN cdr_next_2 cn ON c.ID = cn.cdr_ID
WHERE cn.custom_header_1 = '12345';
</syntaxhighlight>
 
{{Note|The GUI's [[Reports|Reports]] module does not support grouping by custom headers. Use direct SQL queries.}}
=== <code>cdr_custom_headers</code> Table Structure ===
 
The <code>cdr_custom_headers</code> table stores the configuration for custom header extraction:
 
{| class="wikitable"
|-
! Column !! Description
|-
| <code>header_field</code> || SIP header name to capture
|-
| <code>dynamic_table</code> || Target table number (cdr_next_X)
|-
| <code>dynamic_column</code> || Target column number (custom_header_X)
|-
| <code>type</code> || Extraction type (first/last/nth)
|-
| <code>nth_occurrence</code> || '''(New in 35.x)''' Specific occurrence number to extract when type is "nth" (e.g., 2 for second occurrence)
|}
 
{{Note|The <code>nth_occurrence</code> column uses packet timestamps for accurate ordering, ensuring correct results even when packet time-based reordering is disabled in the sniffer.}}
== SIP Message Custom Headers ==
 
Per-message headers are stored in <code>sip_msg_next_X</code> tables. Use <code>sip_msg_custom_headers</code> to find the location.
 
= System Table =
 
The <code>system</code> table stores configuration including <code>manager_key</code> for sensor operations.
 
<syntaxhighlight lang="sql">
SELECT * FROM voipmonitor.`system` WHERE type='manager_key'\G
</syntaxhighlight>
 
{{Warning|1=If sniffer fails with "failed read rsa key", check that <code>mysqlloadconfig</code> is NOT set to <code>no</code> in <code>voipmonitor.conf</code>.}}
 
== Synchronizing manager_key Between Databases ==
 
When multiple GUI instances share sensors (e.g., after database migration), mismatched <code>manager_key</code> values cause PCAP download failures.
 
'''Symptoms:''' "Unable to download PCAP" errors, empty SIP history in one GUI.
 
'''Fix:'''
<syntaxhighlight lang="sql">
-- On active database: get the key
SELECT content FROM voipmonitor.`system` WHERE type='manager_key';
 
-- On old database: update the key
UPDATE voipmonitor.`system`
SET content = '{"key":"<key_from_active>","iv":"<iv_from_active>"}'
WHERE type = 'manager_key';
</syntaxhighlight>
 
Users must log out and log back in after the update.
 
= Troubleshooting =
 
== Missing <code>sport</code>/<code>dport</code> in cdr_rtp ==
 
If RTP streams from overlapping calls are incorrectly merged, add the missing columns:
 
<syntaxhighlight lang="sql">
ALTER TABLE cdr_rtp ADD `sport` smallint unsigned DEFAULT NULL AFTER `saddr`;
ALTER TABLE cdr_rtp ADD `dport` smallint unsigned DEFAULT NULL AFTER `daddr`;
</syntaxhighlight>
 
{{Warning|1=<code>ALTER TABLE</code> can lock large tables for hours. Run during maintenance windows.}}
 
To find other recommended schema changes:
<syntaxhighlight lang="bash">
journalctl -u voipmonitor | grep ALTER
</syntaxhighlight>
 
= See Also =
 
* [[Call_Detail_Record_-_CDR|CDR View (GUI)]]
* [[Database_troubleshooting|Database Troubleshooting]]
* [[Sniffer_configuration#Database_Configuration|Database Configuration]]
* [[WEB_API|Web API Documentation]]
 
 
= AI Summary for RAG =
 
'''Summary:''' VoIPmonitor database schema reference. Main table <code>cdr</code> stores one record per call with timing, participants, and quality metrics. Column naming: <code>a_</code>=caller, <code>b_</code>=callee, <code>_multNN</code>=divide by NN. Key tables: <code>cdr_next</code> (1:1, contains <code>fbasename</code> for PCAP linking), <code>cdr_rtp</code> (per-stream stats, video streams have payload≥10000), <code>cdr_proxy</code> (proxy chain IPs), <code>cdr_sdp</code> (SDP-negotiated ports), <code>cdr_siphistory</code> (SIP method history, requires <code>save_sip_history=all</code>). Custom headers stored in <code>cdr_next_X</code> tables; query <code>cdr_custom_headers</code> for mappings. System table stores <code>manager_key</code>; mismatch causes PCAP download failures.
 
'''Keywords:''' database, schema, cdr, mysql, cdr_next, fbasename, cdr_rtp, video payload, cdr_proxy, cdr_sdp, cdr_siphistory, INET_NTOA, INET_ATON, sipcallerip, custom headers, cdr_custom_headers, manager_key, packet loss, jitter, mos, rtcp, a_rtcp_avgrtd_mult10, latency


== AI Summary for RAG ==
'''Summary:''' This document provides a reference for the VoIPmonitor database schema, focusing on the main `cdr` table and its most important related tables. It explains the column naming conventions, such as the `a_` (caller) and `b_` (callee) prefixes for RTP streams and the `_multNN` suffix indicating a value needs to be divided. The `cdr` table columns are logically grouped into categories: General Call Information (ID, calldate), Call Timing (duration, connect_duration, PDD), Participant Information (caller, called, IPs, User-Agents), and detailed Network Quality Metrics. The quality metrics section details columns for Packet Loss (lost counts, distribution), Jitter (average, max, PDV distribution), Mean Opinion Score (MOS), and RTCP-reported statistics. Finally, it briefly describes the purpose of key related tables like `cdr_next` (for Call-ID), `cdr_rtp`, `cdr_dtmf`, `cdr_proxy`, and the lookup tables `cdr_sip_response` and `cdr_ua`.
'''Keywords:''' database, schema, table, column, cdr, cdr_next, cdr_rtp, database schema, sql, mysql, mariadb, calldate, duration, packet loss, jitter, mos, rtcp, pdd, a_lost, b_lost, a_avgjitter, mos_f1, a_mos_adapt, fbasename, user-agent
'''Key Questions:'''
'''Key Questions:'''
* What do the `a_` and `b_` prefixes mean in the `cdr` table?
* What do a_ and b_ prefixes mean in cdr table columns?
* How are jitter and packet loss stored in the VoIPmonitor database?
* How do I convert sipcallerip to readable IP address?
* What is the difference between `duration` and `connect_duration`?
* How do I get the PCAP filename (fbasename) for a call?
* How can I get the Call-ID for a specific call from the database?
* How do I query video streams in cdr_rtp (payload >= 10000)?
* Which table stores the User-Agent of the phone?
* How do I find calls that went through a specific proxy?
* How do I convert the `sipcallerip` integer back to an IP address in SQL?
* How do I find calls with REFER or other SIP methods?
* What is the purpose of the `cdr_next` table?
* How do I query CDR custom headers?
* How are MOS scores stored in the database?
* Why does "failed read rsa key" error occur on startup?
* How do I sync manager_key between multiple databases?
* Why can't old GUI download PCAP files after database migration?

Latest revision as of 11:29, 19 January 2026


Database schema reference for VoIPmonitor. The cdr table stores one record per call; related tables store RTP statistics, custom headers, and metadata.

Schema Overview

-- List all CDR-related tables
SHOW TABLES LIKE 'cdr%';

Column Naming Conventions

Pattern Meaning Example
a_ Caller's stream (A-leg) a_lost, a_mos_f1_mult10
b_ Callee's stream (B-leg) b_lost, b_mos_f1_mult10
_mult10 Divide by 10 jitter_mult10 = 15 → jitter = 1.5
_mult100 Divide by 100 Percentage values
_mult1000 Divide by 1000 packet_loss_perc_mult1000

The cdr Table

Call Information & Timing

Column Description
ID Primary key
id_sensor Sensor ID (from voipmonitor.conf)
calldate / callend Start/end timestamps
duration Total length in seconds
connect_duration Connected time (after 200 OK)
progress_time Time to first provisional response (PDD)
bye 1 = normal termination with BYE
whohanged Which party sent BYE (caller/callee)
lastSIPresponse_id FK to cdr_sip_response

Participant Information

Column Description
caller / called Numbers from From/To headers
caller_domain / called_domain Domain parts
sipcallerip / sipcalledip IPs stored as integers (use INET_NTOA())
a_ua_id / b_ua_id FK to cdr_ua (User-Agent)

Quality Metrics

Packet Loss

Column Description
a_lost / b_lost Total RTP packets lost
packet_loss_perc_mult1000 Max loss percentage × 1000
a_sl1 - a_sl10 Distribution of consecutive loss events

Jitter & Delay

Column Description
a_avgjitter_mult10 Average jitter × 10
a_maxjitter Maximum jitter observed
a_d50 - a_d300 PDV distribution buckets (0-50ms, 50-70ms, etc.)

MOS (Mean Opinion Score)

Column Description
a_mos_f1_mult10 MOS with 50ms fixed buffer × 10
a_mos_f2_mult10 MOS with 200ms fixed buffer × 10
a_mos_adapt_mult10 MOS with adaptive buffer × 10
mos_min_mult10 Lowest MOS between A/B streams × 10

RTCP-Reported Metrics

These come from endpoint RTCP reports, not calculated by the sniffer. See Sniffer vs RTCP Packet Loss.

Column Description
a_rtcp_loss Packets lost (reported by endpoint)
a_rtcp_avgfr_mult10 Average fraction loss × 10
a_rtcp_avgjitter_mult10 Average jitter × 10
a_rtcp_avgrtd_mult10 Roundtrip delay (latency) in ms × 10

Related Tables

cdr_next (1:1 with cdr)

Extension table linked via cdr_ID.

Column Description
fbasename Derived from SIP Call-ID; used to link PCAP files. Special characters replaced with underscores (convertchar in config)
match_header Content of custom header for linking call legs (matchheader in config)

cdr_rtp (Per-Stream RTP Stats)

One row per RTP stream. Multiple streams per call.

Column Description
cdr_ID FK to cdr
ssrc RTP Synchronization Source identifier
saddr / daddr Source/destination IP (as integer)
sport / dport Source/destination port
payload Codec type. Video streams: payload ≥ 10000 (offset applied)
received / lost Packet counts

💡 Tip:

cdr_sdp (SDP-Negotiated Ports)

Stores media endpoints declared in SDP. Requires save_sdp_ipport = yes.

Column Description
ip / port Media IP and port from SDP
is_caller 1 = caller side, 0 = callee side

ℹ️ Note: cdr_sdp stores what endpoints agreed to use in SDP; cdr_rtp stores what was actually sent.

cdr_proxy (Proxy Chain)

IP addresses of SIP proxies the call traversed.

cdr_siphistory (SIP Method History)

Stores SIP requests/responses including REFER, BYE, CANCEL, OPTIONS, etc. Requires save_sip_history = all.

cdr_ua / cdr_sip_response (Lookup Tables)

Lookup tables for User-Agent strings and SIP response texts.

Common SQL Queries

Basic CDR Query with IP Conversion

SELECT
    ID, caller, called,
    INET_NTOA(sipcallerip) AS caller_ip,
    INET_NTOA(sipcalledip) AS called_ip,
    duration
FROM cdr
WHERE calldate >= '2024-01-01'
  AND sipcallerip = INET_ATON('192.168.1.50')
LIMIT 10;

Get PCAP Filename (fbasename)

SELECT c.ID, c.caller, c.called, cn.fbasename
FROM cdr c
JOIN cdr_next cn ON c.ID = cn.cdr_ID
WHERE c.ID = 12345;

Get User-Agent Strings

SELECT c.ID, ua_a.ua AS caller_ua, ua_b.ua AS callee_ua
FROM cdr c
LEFT JOIN cdr_ua ua_a ON c.a_ua_id = ua_a.ID
LEFT JOIN cdr_ua ua_b ON c.b_ua_id = ua_b.ID
WHERE c.calldate >= CURDATE() - INTERVAL 1 DAY
LIMIT 10;

Find Calls Through Specific Proxy

SELECT c.ID, c.calldate, c.caller, c.called,
       INET_NTOA(cp.dst) AS proxy_ip
FROM cdr c
JOIN cdr_proxy cp ON c.ID = cp.cdr_ID
WHERE cp.dst = INET_ATON('192.168.1.100')
LIMIT 100;

Query Video Stream Quality

SELECT cdr_ID, payload,
       INET_NTOA(saddr) AS src_ip,
       received, lost,
       (lost / (received + lost)) * 100 AS loss_pct
FROM cdr_rtp
WHERE payload >= 10000  -- Video streams only
  AND cdr_ID = 12345;

Find Calls with Specific SIP Method (REFER)

SELECT cdr_ID, callid, calldate
FROM cdr_siphistory
WHERE method = 'REFER'
  AND calldate > '2024-01-01'
GROUP BY cdr_ID;

Custom Headers

CDR-Level Custom Headers

CDR custom headers are stored in cdr_next_X tables. Use cdr_custom_headers to find the location:

-- Step 1: Find table/column for header
SELECT dynamic_table, dynamic_column
FROM cdr_custom_headers
WHERE header_field = 'X-Customer-ID';
-- Result: dynamic_table=2, dynamic_column=1 → cdr_next_2.custom_header_1

-- Step 2: Query the data
SELECT c.ID, c.calldate, cn.custom_header_1 AS customer_id
FROM cdr c
JOIN cdr_next_2 cn ON c.ID = cn.cdr_ID
WHERE cn.custom_header_1 = '12345';

ℹ️ Note: The GUI's Reports module does not support grouping by custom headers. Use direct SQL queries.

cdr_custom_headers Table Structure

The cdr_custom_headers table stores the configuration for custom header extraction:

Column Description
header_field SIP header name to capture
dynamic_table Target table number (cdr_next_X)
dynamic_column Target column number (custom_header_X)
type Extraction type (first/last/nth)
nth_occurrence (New in 35.x) Specific occurrence number to extract when type is "nth" (e.g., 2 for second occurrence)

ℹ️ Note: The nth_occurrence column uses packet timestamps for accurate ordering, ensuring correct results even when packet time-based reordering is disabled in the sniffer.

SIP Message Custom Headers

Per-message headers are stored in sip_msg_next_X tables. Use sip_msg_custom_headers to find the location.

System Table

The system table stores configuration including manager_key for sensor operations.

SELECT * FROM voipmonitor.`system` WHERE type='manager_key'\G

⚠️ Warning: If sniffer fails with "failed read rsa key", check that mysqlloadconfig is NOT set to no in voipmonitor.conf.

Synchronizing manager_key Between Databases

When multiple GUI instances share sensors (e.g., after database migration), mismatched manager_key values cause PCAP download failures.

Symptoms: "Unable to download PCAP" errors, empty SIP history in one GUI.

Fix:

-- On active database: get the key
SELECT content FROM voipmonitor.`system` WHERE type='manager_key';

-- On old database: update the key
UPDATE voipmonitor.`system`
SET content = '{"key":"<key_from_active>","iv":"<iv_from_active>"}'
WHERE type = 'manager_key';

Users must log out and log back in after the update.

Troubleshooting

Missing sport/dport in cdr_rtp

If RTP streams from overlapping calls are incorrectly merged, add the missing columns:

ALTER TABLE cdr_rtp ADD `sport` smallint unsigned DEFAULT NULL AFTER `saddr`;
ALTER TABLE cdr_rtp ADD `dport` smallint unsigned DEFAULT NULL AFTER `daddr`;

⚠️ Warning: ALTER TABLE can lock large tables for hours. Run during maintenance windows.

To find other recommended schema changes:

journalctl -u voipmonitor | grep ALTER

See Also


AI Summary for RAG

Summary: VoIPmonitor database schema reference. Main table cdr stores one record per call with timing, participants, and quality metrics. Column naming: a_=caller, b_=callee, _multNN=divide by NN. Key tables: cdr_next (1:1, contains fbasename for PCAP linking), cdr_rtp (per-stream stats, video streams have payload≥10000), cdr_proxy (proxy chain IPs), cdr_sdp (SDP-negotiated ports), cdr_siphistory (SIP method history, requires save_sip_history=all). Custom headers stored in cdr_next_X tables; query cdr_custom_headers for mappings. System table stores manager_key; mismatch causes PCAP download failures.

Keywords: database, schema, cdr, mysql, cdr_next, fbasename, cdr_rtp, video payload, cdr_proxy, cdr_sdp, cdr_siphistory, INET_NTOA, INET_ATON, sipcallerip, custom headers, cdr_custom_headers, manager_key, packet loss, jitter, mos, rtcp, a_rtcp_avgrtd_mult10, latency

Key Questions:

  • What do a_ and b_ prefixes mean in cdr table columns?
  • How do I convert sipcallerip to readable IP address?
  • How do I get the PCAP filename (fbasename) for a call?
  • How do I query video streams in cdr_rtp (payload >= 10000)?
  • How do I find calls that went through a specific proxy?
  • How do I find calls with REFER or other SIP methods?
  • How do I query CDR custom headers?
  • Why does "failed read rsa key" error occur on startup?
  • How do I sync manager_key between multiple databases?
  • Why can't old GUI download PCAP files after database migration?