Database structure: Difference between revisions

From VoIPmonitor.org
(Add section explaining difference between sniffer and RTCP packet loss for troubleshooting)
(VS-1789: Document nth_occurrence column in cdr_custom_headers table)
 
(23 intermediate revisions by the same user not shown)
Line 1: Line 1:
{{DISPLAYTITLE:CDR Database Table Schema}}
{{DISPLAYTITLE:CDR Database Table Schema}}


'''This document provides an overview of the most important tables and columns in the VoIPmonitor database schema. It is intended as a reference for administrators, developers, or anyone directly querying the database.'''
'''Database schema reference for VoIPmonitor. The <code>cdr</code> table stores one record per call; related tables store RTP statistics, custom headers, and metadata.'''


The core of the VoIPmonitor database is the `cdr` (Call Detail Record) table, which stores one primary record for each monitored call. Many other tables are linked to `cdr` to store additional, more detailed information.
= Schema Overview =


== Understanding Column Naming Conventions ==
<syntaxhighlight lang="sql">
Before diving into the columns, it's important to understand these common naming patterns:
-- List all CDR-related tables
*'''`a_` prefix:''' Refers to the "A-leg" or the '''caller's''' media stream (RTP sent from the caller).
SHOW TABLES LIKE 'cdr%';
*'''`b_` prefix:''' Refers to the "B-leg" or the '''callee's''' media stream (RTP sent from the callee).
</syntaxhighlight>
*'''`_mult10` / `_mult100` / `_mult1000`:''' Indicates that the stored integer value must be '''divided''' by this number to get the real decimal value. For example, a `jitter_mult10` value of `15` represents a real jitter of `1.5`.


== The `cdr` Table ==
<kroki lang="plantuml">
This is the main table containing the primary information for every call.
@startuml
skinparam shadowing false
skinparam defaultFontName Arial
skinparam linetype ortho


=== General Call Information ===
entity "cdr" as cdr {
;`ID`
  * ID : bigint <<PK>>
:The unique primary key for the record.
  --
;`id_sensor`
  id_sensor, calldate, callend
:The ID of the sensor that processed this call, corresponding to `id_sensor` in `voipmonitor.conf`.
  caller, called, duration
;`calldate`
  sipcallerip, sipcalledip
:The start time of the call, typically measured from the first `INVITE` packet.
  a_lost, b_lost, a_mos_f1_mult10
;`callend`
  lastSIPresponse_id <<FK>>
:The end time of the call, measured from the last packet associated with the call.
  a_ua_id, b_ua_id <<FK>>
;`bye`
}
:An indicator of how the call was terminated. `1` typically means a normal termination with a `BYE` request.
;`whohanged`
:Indicates which party sent the `BYE` message first (caller or callee).
;`lastSIPresponse_id`
:A reference to the `cdr_sip_response` table, linking to the last final SIP response for the call.


=== Call Timing & Duration ===
entity "cdr_next" as cdr_next {
;`duration`
  * cdr_ID <<PK,FK>>
:The total length of the call in seconds (`callend` - `calldate`).
  --
;`connect_duration`
  fbasename, match_header
:The length of the connected portion of the call in seconds, measured from the first `200 OK` to the `callend`.
}
;`progress_time`
:The time in seconds from the initial `INVITE` to the first provisional response (e.g., `180 Ringing`). This is often referred to as [[Glossary#Post_Dial_Delay_.28PDD.29|Post-Dial Delay (PDD)]].
;`first_rtp_time`
:The time in seconds from the start of the call to the detection of the very first RTP packet.


=== Participant Information ===
entity "cdr_rtp" as cdr_rtp {
;`caller` / `called`
  cdr_ID <<FK>>
:The user part of the number from the `From` and `To` SIP headers, respectively.
  --
;`caller_domain` / `called_domain`
  saddr, daddr, ssrc
:The domain part from the `From` and `To` SIP headers.
  received, lost, payload
;`caller_reverse` / `called_reverse`
}
:Reversed versions of the `caller` and `called` fields, used internally for efficient `LIKE '%search'` queries in the database.
;`callername`
:The display name part from the `From` SIP header.
;`sipcallerip` / `sipcalledip`
:The source and destination IP addresses from the SIP signaling. These are stored as integers; use the `INET_NTOA()` or `INET6_NTOA()` SQL function to convert them to a string representation.
;`a_ua_id` / `b_ua_id`
:A reference to the `cdr_ua` table, linking to the User-Agent string of the caller's and callee's device.


=== Network Quality Metrics ===
entity "cdr_proxy" as cdr_proxy {
==== Packet Loss ====
  cdr_ID <<FK>>
;`a_lost` / `b_lost`
  --
:The total number of RTP packets lost, detected by sequence number gaps.
  dst (proxy IP)
;`lost`
}
:The higher value between `a_lost` and `b_lost`.
;`packet_loss_perc_mult1000`
:The maximum packet loss percentage from either stream, multiplied by 1000.
;`a_sl1` - `a_sl10` / `b_sl1` - `b_sl10`
:A distribution of consecutive packet loss events. `a_sl1` counts the number of times a single packet was lost, `a_sl5` counts the number of times 5 packets were lost in a row, etc.


==== Jitter & Delay (PDV) ====
entity "cdr_sdp" as cdr_sdp {
;`a_avgjitter_mult10` / `b_avgjitter_mult10`
  cdr_id <<FK>>
:The average jitter calculated according to RFC3550, multiplied by 10.
  --
;`a_maxjitter` / `b_maxjitter`
  ip, port, is_caller
:The maximum jitter value observed during the call.
}
;`jitter_mult10`
:The higher `maxjitter` value between the two streams, multiplied by 10.
;`a_d50` - `a_d300` / `b_d50` - `b_d300`
:A distribution of Packet Delay Variation (PDV). `a_d50` counts packets with a delay between 0-50ms, `a_d70` counts packets with a delay between 50-70ms, etc.


==== MOS (Mean Opinion Score) ====
entity "cdr_siphistory" as cdr_siphistory {
;`a_mos_f1_mult10` / `b_mos_f1_mult10`
  cdr_ID <<FK>>
:The calculated [[Glossary#VoIPmonitor.27s_Parametric_MOS|parametric MOS score]] based on a simulated 50ms fixed jitter buffer, multiplied by 10.
  --
;`a_mos_f2_mult10` / `b_mos_f2_mult10`
  method, calldate, cseq
:MOS score based on a simulated 200ms fixed jitter buffer, multiplied by 10.
}
;`a_mos_adapt_mult10` / `b_mos_adapt_mult10`
:MOS score based on a simulated adaptive jitter buffer, multiplied by 10.
;`mos_min_mult10`
:The lowest MOS score recorded between the A and B streams, multiplied by 10.


==== RTCP Reported Metrics ====
entity "cdr_ua" as cdr_ua {
These metrics are sourced from RTCP reports sent by the endpoints themselves, as opposed to being calculated by the sniffer.
  * ID <<PK>>
;`a_rtcp_loss` / `b_rtcp_loss`
  --
:Total packets lost as reported by RTCP.
  ua (User-Agent string)
;`a_rtcp_avgfr_mult10` / `b_rtcp_avgfr_mult10`
}
:Average fraction loss reported by RTCP, multiplied by 10.
;`a_rtcp_avgjitter_mult10` / `b_rtcp_avgjitter_mult10`
:Average jitter reported by RTCP, multiplied by 10.


entity "cdr_sip_response" as cdr_sip_response {
  * ID <<PK>>
  --
  lastSIPresponse
}


==== Understanding the Difference Between Sniffer and RTCP Packet Loss ====
cdr ||--|| cdr_next : "1:1"
cdr ||--o{ cdr_rtp : "1:N"
cdr ||--o{ cdr_proxy : "1:N"
cdr ||--o{ cdr_sdp : "1:N"
cdr ||--o{ cdr_siphistory : "1:N"
cdr }o--|| cdr_ua : "N:1"
cdr }o--|| cdr_sip_response : "N:1"
@enduml
</kroki>


VoIPmonitor provides two different packet loss metrics that measure the same concept (lost VoIP packets) from two fundamentally different locations in the network. Understanding the distinction is critical for troubleshooting call quality issues.
== Column Naming Conventions ==


The Two Perspectives:
{| class="wikitable"
;'''Sniffer Loss (`a_lost` / `b_lost`):''' The VoIPmonitor sniffer watches the traffic at its monitoring point (e.g., on the PBX switch) and counts missing packets based on RTP sequence number gaps. This represents what the network path ''up to the monitoring point'' experienced.
|-
;'''RTCP Loss (`a_rtcp_loss` / `b_rtcp_loss`):''' The receiving endpoint (phone, gateway, or SBC) sends RTP Control Protocol (RTCP) Receiver Reports back to the transmitter, listing the cumulative count of packets it failed to receive or decode. This represents what the ''actual end-user device'' experienced.
! Pattern !! Meaning !! Example
|-
| <code>a_</code> || Caller's stream (A-leg) || <code>a_lost</code>, <code>a_mos_f1_mult10</code>
|-
| <code>b_</code> || Callee's stream (B-leg) || <code>b_lost</code>, <code>b_mos_f1_mult10</code>
|-
| <code>_mult10</code> || Divide by 10 || <code>jitter_mult10 = 15</code> → jitter = 1.5
|-
| <code>_mult100</code> || Divide by 100 || Percentage values
|-
| <code>_mult1000</code> || Divide by 1000 || <code>packet_loss_perc_mult1000</code>
|}


Common Scenarios and Interpretations:
= The <code>cdr</code> Table =


;'''1. RTCP Loss is Higher Than Sniffer Loss'''
== Call Information & Timing ==
This indicates a network issue exists in the segment '''after''' the VoIPmonitor sniffer. Typical causes:
* Poor Wi-Fi connectivity at the end-user's home/office
* Network issues in the last-mile connection
* Problems between the monitoring point (e.g., PBX) and the actual receiver device
In this case, the sniffer sees all packets on the wire (low loss), but some packets are lost downstream before reaching the user's phone.


;'''2. Sniffer Loss is Higher Than RTCP Loss'''
{| class="wikitable"
This is unusual and typically indicates:
|-
* The monitoring point itself is dropping packets (e.g., due to high CPU load or interface saturation)
! Column !! Description
* The RTCP reports from the endpoint are not being captured or transmitted properly
|-
| <code>ID</code> || Primary key
|-
| <code>id_sensor</code> || Sensor ID (from <code>voipmonitor.conf</code>)
|-
| <code>calldate</code> / <code>callend</code> || Start/end timestamps
|-
| <code>duration</code> || Total length in seconds
|-
| <code>connect_duration</code> || Connected time (after 200 OK)
|-
| <code>progress_time</code> || Time to first provisional response ([[Glossary#Post_Dial_Delay_.28PDD.29|PDD]])
|-
| <code>bye</code> || <code>1</code> = normal termination with BYE
|-
| <code>whohanged</code> || Which party sent BYE (caller/callee)
|-
| <code>lastSIPresponse_id</code> || FK to <code>cdr_sip_response</code>
|}


;'''3. High RTCP Loss but MOS is Still Good'''
== Participant Information ==
This can occur when:
* Packet Loss Concealment (PLC) is effectively hiding scattered losses
* The endpoint's jitter buffer is successfully absorbing jitter-related delays
* The MOS calculation (based on sniffer loss) doesn't fully represent the user's actual experience


Troubleshooting Guidance:
{| class="wikitable"
* '''Trust RTCP loss for user experience''': For calls where users report quality issues, `a_rtcp_loss`/`b_rtcp_loss` is more authoritative because it reflects what the device actually received.
|-
* '''Use sniffer loss for network health''': Monitoring `a_lost`/`b_lost` is useful for capacity planning and detecting issues in the network segment you control.
! Column !! Description
* '''Downstream issues''': When RTCP loss is higher, investigate the network segment between the monitoring point and the end-user device (check Wi-Fi, local network, ISP issues).
|-
| <code>caller</code> / <code>called</code> || Numbers from From/To headers
|-
| <code>caller_domain</code> / <code>called_domain</code> || Domain parts
|-
| <code>sipcallerip</code> / <code>sipcalledip</code> || IPs stored as integers (use <code>INET_NTOA()</code>)
|-
| <code>a_ua_id</code> / <code>b_ua_id</code> || FK to <code>cdr_ua</code> (User-Agent)
|}


== Quality Metrics ==


=== Packet Loss ===
{| class="wikitable"
|-
! Column !! Description
|-
| <code>a_lost</code> / <code>b_lost</code> || Total RTP packets lost
|-
| <code>packet_loss_perc_mult1000</code> || Max loss percentage × 1000
|-
| <code>a_sl1</code> - <code>a_sl10</code> || Distribution of consecutive loss events
|}


=== Other Information ===
=== Jitter & Delay ===
;`dscp`
{| class="wikitable"
:Stores the DSCP values from SIP and RTP packets for Quality of Service (QoS) analysis.
|-
;`payload`
! Column !! Description
:The codec payload type number used during the call.
|-
| <code>a_avgjitter_mult10</code> || Average jitter × 10
|-
| <code>a_maxjitter</code> || Maximum jitter observed
|-
| <code>a_d50</code> - <code>a_d300</code> || PDV distribution buckets (0-50ms, 50-70ms, etc.)
|}


== Important Related Tables ==
=== MOS (Mean Opinion Score) ===
{| class="wikitable"
|-
! Column !! Description
|-
| <code>a_mos_f1_mult10</code> || [[Comprehensive_Guide_to_VoIP_Voice_Quality#MOS_Calculation_in_VoIPmonitor|MOS]] with 50ms fixed buffer × 10
|-
| <code>a_mos_f2_mult10</code> || MOS with 200ms fixed buffer × 10
|-
| <code>a_mos_adapt_mult10</code> || MOS with adaptive buffer × 10
|-
| <code>mos_min_mult10</code> || Lowest MOS between A/B streams × 10
|}


; `cdr_next`
=== RTCP-Reported Metrics ===
:A 1-to-1 extension of the `cdr` table that stores additional data to keep the main `cdr` table smaller and faster.
These come from endpoint RTCP reports, not calculated by the sniffer. See [[Call_Detail_Record_-_CDR#Understanding_Sniffer_vs_RTCP_Packet_Loss|Sniffer vs RTCP Packet Loss]].
:*'''fbasename:''' A string derived from the SIP `Call-ID`, used to link PCAP files to the CDR.
:*'''match_header:''' Stores the content of a custom header (defined by `matchheader` in `voipmonitor.conf`) used for linking call legs.


; `cdr_rtp`
{| class="wikitable"
:Stores detailed statistics for each individual RTP stream within a call, used by the GUI to render stream information in the call history view.
|-
! Column !! Description
|-
| <code>a_rtcp_loss</code> || Packets lost (reported by endpoint)
|-
| <code>a_rtcp_avgfr_mult10</code> || Average fraction loss × 10
|-
| <code>a_rtcp_avgjitter_mult10</code> || Average jitter × 10
|-
| <code>a_rtcp_avgrtd_mult10</code> || '''Roundtrip delay (latency)''' in ms × 10
|}


; `cdr_dtmf`
= Related Tables =
:Stores detected DTMF key presses (if `dtmf2db` is enabled).


; `cdr_proxy`
== <code>cdr_next</code> (1:1 with cdr) ==
:Stores the IP addresses of all SIP proxies that the call traversed.


; `cdr_sip_response`
Extension table linked via <code>cdr_ID</code>.
:A lookup table storing the text of unique SIP responses (e.g., "404 Not Found"). The `cdr` table references this table by ID to save space.


; `cdr_ua`
{| class="wikitable"
:A lookup table storing the `User-Agent` strings of SIP devices.
|-
! Column !! Description
|-
| <code>fbasename</code> || Derived from SIP Call-ID; used to link PCAP files. Special characters replaced with underscores (<code>convertchar</code> in config)
|-
| <code>match_header</code> || Content of custom header for linking call legs (<code>matchheader</code> in config)
|}
 
== <code>cdr_rtp</code> (Per-Stream RTP Stats) ==
 
One row per RTP stream. Multiple streams per call.
 
{| class="wikitable"
|-
! Column !! Description
|-
| <code>cdr_ID</code> || FK to <code>cdr</code>
|-
| <code>ssrc</code> || RTP Synchronization Source identifier
|-
| <code>saddr</code> / <code>daddr</code> || Source/destination IP (as integer)
|-
| <code>sport</code> / <code>dport</code> || Source/destination port
|-
| <code>payload</code> || Codec type. '''Video streams: payload ≥ 10000''' (offset applied)
|-
| <code>received</code> / <code>lost</code> || Packet counts
|}
 
{{Tip|To query video streams only: <code>WHERE payload >= 10000</code>}}
 
== <code>cdr_sdp</code> (SDP-Negotiated Ports) ==
 
Stores media endpoints declared in SDP. Requires <code>save_sdp_ipport = yes</code>.
 
{| class="wikitable"
|-
! Column !! Description
|-
| <code>ip</code> / <code>port</code> || Media IP and port from SDP
|-
| <code>is_caller</code> || 1 = caller side, 0 = callee side
|}
 
{{Note|<code>cdr_sdp</code> stores what endpoints '''agreed to use''' in SDP; <code>cdr_rtp</code> stores what was '''actually sent'''.}}
 
== <code>cdr_proxy</code> (Proxy Chain) ==
 
IP addresses of SIP proxies the call traversed.
 
== <code>cdr_siphistory</code> (SIP Method History) ==
 
Stores SIP requests/responses including REFER, BYE, CANCEL, OPTIONS, etc. Requires <code>save_sip_history = all</code>.
 
== <code>cdr_ua</code> / <code>cdr_sip_response</code> (Lookup Tables) ==
 
Lookup tables for User-Agent strings and SIP response texts.
 
= Common SQL Queries =
 
== Basic CDR Query with IP Conversion ==
 
<syntaxhighlight lang="sql">
SELECT
    ID, caller, called,
    INET_NTOA(sipcallerip) AS caller_ip,
    INET_NTOA(sipcalledip) AS called_ip,
    duration
FROM cdr
WHERE calldate >= '2024-01-01'
  AND sipcallerip = INET_ATON('192.168.1.50')
LIMIT 10;
</syntaxhighlight>
 
== Get PCAP Filename (fbasename) ==
 
<syntaxhighlight lang="sql">
SELECT c.ID, c.caller, c.called, cn.fbasename
FROM cdr c
JOIN cdr_next cn ON c.ID = cn.cdr_ID
WHERE c.ID = 12345;
</syntaxhighlight>
 
== Get User-Agent Strings ==
 
<syntaxhighlight lang="sql">
SELECT c.ID, ua_a.ua AS caller_ua, ua_b.ua AS callee_ua
FROM cdr c
LEFT JOIN cdr_ua ua_a ON c.a_ua_id = ua_a.ID
LEFT JOIN cdr_ua ua_b ON c.b_ua_id = ua_b.ID
WHERE c.calldate >= CURDATE() - INTERVAL 1 DAY
LIMIT 10;
</syntaxhighlight>
 
== Find Calls Through Specific Proxy ==
 
<syntaxhighlight lang="sql">
SELECT c.ID, c.calldate, c.caller, c.called,
      INET_NTOA(cp.dst) AS proxy_ip
FROM cdr c
JOIN cdr_proxy cp ON c.ID = cp.cdr_ID
WHERE cp.dst = INET_ATON('192.168.1.100')
LIMIT 100;
</syntaxhighlight>
 
== Query Video Stream Quality ==
 
<syntaxhighlight lang="sql">
SELECT cdr_ID, payload,
      INET_NTOA(saddr) AS src_ip,
      received, lost,
      (lost / (received + lost)) * 100 AS loss_pct
FROM cdr_rtp
WHERE payload >= 10000  -- Video streams only
  AND cdr_ID = 12345;
</syntaxhighlight>
 
== Find Calls with Specific SIP Method (REFER) ==
 
<syntaxhighlight lang="sql">
SELECT cdr_ID, callid, calldate
FROM cdr_siphistory
WHERE method = 'REFER'
  AND calldate > '2024-01-01'
GROUP BY cdr_ID;
</syntaxhighlight>
 
= Custom Headers =
 
== CDR-Level Custom Headers ==
 
CDR custom headers are stored in <code>cdr_next_X</code> tables. Use <code>cdr_custom_headers</code> to find the location:
 
<syntaxhighlight lang="sql">
-- Step 1: Find table/column for header
SELECT dynamic_table, dynamic_column
FROM cdr_custom_headers
WHERE header_field = 'X-Customer-ID';
-- Result: dynamic_table=2, dynamic_column=1 → cdr_next_2.custom_header_1
 
-- Step 2: Query the data
SELECT c.ID, c.calldate, cn.custom_header_1 AS customer_id
FROM cdr c
JOIN cdr_next_2 cn ON c.ID = cn.cdr_ID
WHERE cn.custom_header_1 = '12345';
</syntaxhighlight>
 
{{Note|The GUI's [[Reports|Reports]] module does not support grouping by custom headers. Use direct SQL queries.}}
=== <code>cdr_custom_headers</code> Table Structure ===
 
The <code>cdr_custom_headers</code> table stores the configuration for custom header extraction:
 
{| class="wikitable"
|-
! Column !! Description
|-
| <code>header_field</code> || SIP header name to capture
|-
| <code>dynamic_table</code> || Target table number (cdr_next_X)
|-
| <code>dynamic_column</code> || Target column number (custom_header_X)
|-
| <code>type</code> || Extraction type (first/last/nth)
|-
| <code>nth_occurrence</code> || '''(New in 35.x)''' Specific occurrence number to extract when type is "nth" (e.g., 2 for second occurrence)
|}
 
{{Note|The <code>nth_occurrence</code> column uses packet timestamps for accurate ordering, ensuring correct results even when packet time-based reordering is disabled in the sniffer.}}
== SIP Message Custom Headers ==
 
Per-message headers are stored in <code>sip_msg_next_X</code> tables. Use <code>sip_msg_custom_headers</code> to find the location.
 
= System Table =
 
The <code>system</code> table stores configuration including <code>manager_key</code> for sensor operations.
 
<syntaxhighlight lang="sql">
SELECT * FROM voipmonitor.`system` WHERE type='manager_key'\G
</syntaxhighlight>
 
{{Warning|1=If sniffer fails with "failed read rsa key", check that <code>mysqlloadconfig</code> is NOT set to <code>no</code> in <code>voipmonitor.conf</code>.}}
 
== Synchronizing manager_key Between Databases ==
 
When multiple GUI instances share sensors (e.g., after database migration), mismatched <code>manager_key</code> values cause PCAP download failures.
 
'''Symptoms:''' "Unable to download PCAP" errors, empty SIP history in one GUI.
 
'''Fix:'''
<syntaxhighlight lang="sql">
-- On active database: get the key
SELECT content FROM voipmonitor.`system` WHERE type='manager_key';
 
-- On old database: update the key
UPDATE voipmonitor.`system`
SET content = '{"key":"<key_from_active>","iv":"<iv_from_active>"}'
WHERE type = 'manager_key';
</syntaxhighlight>
 
Users must log out and log back in after the update.
 
= Troubleshooting =
 
== Missing <code>sport</code>/<code>dport</code> in cdr_rtp ==
 
If RTP streams from overlapping calls are incorrectly merged, add the missing columns:
 
<syntaxhighlight lang="sql">
ALTER TABLE cdr_rtp ADD `sport` smallint unsigned DEFAULT NULL AFTER `saddr`;
ALTER TABLE cdr_rtp ADD `dport` smallint unsigned DEFAULT NULL AFTER `daddr`;
</syntaxhighlight>
 
{{Warning|1=<code>ALTER TABLE</code> can lock large tables for hours. Run during maintenance windows.}}
 
To find other recommended schema changes:
<syntaxhighlight lang="bash">
journalctl -u voipmonitor | grep ALTER
</syntaxhighlight>
 
= See Also =
 
* [[Call_Detail_Record_-_CDR|CDR View (GUI)]]
* [[Database_troubleshooting|Database Troubleshooting]]
* [[Sniffer_configuration#Database_Configuration|Database Configuration]]
* [[WEB_API|Web API Documentation]]
 
 
= AI Summary for RAG =
 
'''Summary:''' VoIPmonitor database schema reference. Main table <code>cdr</code> stores one record per call with timing, participants, and quality metrics. Column naming: <code>a_</code>=caller, <code>b_</code>=callee, <code>_multNN</code>=divide by NN. Key tables: <code>cdr_next</code> (1:1, contains <code>fbasename</code> for PCAP linking), <code>cdr_rtp</code> (per-stream stats, video streams have payload≥10000), <code>cdr_proxy</code> (proxy chain IPs), <code>cdr_sdp</code> (SDP-negotiated ports), <code>cdr_siphistory</code> (SIP method history, requires <code>save_sip_history=all</code>). Custom headers stored in <code>cdr_next_X</code> tables; query <code>cdr_custom_headers</code> for mappings. System table stores <code>manager_key</code>; mismatch causes PCAP download failures.
 
'''Keywords:''' database, schema, cdr, mysql, cdr_next, fbasename, cdr_rtp, video payload, cdr_proxy, cdr_sdp, cdr_siphistory, INET_NTOA, INET_ATON, sipcallerip, custom headers, cdr_custom_headers, manager_key, packet loss, jitter, mos, rtcp, a_rtcp_avgrtd_mult10, latency


== AI Summary for RAG ==
'''Summary:''' This document provides a reference for the VoIPmonitor database schema, focusing on the main `cdr` table and its most important related tables. It explains the column naming conventions, such as the `a_` (caller) and `b_` (callee) prefixes for RTP streams and the `_multNN` suffix indicating a value needs to be divided. The `cdr` table columns are logically grouped into categories: General Call Information (ID, calldate), Call Timing (duration, connect_duration, PDD), Participant Information (caller, called, IPs, User-Agents), and detailed Network Quality Metrics. The quality metrics section details columns for Packet Loss (lost counts, distribution), Jitter (average, max, PDV distribution), Mean Opinion Score (MOS), and RTCP-reported statistics. A dedicated section explains the conceptual difference between sniffer-calculated packet loss (based on RTP sequence gaps at the monitoring point) and RTCP-reported loss (from the endpoint device), including troubleshooting scenarios for when values differ. Finally, it briefly describes the purpose of key related tables like `cdr_next` (for Call-ID), `cdr_rtp`, `cdr_dtmf`, `cdr_proxy`, and the lookup tables `cdr_sip_response` and `cdr_ua`.
'''Keywords:''' database, schema, table, column, cdr, cdr_next, cdr_rtp, database schema, sql, mysql, mariadb, calldate, duration, packet loss, jitter, mos, rtcp, pdd, a_lost, b_lost, a_avgjitter, mos_f1, a_mos_adapt, fbasename, user-agent, rtcp_loss, troubleshooting, downstream issues, endpoint reported loss
'''Key Questions:'''
'''Key Questions:'''
* What do the `a_` and `b_` prefixes mean in the `cdr` table?
* What do a_ and b_ prefixes mean in cdr table columns?
* How are jitter and packet loss stored in the VoIPmonitor database?
* How do I convert sipcallerip to readable IP address?
* What is the difference between `duration` and `connect_duration`?
* How do I get the PCAP filename (fbasename) for a call?
* How can I get the Call-ID for a specific call from the database?
* How do I query video streams in cdr_rtp (payload >= 10000)?
* Which table stores the User-Agent of the phone?
* How do I find calls that went through a specific proxy?
* How do I convert the `sipcallerip` integer back to an IP address in SQL?
* How do I find calls with REFER or other SIP methods?
* What is the purpose of the `cdr_next` table?
* How do I query CDR custom headers?
* How are MOS scores stored in the database?
* Why does "failed read rsa key" error occur on startup?
* What is the difference between sniffer loss (`a_lost`) and RTCP loss (`a_rtcp_loss`)?
* How do I sync manager_key between multiple databases?
* Why might RTCP loss be higher than sniffer loss?
* Why can't old GUI download PCAP files after database migration?
* What does it mean when RTCP loss is high but MOS is still good?

Latest revision as of 11:29, 19 January 2026


Database schema reference for VoIPmonitor. The cdr table stores one record per call; related tables store RTP statistics, custom headers, and metadata.

Schema Overview

-- List all CDR-related tables
SHOW TABLES LIKE 'cdr%';

Column Naming Conventions

Pattern Meaning Example
a_ Caller's stream (A-leg) a_lost, a_mos_f1_mult10
b_ Callee's stream (B-leg) b_lost, b_mos_f1_mult10
_mult10 Divide by 10 jitter_mult10 = 15 → jitter = 1.5
_mult100 Divide by 100 Percentage values
_mult1000 Divide by 1000 packet_loss_perc_mult1000

The cdr Table

Call Information & Timing

Column Description
ID Primary key
id_sensor Sensor ID (from voipmonitor.conf)
calldate / callend Start/end timestamps
duration Total length in seconds
connect_duration Connected time (after 200 OK)
progress_time Time to first provisional response (PDD)
bye 1 = normal termination with BYE
whohanged Which party sent BYE (caller/callee)
lastSIPresponse_id FK to cdr_sip_response

Participant Information

Column Description
caller / called Numbers from From/To headers
caller_domain / called_domain Domain parts
sipcallerip / sipcalledip IPs stored as integers (use INET_NTOA())
a_ua_id / b_ua_id FK to cdr_ua (User-Agent)

Quality Metrics

Packet Loss

Column Description
a_lost / b_lost Total RTP packets lost
packet_loss_perc_mult1000 Max loss percentage × 1000
a_sl1 - a_sl10 Distribution of consecutive loss events

Jitter & Delay

Column Description
a_avgjitter_mult10 Average jitter × 10
a_maxjitter Maximum jitter observed
a_d50 - a_d300 PDV distribution buckets (0-50ms, 50-70ms, etc.)

MOS (Mean Opinion Score)

Column Description
a_mos_f1_mult10 MOS with 50ms fixed buffer × 10
a_mos_f2_mult10 MOS with 200ms fixed buffer × 10
a_mos_adapt_mult10 MOS with adaptive buffer × 10
mos_min_mult10 Lowest MOS between A/B streams × 10

RTCP-Reported Metrics

These come from endpoint RTCP reports, not calculated by the sniffer. See Sniffer vs RTCP Packet Loss.

Column Description
a_rtcp_loss Packets lost (reported by endpoint)
a_rtcp_avgfr_mult10 Average fraction loss × 10
a_rtcp_avgjitter_mult10 Average jitter × 10
a_rtcp_avgrtd_mult10 Roundtrip delay (latency) in ms × 10

Related Tables

cdr_next (1:1 with cdr)

Extension table linked via cdr_ID.

Column Description
fbasename Derived from SIP Call-ID; used to link PCAP files. Special characters replaced with underscores (convertchar in config)
match_header Content of custom header for linking call legs (matchheader in config)

cdr_rtp (Per-Stream RTP Stats)

One row per RTP stream. Multiple streams per call.

Column Description
cdr_ID FK to cdr
ssrc RTP Synchronization Source identifier
saddr / daddr Source/destination IP (as integer)
sport / dport Source/destination port
payload Codec type. Video streams: payload ≥ 10000 (offset applied)
received / lost Packet counts

💡 Tip:

cdr_sdp (SDP-Negotiated Ports)

Stores media endpoints declared in SDP. Requires save_sdp_ipport = yes.

Column Description
ip / port Media IP and port from SDP
is_caller 1 = caller side, 0 = callee side

ℹ️ Note: cdr_sdp stores what endpoints agreed to use in SDP; cdr_rtp stores what was actually sent.

cdr_proxy (Proxy Chain)

IP addresses of SIP proxies the call traversed.

cdr_siphistory (SIP Method History)

Stores SIP requests/responses including REFER, BYE, CANCEL, OPTIONS, etc. Requires save_sip_history = all.

cdr_ua / cdr_sip_response (Lookup Tables)

Lookup tables for User-Agent strings and SIP response texts.

Common SQL Queries

Basic CDR Query with IP Conversion

SELECT
    ID, caller, called,
    INET_NTOA(sipcallerip) AS caller_ip,
    INET_NTOA(sipcalledip) AS called_ip,
    duration
FROM cdr
WHERE calldate >= '2024-01-01'
  AND sipcallerip = INET_ATON('192.168.1.50')
LIMIT 10;

Get PCAP Filename (fbasename)

SELECT c.ID, c.caller, c.called, cn.fbasename
FROM cdr c
JOIN cdr_next cn ON c.ID = cn.cdr_ID
WHERE c.ID = 12345;

Get User-Agent Strings

SELECT c.ID, ua_a.ua AS caller_ua, ua_b.ua AS callee_ua
FROM cdr c
LEFT JOIN cdr_ua ua_a ON c.a_ua_id = ua_a.ID
LEFT JOIN cdr_ua ua_b ON c.b_ua_id = ua_b.ID
WHERE c.calldate >= CURDATE() - INTERVAL 1 DAY
LIMIT 10;

Find Calls Through Specific Proxy

SELECT c.ID, c.calldate, c.caller, c.called,
       INET_NTOA(cp.dst) AS proxy_ip
FROM cdr c
JOIN cdr_proxy cp ON c.ID = cp.cdr_ID
WHERE cp.dst = INET_ATON('192.168.1.100')
LIMIT 100;

Query Video Stream Quality

SELECT cdr_ID, payload,
       INET_NTOA(saddr) AS src_ip,
       received, lost,
       (lost / (received + lost)) * 100 AS loss_pct
FROM cdr_rtp
WHERE payload >= 10000  -- Video streams only
  AND cdr_ID = 12345;

Find Calls with Specific SIP Method (REFER)

SELECT cdr_ID, callid, calldate
FROM cdr_siphistory
WHERE method = 'REFER'
  AND calldate > '2024-01-01'
GROUP BY cdr_ID;

Custom Headers

CDR-Level Custom Headers

CDR custom headers are stored in cdr_next_X tables. Use cdr_custom_headers to find the location:

-- Step 1: Find table/column for header
SELECT dynamic_table, dynamic_column
FROM cdr_custom_headers
WHERE header_field = 'X-Customer-ID';
-- Result: dynamic_table=2, dynamic_column=1 → cdr_next_2.custom_header_1

-- Step 2: Query the data
SELECT c.ID, c.calldate, cn.custom_header_1 AS customer_id
FROM cdr c
JOIN cdr_next_2 cn ON c.ID = cn.cdr_ID
WHERE cn.custom_header_1 = '12345';

ℹ️ Note: The GUI's Reports module does not support grouping by custom headers. Use direct SQL queries.

cdr_custom_headers Table Structure

The cdr_custom_headers table stores the configuration for custom header extraction:

Column Description
header_field SIP header name to capture
dynamic_table Target table number (cdr_next_X)
dynamic_column Target column number (custom_header_X)
type Extraction type (first/last/nth)
nth_occurrence (New in 35.x) Specific occurrence number to extract when type is "nth" (e.g., 2 for second occurrence)

ℹ️ Note: The nth_occurrence column uses packet timestamps for accurate ordering, ensuring correct results even when packet time-based reordering is disabled in the sniffer.

SIP Message Custom Headers

Per-message headers are stored in sip_msg_next_X tables. Use sip_msg_custom_headers to find the location.

System Table

The system table stores configuration including manager_key for sensor operations.

SELECT * FROM voipmonitor.`system` WHERE type='manager_key'\G

⚠️ Warning: If sniffer fails with "failed read rsa key", check that mysqlloadconfig is NOT set to no in voipmonitor.conf.

Synchronizing manager_key Between Databases

When multiple GUI instances share sensors (e.g., after database migration), mismatched manager_key values cause PCAP download failures.

Symptoms: "Unable to download PCAP" errors, empty SIP history in one GUI.

Fix:

-- On active database: get the key
SELECT content FROM voipmonitor.`system` WHERE type='manager_key';

-- On old database: update the key
UPDATE voipmonitor.`system`
SET content = '{"key":"<key_from_active>","iv":"<iv_from_active>"}'
WHERE type = 'manager_key';

Users must log out and log back in after the update.

Troubleshooting

Missing sport/dport in cdr_rtp

If RTP streams from overlapping calls are incorrectly merged, add the missing columns:

ALTER TABLE cdr_rtp ADD `sport` smallint unsigned DEFAULT NULL AFTER `saddr`;
ALTER TABLE cdr_rtp ADD `dport` smallint unsigned DEFAULT NULL AFTER `daddr`;

⚠️ Warning: ALTER TABLE can lock large tables for hours. Run during maintenance windows.

To find other recommended schema changes:

journalctl -u voipmonitor | grep ALTER

See Also


AI Summary for RAG

Summary: VoIPmonitor database schema reference. Main table cdr stores one record per call with timing, participants, and quality metrics. Column naming: a_=caller, b_=callee, _multNN=divide by NN. Key tables: cdr_next (1:1, contains fbasename for PCAP linking), cdr_rtp (per-stream stats, video streams have payload≥10000), cdr_proxy (proxy chain IPs), cdr_sdp (SDP-negotiated ports), cdr_siphistory (SIP method history, requires save_sip_history=all). Custom headers stored in cdr_next_X tables; query cdr_custom_headers for mappings. System table stores manager_key; mismatch causes PCAP download failures.

Keywords: database, schema, cdr, mysql, cdr_next, fbasename, cdr_rtp, video payload, cdr_proxy, cdr_sdp, cdr_siphistory, INET_NTOA, INET_ATON, sipcallerip, custom headers, cdr_custom_headers, manager_key, packet loss, jitter, mos, rtcp, a_rtcp_avgrtd_mult10, latency

Key Questions:

  • What do a_ and b_ prefixes mean in cdr table columns?
  • How do I convert sipcallerip to readable IP address?
  • How do I get the PCAP filename (fbasename) for a call?
  • How do I query video streams in cdr_rtp (payload >= 10000)?
  • How do I find calls that went through a specific proxy?
  • How do I find calls with REFER or other SIP methods?
  • How do I query CDR custom headers?
  • Why does "failed read rsa key" error occur on startup?
  • How do I sync manager_key between multiple databases?
  • Why can't old GUI download PCAP files after database migration?