Database structure: Difference between revisions

From VoIPmonitor.org
(Add cdr_custom_headers table documentation for CDR-level custom SIP headers)
(VS-1789: Document nth_occurrence column in cdr_custom_headers table)
 
(17 intermediate revisions by the same user not shown)
Line 1: Line 1:
{{DISPLAYTITLE:CDR Database Table Schema}}
{{DISPLAYTITLE:CDR Database Table Schema}}


'''This document provides an overview of the most important tables and columns in the VoIPmonitor database schema. It is intended as a reference for administrators, developers, or anyone directly querying the database.'''
'''Database schema reference for VoIPmonitor. The <code>cdr</code> table stores one record per call; related tables store RTP statistics, custom headers, and metadata.'''


The core of the VoIPmonitor database is the <code>cdr</code> (Call Detail Record) table, which stores one primary record for each monitored call. Many other tables are linked to <code>cdr</code> to store additional, more detailed information.
= Schema Overview =


== Database Schema Overview ==
<syntaxhighlight lang="sql">
 
-- List all CDR-related tables
The following diagram shows the relationships between the main VoIPmonitor database tables:
SHOW TABLES LIKE 'cdr%';
</syntaxhighlight>


<kroki lang="plantuml">
<kroki lang="plantuml">
Line 18: Line 19:
   * ID : bigint <<PK>>
   * ID : bigint <<PK>>
   --
   --
   id_sensor : int
   id_sensor, calldate, callend
  calldate : datetime
   caller, called, duration
  callend : datetime
   sipcallerip, sipcalledip
   caller : varchar
   a_lost, b_lost, a_mos_f1_mult10
  called : varchar
   lastSIPresponse_id <<FK>>
   sipcallerip : int
   a_ua_id, b_ua_id <<FK>>
  sipcalledip : int
   duration : int
   connect_duration : int
  a_ua_id : int <<FK>>
   b_ua_id : int <<FK>>
  lastSIPresponse_id : int <<FK>>
  a_lost / b_lost : int
  a_mos_f1_mult10 : tinyint
  ...
}
}


entity "cdr_next" as cdr_next {
entity "cdr_next" as cdr_next {
   * cdr_ID : bigint <<PK,FK>>
   * cdr_ID <<PK,FK>>
   --
   --
   fbasename : varchar
   fbasename, match_header
  match_header : varchar
  ...
}
}


entity "cdr_rtp" as cdr_rtp {
entity "cdr_rtp" as cdr_rtp {
   * ID : bigint <<PK>>
   cdr_ID <<FK>>
   --
   --
  cdr_ID : bigint <<FK>>
   saddr, daddr, ssrc
   saddr : int
   received, lost, payload
  daddr : int
  ssrc : bigint
   received : int
  lost : int
  ...
}
}


entity "cdr_dtmf" as cdr_dtmf {
entity "cdr_proxy" as cdr_proxy {
   * ID : bigint <<PK>>
   cdr_ID <<FK>>
   --
   --
   cdr_ID : bigint <<FK>>
   dst (proxy IP)
  dtmf : char
  ts : bigint
  ...
}
}


entity "cdr_proxy" as cdr_proxy {
entity "cdr_sdp" as cdr_sdp {
   * ID : bigint <<PK>>
   cdr_id <<FK>>
   --
   --
   cdr_ID : bigint <<FK>>
   ip, port, is_caller
  dst : int
  ...
}
}


entity "cdr_sip_response" as cdr_sip_response {
entity "cdr_siphistory" as cdr_siphistory {
   * ID : int <<PK>>
   cdr_ID <<FK>>
   --
   --
   lastSIPresponse : varchar
   method, calldate, cseq
}
}


entity "cdr_ua" as cdr_ua {
entity "cdr_ua" as cdr_ua {
   * ID : int <<PK>>
   * ID <<PK>>
   --
   --
   ua : varchar
   ua (User-Agent string)
}
 
entity "cdr_sip_response" as cdr_sip_response {
  * ID <<PK>>
  --
  lastSIPresponse
}
}


cdr ||--|| cdr_next : "1:1"
cdr ||--|| cdr_next : "1:1"
cdr ||--o{ cdr_rtp : "1:N"
cdr ||--o{ cdr_rtp : "1:N"
cdr ||--o{ cdr_dtmf : "1:N"
cdr ||--o{ cdr_proxy : "1:N"
cdr ||--o{ cdr_proxy : "1:N"
cdr ||--o{ cdr_sdp : "1:N"
cdr ||--o{ cdr_siphistory : "1:N"
cdr }o--|| cdr_ua : "N:1"
cdr }o--|| cdr_sip_response : "N:1"
cdr }o--|| cdr_sip_response : "N:1"
cdr }o--|| cdr_ua : "N:1 (a_ua_id)"
cdr }o--|| cdr_ua : "N:1 (b_ua_id)"
@enduml
@enduml
</kroki>
</kroki>


== Understanding Column Naming Conventions ==
== Column Naming Conventions ==
 
Before diving into the columns, it's important to understand these common naming patterns:


{| class="wikitable"
{| class="wikitable"
Line 102: Line 86:
! Pattern !! Meaning !! Example
! Pattern !! Meaning !! Example
|-
|-
| <code>a_</code> prefix || Caller's media stream (A-leg, RTP sent from caller) || <code>a_lost</code>, <code>a_mos_f1_mult10</code>
| <code>a_</code> || Caller's stream (A-leg) || <code>a_lost</code>, <code>a_mos_f1_mult10</code>
|-
|-
| <code>b_</code> prefix || Callee's media stream (B-leg, RTP sent from callee) || <code>b_lost</code>, <code>b_mos_f1_mult10</code>
| <code>b_</code> || Callee's stream (B-leg) || <code>b_lost</code>, <code>b_mos_f1_mult10</code>
|-
|-
| <code>_mult10</code> || Divide stored value by 10 to get real value || <code>jitter_mult10 = 15</code> means jitter = 1.5
| <code>_mult10</code> || Divide by 10 || <code>jitter_mult10 = 15</code> jitter = 1.5
|-
|-
| <code>_mult100</code> || Divide stored value by 100 to get real value || Used for percentage values
| <code>_mult100</code> || Divide by 100 || Percentage values
|-
|-
| <code>_mult1000</code> || Divide stored value by 1000 to get real value || <code>packet_loss_perc_mult1000</code>
| <code>_mult1000</code> || Divide by 1000 || <code>packet_loss_perc_mult1000</code>
|}
|}


== The <code>cdr</code> Table ==
= The <code>cdr</code> Table =
 
This is the main table containing the primary information for every call.


=== General Call Information ===
== Call Information & Timing ==


{| class="wikitable"
{| class="wikitable"
|-
|-
! Column !! Type !! Description
! Column !! Description
|-
| <code>ID</code> || Primary key
|-
| <code>id_sensor</code> || Sensor ID (from <code>voipmonitor.conf</code>)
|-
|-
| <code>ID</code> || bigint || The unique primary key for the record
| <code>calldate</code> / <code>callend</code> || Start/end timestamps
|-
|-
| <code>id_sensor</code> || int || The ID of the sensor that processed this call (corresponds to <code>id_sensor</code> in <code>voipmonitor.conf</code>)
| <code>duration</code> || Total length in seconds
|-
|-
| <code>calldate</code> || datetime || The start time of the call, measured from the first <code>INVITE</code> packet
| <code>connect_duration</code> || Connected time (after 200 OK)
|-
|-
| <code>callend</code> || datetime || The end time of the call, measured from the last packet associated with the call
| <code>progress_time</code> || Time to first provisional response ([[Glossary#Post_Dial_Delay_.28PDD.29|PDD]])
|-
|-
| <code>bye</code> || tinyint || Indicates how the call was terminated (<code>1</code> = normal termination with <code>BYE</code>)
| <code>bye</code> || <code>1</code> = normal termination with BYE
|-
|-
| <code>whohanged</code> || enum || Indicates which party sent the <code>BYE</code> message first (caller or callee)
| <code>whohanged</code> || Which party sent BYE (caller/callee)
|-
|-
| <code>lastSIPresponse_id</code> || int || Foreign key to <code>cdr_sip_response</code> table for the last final SIP response
| <code>lastSIPresponse_id</code> || FK to <code>cdr_sip_response</code>
|}
|}


=== Call Timing & Duration ===
== Participant Information ==


{| class="wikitable"
{| class="wikitable"
|-
|-
! Column !! Type !! Description
! Column !! Description
|-
|-
| <code>duration</code> || int || Total call length in seconds (<code>callend - calldate</code>)
| <code>caller</code> / <code>called</code> || Numbers from From/To headers
|-
|-
| <code>connect_duration</code> || int || Connected portion length in seconds (from first <code>200 OK</code> to <code>callend</code>)
| <code>caller_domain</code> / <code>called_domain</code> || Domain parts
|-
|-
| <code>progress_time</code> || int || Time from <code>INVITE</code> to first provisional response (e.g., <code>180 Ringing</code>). Also known as [[Glossary#Post_Dial_Delay_.28PDD.29|Post-Dial Delay (PDD)]]
| <code>sipcallerip</code> / <code>sipcalledip</code> || IPs stored as integers (use <code>INET_NTOA()</code>)
|-
|-
| <code>first_rtp_time</code> || int || Time from call start to first RTP packet detection
| <code>a_ua_id</code> / <code>b_ua_id</code> || FK to <code>cdr_ua</code> (User-Agent)
|}
|}


=== Participant Information ===
== Quality Metrics ==


=== Packet Loss ===
{| class="wikitable"
{| class="wikitable"
|-
|-
! Column !! Type !! Description
! Column !! Description
|-
|-
| <code>caller</code> / <code>called</code> || varchar || User part of the number from <code>From</code> and <code>To</code> SIP headers
| <code>a_lost</code> / <code>b_lost</code> || Total RTP packets lost
|-
|-
| <code>caller_domain</code> / <code>called_domain</code> || varchar || Domain part from <code>From</code> and <code>To</code> SIP headers
| <code>packet_loss_perc_mult1000</code> || Max loss percentage × 1000
|-
|-
| <code>caller_reverse</code> / <code>called_reverse</code> || varchar || Reversed strings for efficient <code>LIKE '%search'</code> queries
| <code>a_sl1</code> - <code>a_sl10</code> || Distribution of consecutive loss events
|-
| <code>callername</code> || varchar || Display name from the <code>From</code> SIP header
|-
| <code>sipcallerip</code> / <code>sipcalledip</code> || int unsigned || Source/destination IP addresses stored as integers (use <code>INET_NTOA()</code> to convert)
|-
| <code>a_ua_id</code> / <code>b_ua_id</code> || int || Foreign keys to <code>cdr_ua</code> table for User-Agent strings
|}
|}


{{Note|
=== Jitter & Delay ===
'''How User-Agent is extracted:'''
* '''Caller (<code>a_ua_id</code>):''' From the <code>User-Agent</code> header in the SIP INVITE request
* '''Callee (<code>b_ua_id</code>):''' From the <code>User-Agent</code> header in the SIP response (typically <code>200 OK</code>)
* Blank values indicate the <code>User-Agent</code> header was not present
}}
 
'''Example: Converting IP addresses in SQL queries:'''
<syntaxhighlight lang="sql">
-- Get calls with readable IP addresses
SELECT
    ID,
    caller,
    called,
    INET_NTOA(sipcallerip) AS caller_ip,
    INET_NTOA(sipcalledip) AS called_ip,
    duration
FROM cdr
WHERE calldate >= '2024-01-01'
LIMIT 10;
</syntaxhighlight>
 
=== Network Quality Metrics ===
 
==== Packet Loss ====
 
{| class="wikitable"
{| class="wikitable"
|-
|-
! Column !! Description
! Column !! Description
|-
|-
| <code>a_lost</code> / <code>b_lost</code> || Total RTP packets lost (detected by sequence number gaps)
| <code>a_avgjitter_mult10</code> || Average jitter × 10
|-
|-
| <code>lost</code> || Higher value between <code>a_lost</code> and <code>b_lost</code>
| <code>a_maxjitter</code> || Maximum jitter observed
|-
|-
| <code>packet_loss_perc_mult1000</code> || Maximum packet loss percentage, multiplied by 1000
| <code>a_d50</code> - <code>a_d300</code> || PDV distribution buckets (0-50ms, 50-70ms, etc.)
|-
| <code>a_sl1</code> - <code>a_sl10</code> / <code>b_sl1</code> - <code>b_sl10</code> || Distribution of consecutive packet loss events (sl1 = single packet lost, sl5 = 5 packets in a row, etc.)
|}
|}


==== Jitter & Delay (PDV) ====
=== MOS (Mean Opinion Score) ===
 
{| class="wikitable"
{| class="wikitable"
|-
|-
! Column !! Description
! Column !! Description
|-
|-
| <code>a_avgjitter_mult10</code> / <code>b_avgjitter_mult10</code> || Average jitter per RFC 3550, multiplied by 10
| <code>a_mos_f1_mult10</code> || [[Comprehensive_Guide_to_VoIP_Voice_Quality#MOS_Calculation_in_VoIPmonitor|MOS]] with 50ms fixed buffer × 10
|-
|-
| <code>a_maxjitter</code> / <code>b_maxjitter</code> || Maximum jitter value observed during the call
| <code>a_mos_f2_mult10</code> || MOS with 200ms fixed buffer × 10
|-
|-
| <code>jitter_mult10</code> || Higher <code>maxjitter</code> value between streams, multiplied by 10
| <code>a_mos_adapt_mult10</code> || MOS with adaptive buffer × 10
|-
|-
| <code>a_d50</code> - <code>a_d300</code> / <code>b_d50</code> - <code>b_d300</code> || Packet Delay Variation (PDV) distribution buckets (d50 = 0-50ms, d70 = 50-70ms, etc.)
| <code>mos_min_mult10</code> || Lowest MOS between A/B streams × 10
|}
|}


==== MOS (Mean Opinion Score) ====
=== RTCP-Reported Metrics ===
These come from endpoint RTCP reports, not calculated by the sniffer. See [[Call_Detail_Record_-_CDR#Understanding_Sniffer_vs_RTCP_Packet_Loss|Sniffer vs RTCP Packet Loss]].


{| class="wikitable"
{| class="wikitable"
Line 232: Line 186:
! Column !! Description
! Column !! Description
|-
|-
| <code>a_mos_f1_mult10</code> / <code>b_mos_f1_mult10</code> || [[Comprehensive_Guide_to_VoIP_Voice_Quality#MOS_Calculation_in_VoIPmonitor|Parametric MOS]] with 50ms fixed jitter buffer, multiplied by 10
| <code>a_rtcp_loss</code> || Packets lost (reported by endpoint)
|-
|-
| <code>a_mos_f2_mult10</code> / <code>b_mos_f2_mult10</code> || MOS with 200ms fixed jitter buffer, multiplied by 10
| <code>a_rtcp_avgfr_mult10</code> || Average fraction loss × 10
|-
|-
| <code>a_mos_adapt_mult10</code> / <code>b_mos_adapt_mult10</code> || MOS with adaptive jitter buffer (up to 500ms), multiplied by 10
| <code>a_rtcp_avgjitter_mult10</code> || Average jitter × 10
|-
|-
| <code>mos_min_mult10</code> || Lowest MOS score between A and B streams, multiplied by 10
| <code>a_rtcp_avgrtd_mult10</code> || '''Roundtrip delay (latency)''' in ms × 10
|}
|}


==== RTCP Reported Metrics ====
= Related Tables =


These metrics come from RTCP Receiver Reports sent by the endpoints, not calculated by the sniffer. For a detailed explanation of the difference between sniffer-calculated and RTCP-reported metrics, see [[Call_Detail_Record_-_CDR#Understanding_Sniffer_vs_RTCP_Packet_Loss|Understanding Sniffer vs RTCP Packet Loss]].
== <code>cdr_next</code> (1:1 with cdr) ==
 
Extension table linked via <code>cdr_ID</code>.


{| class="wikitable"
{| class="wikitable"
Line 249: Line 205:
! Column !! Description
! Column !! Description
|-
|-
| <code>a_rtcp_loss</code> / <code>b_rtcp_loss</code> || Total packets lost as reported by endpoint via RTCP
| <code>fbasename</code> || Derived from SIP Call-ID; used to link PCAP files. Special characters replaced with underscores (<code>convertchar</code> in config)
|-
|-
| <code>a_rtcp_avgfr_mult10</code> / <code>b_rtcp_avgfr_mult10</code> || Average fraction loss from RTCP, multiplied by 10
| <code>match_header</code> || Content of custom header for linking call legs (<code>matchheader</code> in config)
|-
| <code>a_rtcp_avgjitter_mult10</code> / <code>b_rtcp_avgjitter_mult10</code> || Average jitter from RTCP, multiplied by 10
|}
|}


=== Other Columns ===
== <code>cdr_rtp</code> (Per-Stream RTP Stats) ==
 
One row per RTP stream. Multiple streams per call.


{| class="wikitable"
{| class="wikitable"
Line 262: Line 218:
! Column !! Description
! Column !! Description
|-
|-
| <code>dscp</code> || DSCP values from SIP and RTP packets for QoS analysis
| <code>cdr_ID</code> || FK to <code>cdr</code>
|-
| <code>ssrc</code> || RTP Synchronization Source identifier
|-
|-
| <code>payload</code> || Codec payload type number used during the call
| <code>saddr</code> / <code>daddr</code> || Source/destination IP (as integer)
|}
 
== Important Related Tables ==
 
=== <code>cdr_next</code> ===
 
A 1-to-1 extension of the <code>cdr</code> table for additional data, keeping the main table smaller.
 
{| class="wikitable"
|-
|-
! Column !! Description
| <code>sport</code> / <code>dport</code> || Source/destination port
|-
|-
| <code>fbasename</code> || String derived from SIP <code>Call-ID</code>, used to link PCAP files to CDR
| <code>payload</code> || Codec type. '''Video streams: payload ≥ 10000''' (offset applied)
|-
|-
| <code>match_header</code> || Content of custom header (from <code>matchheader</code> in <code>voipmonitor.conf</code>) for linking call legs
| <code>received</code> / <code>lost</code> || Packet counts
|}
|}


'''Example: Finding PCAP filename for a call:'''
{{Tip|To query video streams only: <code>WHERE payload >= 10000</code>}}
<syntaxhighlight lang="sql">
SELECT c.ID, c.caller, c.called, cn.fbasename
FROM cdr c
JOIN cdr_next cn ON c.ID = cn.cdr_ID
WHERE c.ID = 12345;
</syntaxhighlight>
 
{{Note|
'''CDR Custom Headers Storage:'''
CDR-level custom SIP headers (one value per call) are stored dynamically in the <code>cdr_next_X</code> tables, not in the main <code>cdr</code> table. To find where a specific header is stored, you must query the <code>cdr_custom_headers</code> mapping table.
}}


=== <code>cdr_custom_headers</code> ===
== <code>cdr_sdp</code> (SDP-Negotiated Ports) ==


Lookup table that maps custom SIP header names to their storage locations in the <code>cdr_next_X</code> tables. You must query this table first to find the correct table and column for your custom header.
Stores media endpoints declared in SDP. Requires <code>save_sdp_ipport = yes</code>.


{| class="wikitable"
{| class="wikitable"
Line 303: Line 241:
! Column !! Description
! Column !! Description
|-
|-
| <code>header_field</code> || The name of the custom SIP header (e.g., <code>'X-IntNum'</code>)
| <code>ip</code> / <code>port</code> || Media IP and port from SDP
|-
|-
| <code>dynamic_table</code> || The table number (e.g., <code>2</code> means <code>cdr_next_2</code>)
| <code>is_caller</code> || 1 = caller side, 0 = callee side
|-
| <code>dynamic_column</code> || The column number (e.g., <code>1</code> means <code>custom_header_1</code>)
|}
|}


{{Tip|
{{Note|<code>cdr_sdp</code> stores what endpoints '''agreed to use''' in SDP; <code>cdr_rtp</code> stores what was '''actually sent'''.}}
The system automatically creates new <code>cdr_next_X</code> tables when needed and updates this mapping table. Each header gets its own dedicated column, not each call.
 
}}
== <code>cdr_proxy</code> (Proxy Chain) ==
 
IP addresses of SIP proxies the call traversed.


=== Finding CDR Custom Header Storage ===
== <code>cdr_siphistory</code> (SIP Method History) ==


Because CDR custom headers are stored in dynamically created tables, you must use the <code>cdr_custom_headers</code> table to find the correct table and column.
Stores SIP requests/responses including REFER, BYE, CANCEL, OPTIONS, etc. Requires <code>save_sip_history = all</code>.


<syntaxhighlight lang="sql">
== <code>cdr_ua</code> / <code>cdr_sip_response</code> (Lookup Tables) ==
-- Step 1: Find the dynamic table and column for your custom header
SELECT dynamic_table, dynamic_column
FROM cdr_custom_headers
WHERE header_field = 'X-IntNum';
</syntaxhighlight>


This query returns:
Lookup tables for User-Agent strings and SIP response texts.
* <code>dynamic_table</code>: The table number (e.g., <code>2</code> means <code>cdr_next_2</code>)
* <code>dynamic_column</code>: The column number (e.g., <code>1</code> means <code>custom_header_1</code>)


=== Querying CDRs with Custom Header Values ===
= Common SQL Queries =


Once you have the table and column information, you can query the <code>cdr_next_X</code> table for calls containing the desired header value.
== Basic CDR Query with IP Conversion ==


<syntaxhighlight lang="sql">
<syntaxhighlight lang="sql">
-- Example: Query cdr_next_2 for custom_header_1
SELECT
SELECT
     cdr.id,
     ID, caller, called,
    cdr.calldate,
     INET_NTOA(sipcallerip) AS caller_ip,
     cdr.caller,
     INET_NTOA(sipcalledip) AS called_ip,
     cdr.called,
     duration
     cdr_next_2.custom_header_1 AS X_IntNum
FROM cdr
FROM cdr
JOIN cdr_next_2 ON cdr.id = cdr_next_2.cdr_ID
WHERE calldate >= '2024-01-01'
WHERE cdr_next_2.custom_header_1 IS NOT NULL
  AND sipcallerip = INET_ATON('192.168.1.50')
ORDER BY cdr.calldate DESC
LIMIT 10;
LIMIT 10;
</syntaxhighlight>
</syntaxhighlight>


=== Complete Example Workflow ===
== Get PCAP Filename (fbasename) ==


<syntaxhighlight lang="sql">
<syntaxhighlight lang="sql">
-- Example: Find CDRs where X-Customer-ID = '12345'
SELECT c.ID, c.caller, c.called, cn.fbasename
FROM cdr c
JOIN cdr_next cn ON c.ID = cn.cdr_ID
WHERE c.ID = 12345;
</syntaxhighlight>


-- Step 1: Find where X-Customer-ID is stored
== Get User-Agent Strings ==
SELECT dynamic_table, dynamic_column
 
FROM cdr_custom_headers
<syntaxhighlight lang="sql">
WHERE header_field = 'X-Customer-ID';
SELECT c.ID, ua_a.ua AS caller_ua, ua_b.ua AS callee_ua
FROM cdr c
LEFT JOIN cdr_ua ua_a ON c.a_ua_id = ua_a.ID
LEFT JOIN cdr_ua ua_b ON c.b_ua_id = ua_b.ID
WHERE c.calldate >= CURDATE() - INTERVAL 1 DAY
LIMIT 10;
</syntaxhighlight>


-- Result example: dynamic_table=2, dynamic_column=1
== Find Calls Through Specific Proxy ==
-- This means the header is stored in cdr_next_2.custom_header_1


-- Step 2: Query CDRs matching the custom header value
<syntaxhighlight lang="sql">
SELECT
SELECT c.ID, c.calldate, c.caller, c.called,
    c.ID,
      INET_NTOA(cp.dst) AS proxy_ip
    c.calldate,
    c.caller,
    c.called,
    c.duration,
    cn.custom_header_1 AS customer_id
FROM cdr c
FROM cdr c
JOIN cdr_next_2 cn ON c.ID = cn.cdr_ID
JOIN cdr_proxy cp ON c.ID = cp.cdr_ID
WHERE cn.custom_header_1 = '12345'
WHERE cp.dst = INET_ATON('192.168.1.100')
ORDER BY c.calldate DESC
LIMIT 100;
LIMIT 100;
</syntaxhighlight>
</syntaxhighlight>


{{Tip|
== Query Video Stream Quality ==
You can use variables or dynamic SQL to combine both steps into a single query when working programmatically.
}}


=== <code>cdr_rtp</code> ===
<syntaxhighlight lang="sql">
SELECT cdr_ID, payload,
      INET_NTOA(saddr) AS src_ip,
      received, lost,
      (lost / (received + lost)) * 100 AS loss_pct
FROM cdr_rtp
WHERE payload >= 10000  -- Video streams only
  AND cdr_ID = 12345;
</syntaxhighlight>


Stores detailed statistics for each individual RTP stream within a call. Multiple rows per call (one per stream).
== Find Calls with Specific SIP Method (REFER) ==


=== <code>cdr_dtmf</code> ===
<syntaxhighlight lang="sql">
SELECT cdr_ID, callid, calldate
FROM cdr_siphistory
WHERE method = 'REFER'
  AND calldate > '2024-01-01'
GROUP BY cdr_ID;
</syntaxhighlight>


Stores detected DTMF key presses. Requires <code>dtmf2db = yes</code> in <code>voipmonitor.conf</code>.
= Custom Headers =


=== <code>cdr_proxy</code> ===
== CDR-Level Custom Headers ==


Stores IP addresses of all SIP proxies the call traversed.
CDR custom headers are stored in <code>cdr_next_X</code> tables. Use <code>cdr_custom_headers</code> to find the location:


=== <code>cdr_sip_response</code> ===
<syntaxhighlight lang="sql">
 
-- Step 1: Find table/column for header
Lookup table for unique SIP response texts (e.g., "404 Not Found"). The <code>cdr</code> table references by ID to save space.
SELECT dynamic_table, dynamic_column
 
FROM cdr_custom_headers
=== <code>cdr_ua</code> ===
WHERE header_field = 'X-Customer-ID';
 
-- Result: dynamic_table=2, dynamic_column=1 → cdr_next_2.custom_header_1
Lookup table for <code>User-Agent</code> strings of SIP devices.


'''Example: Getting User-Agent strings for calls:'''
-- Step 2: Query the data
<syntaxhighlight lang="sql">
SELECT c.ID, c.calldate, cn.custom_header_1 AS customer_id
SELECT
    c.ID,
    c.caller,
    c.called,
    ua_a.ua AS caller_user_agent,
    ua_b.ua AS callee_user_agent
FROM cdr c
FROM cdr c
LEFT JOIN cdr_ua ua_a ON c.a_ua_id = ua_a.ID
JOIN cdr_next_2 cn ON c.ID = cn.cdr_ID
LEFT JOIN cdr_ua ua_b ON c.b_ua_id = ua_b.ID
WHERE cn.custom_header_1 = '12345';
WHERE c.calldate >= CURDATE() - INTERVAL 1 DAY
LIMIT 10;
</syntaxhighlight>
</syntaxhighlight>


=== <code>cdr_siphistory</code> ===
{{Note|The GUI's [[Reports|Reports]] module does not support grouping by custom headers. Use direct SQL queries.}}
=== <code>cdr_custom_headers</code> Table Structure ===


Stores detailed SIP request and response history for each call, including subdialog methods such as `REFER`, `BYE`, `CANCEL`, `OPTIONS`, `SUBSCRIBE`, `NOTIFY`, and `MESSAGE`. This table enables filtering CDRs by specific SIP methods via the GUI filter dropdown. Requires <code>save_sip_history = all</code> in <code>voipmonitor.conf</code>.
The <code>cdr_custom_headers</code> table stores the configuration for custom header extraction:


{| class="wikitable"
{| class="wikitable"
Line 423: Line 358:
! Column !! Description
! Column !! Description
|-
|-
| <code>cdr_ID</code> || Foreign key to <code>cdr</code> table (links to the primary call record)
| <code>header_field</code> || SIP header name to capture
|-
|-
| <code>method</code> || SIP request method (e.g., INVITE, REFER, BYE, CANCEL, OPTIONS, etc.)
| <code>dynamic_table</code> || Target table number (cdr_next_X)
|-
|-
| <code>calldate</code> || Timestamp of the SIP message
| <code>dynamic_column</code> || Target column number (custom_header_X)
|-
|-
| <code>callid</code> || SIP Call-ID header value
| <code>type</code> || Extraction type (first/last/nth)
|-
|-
| <code>cseq</code> || SIP CSeq (Command Sequence) number
| <code>nth_occurrence</code> || '''(New in 35.x)''' Specific occurrence number to extract when type is "nth" (e.g., 2 for second occurrence)
|}
|}


'''Prerequisite Configuration:'''
{{Note|The <code>nth_occurrence</code> column uses packet timestamps for accurate ordering, ensuring correct results even when packet time-based reordering is disabled in the sniffer.}}
Configure in <code>/etc/voipmonitor.conf</code>:
== SIP Message Custom Headers ==
<syntaxhighlight lang="ini">
 
save_sip_history = all
Per-message headers are stored in <code>sip_msg_next_X</code> tables. Use <code>sip_msg_custom_headers</code> to find the location.
</syntaxhighlight>
 
Restart the voipmonitor sniffer after changing this configuration.
= System Table =


'''Example: Find all calls containing a SIP REFER method:'''
The <code>system</code> table stores configuration including <code>manager_key</code> for sensor operations.
<syntaxhighlight lang="sql">
SELECT
    cdr_ID,
    callid,
    calldate
FROM cdr_siphistory
WHERE calldate > '2024-01-01 00:00:00'
  AND method = 'REFER'
GROUP BY cdr_ID;
</syntaxhighlight>


'''Example: Find calls with OPTIONS or SUBSCRIBE messages (for troubleshooting qualify pings):'''
<syntaxhighlight lang="sql">
<syntaxhighlight lang="sql">
SELECT
SELECT * FROM voipmonitor.`system` WHERE type='manager_key'\G
    cdr_ID,
    callid,
    method,
    calldate
FROM cdr_siphistory
WHERE calldate > CURDATE() - INTERVAL 1 DAY
  AND method IN ('OPTIONS', 'SUBSCRIBE')
ORDER BY calldate DESC
LIMIT 20;
</syntaxhighlight>
</syntaxhighlight>


== SIP Message Custom Headers ==
{{Warning|1=If sniffer fails with "failed read rsa key", check that <code>mysqlloadconfig</code> is NOT set to <code>no</code> in <code>voipmonitor.conf</code>.}}


VoIPmonitor can capture custom SIP headers from individual SIP messages (not just CDR-level headers). These are stored dynamically in the <code>sip_msg_next_X</code> tables, where each custom header gets its own dedicated column.
== Synchronizing manager_key Between Databases ==


{{Note|
When multiple GUI instances share sensors (e.g., after database migration), mismatched <code>manager_key</code> values cause PCAP download failures.
'''CDR vs SIP Message Headers:'''
* <strong>CDR custom headers</strong> are stored in the <code>cdr_next</code> table for call-level data (one value per call)
* <strong>SIP message custom headers</strong> are stored in <code>sip_msg_next_X</code> tables for individual SIP messages (one value per message)
}}


=== Finding the Table and Column for a Custom Header ===
'''Symptoms:''' "Unable to download PCAP" errors, empty SIP history in one GUI.


Because custom SIP headers are stored in dynamically created tables, you must first query the <code>sip_msg_custom_headers</code> table to find which table and column contains your header.
'''Fix:'''
<syntaxhighlight lang="sql">
-- On active database: get the key
SELECT content FROM voipmonitor.`system` WHERE type='manager_key';


<syntaxhighlight lang="sql">
-- On old database: update the key
-- Step 1: Find the dynamic table and column for your custom header
UPDATE voipmonitor.`system`
SELECT dynamic_table, dynamic_column
SET content = '{"key":"<key_from_active>","iv":"<iv_from_active>"}'
FROM sip_msg_custom_headers
WHERE type = 'manager_key';
WHERE header_field = 'X-My-Custom-Header';
</syntaxhighlight>
</syntaxhighlight>


This query returns:
Users must log out and log back in after the update.
* <code>dynamic_table</code>: The table number (e.g., <code>1</code> means <code>sip_msg_next_1</code>)
* <code>dynamic_column</code>: The column number (e.g., <code>2</code> means <code>custom_header_2</code>)


=== Querying SIP Messages with Custom Header Values ===
= Troubleshooting =


Once you have the table and column information, you can query the <code>sip_msg_next_X</code> table for SIP messages containing the desired header value.
== Missing <code>sport</code>/<code>dport</code> in cdr_rtp ==
 
If RTP streams from overlapping calls are incorrectly merged, add the missing columns:


<syntaxhighlight lang="sql">
<syntaxhighlight lang="sql">
-- Example: Query sip_msg_next_1 for custom_header_2 = 'desired-value'
ALTER TABLE cdr_rtp ADD `sport` smallint unsigned DEFAULT NULL AFTER `saddr`;
-- Step 2: Retrieve SIP messages matching the custom header value
ALTER TABLE cdr_rtp ADD `dport` smallint unsigned DEFAULT NULL AFTER `daddr`;
SELECT
    sip_msg_ID,
    time,
    custom_header_2 AS header_value
FROM sip_msg_next_1
WHERE custom_header_2 = 'desired-value'
  AND time > NOW() - INTERVAL 15 MINUTE;
</syntaxhighlight>
</syntaxhighlight>


{{Tip|
{{Warning|1=<code>ALTER TABLE</code> can lock large tables for hours. Run during maintenance windows.}}
You can also join this with other SIP message tables if needed. The SIP messages are linked to CDRs through the <code>sip_msg_ID</code> column and related message tables.
}}


=== Complete Example Workflow ===
To find other recommended schema changes:
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="sql">
journalctl -u voipmonitor | grep ALTER
-- Example: Find SIP messages with X-Customer-ID = '12345' in the last 1 hour
 
-- Step 1: Find where X-Customer-ID is stored
SELECT dynamic_table, dynamic_column
FROM sip_msg_custom_headers
WHERE header_field = 'X-Customer-ID';
 
-- Example result: dynamic_table=3, dynamic_column=1
 
-- Step 2: Query the appropriate table
SELECT
    sm.sip_msg_ID,
    sm.time,
    sm.custom_header_1 AS customer_id
FROM sip_msg_next_3 sm
WHERE sm.custom_header_1 = '12345'
  AND sm.time > NOW() - INTERVAL 1 HOUR
ORDER BY sm.time DESC;
</syntaxhighlight>
</syntaxhighlight>


=== Configuration Requirements ===
= See Also =
 
To capture custom SIP headers, configure them in <code>/etc/voipmonitor.conf</code>:


<syntaxhighlight lang="ini">
* [[Call_Detail_Record_-_CDR|CDR View (GUI)]]
[general]
* [[Database_troubleshooting|Database Troubleshooting]]
# List custom headers to capture from SIP messages
* [[Sniffer_configuration#Database_Configuration|Database Configuration]]
sip_headers = X-Customer-ID, X-Branch-Code, X-Transaction-ID
* [[WEB_API|Web API Documentation]]
</syntaxhighlight>


After modifying the configuration, restart the sensor:
<syntaxhighlight lang="bash">
service voipmonitor restart
</syntaxhighlight>


The system will automatically create new columns in the appropriate <code>sip_msg_next_X</code> table and update the <code>sip_msg_custom_headers</code> mapping table.
= AI Summary for RAG =


== AI Summary for RAG ==
'''Summary:''' VoIPmonitor database schema reference. Main table <code>cdr</code> stores one record per call with timing, participants, and quality metrics. Column naming: <code>a_</code>=caller, <code>b_</code>=callee, <code>_multNN</code>=divide by NN. Key tables: <code>cdr_next</code> (1:1, contains <code>fbasename</code> for PCAP linking), <code>cdr_rtp</code> (per-stream stats, video streams have payload≥10000), <code>cdr_proxy</code> (proxy chain IPs), <code>cdr_sdp</code> (SDP-negotiated ports), <code>cdr_siphistory</code> (SIP method history, requires <code>save_sip_history=all</code>). Custom headers stored in <code>cdr_next_X</code> tables; query <code>cdr_custom_headers</code> for mappings. System table stores <code>manager_key</code>; mismatch causes PCAP download failures.
'''Summary:''' Reference for VoIPmonitor database schema. The <code>cdr</code> table stores one record per call with timing, participant info, and quality metrics. Column naming: <code>a_</code> = caller stream, <code>b_</code> = callee stream, <code>_multNN</code> = divide by NN. Key metrics include packet loss (<code>a_lost</code>), jitter (<code>a_avgjitter_mult10</code>), MOS scores, and RTCP-reported metrics. Related tables: <code>cdr_next</code> (Call-ID via <code>fbasename</code>), <code>cdr_rtp</code> (per-stream stats), <code>cdr_ua</code> (User-Agent lookup). CDR custom headers require querying <code>cdr_custom_headers</code> to find dynamic table/column mappings, then querying <code>cdr_next_X</code> tables. SIP message custom headers use <code>sip_msg_custom_headers</code> and <code>sip_msg_next_X</code> tables.


'''Keywords:''' database, schema, cdr, mysql, mariadb, packet loss, jitter, mos, rtcp, a_lost, b_lost, fbasename, user-agent, cdr_next, cdr_rtp, sip_msg, custom headers, cdr_custom_headers, cdr_next_X, sip_msg_custom_headers, sip_msg_next
'''Keywords:''' database, schema, cdr, mysql, cdr_next, fbasename, cdr_rtp, video payload, cdr_proxy, cdr_sdp, cdr_siphistory, INET_NTOA, INET_ATON, sipcallerip, custom headers, cdr_custom_headers, manager_key, packet loss, jitter, mos, rtcp, a_rtcp_avgrtd_mult10, latency


'''Key Questions:'''
'''Key Questions:'''
* What do the a_ and b_ prefixes mean in the cdr table?
* What do a_ and b_ prefixes mean in cdr table columns?
* How do I convert sipcallerip to a readable IP address?
* How do I convert sipcallerip to readable IP address?
* How do I get the Call-ID for a call from the database?
* How do I get the PCAP filename (fbasename) for a call?
* Which table stores the User-Agent of phones?
* How do I query video streams in cdr_rtp (payload >= 10000)?
* How do I query CDRs with a custom header value?
* How do I find calls that went through a specific proxy?
* How do I find which cdr_next_X table stores my custom header?
* How do I find calls with REFER or other SIP methods?
* How do I query SIP messages with a custom header value?
* How do I query CDR custom headers?
* Why is my custom SIP header not in the cdr table?
* Why does "failed read rsa key" error occur on startup?
* How do I sync manager_key between multiple databases?
* Why can't old GUI download PCAP files after database migration?

Latest revision as of 11:29, 19 January 2026


Database schema reference for VoIPmonitor. The cdr table stores one record per call; related tables store RTP statistics, custom headers, and metadata.

Schema Overview

-- List all CDR-related tables
SHOW TABLES LIKE 'cdr%';

Column Naming Conventions

Pattern Meaning Example
a_ Caller's stream (A-leg) a_lost, a_mos_f1_mult10
b_ Callee's stream (B-leg) b_lost, b_mos_f1_mult10
_mult10 Divide by 10 jitter_mult10 = 15 → jitter = 1.5
_mult100 Divide by 100 Percentage values
_mult1000 Divide by 1000 packet_loss_perc_mult1000

The cdr Table

Call Information & Timing

Column Description
ID Primary key
id_sensor Sensor ID (from voipmonitor.conf)
calldate / callend Start/end timestamps
duration Total length in seconds
connect_duration Connected time (after 200 OK)
progress_time Time to first provisional response (PDD)
bye 1 = normal termination with BYE
whohanged Which party sent BYE (caller/callee)
lastSIPresponse_id FK to cdr_sip_response

Participant Information

Column Description
caller / called Numbers from From/To headers
caller_domain / called_domain Domain parts
sipcallerip / sipcalledip IPs stored as integers (use INET_NTOA())
a_ua_id / b_ua_id FK to cdr_ua (User-Agent)

Quality Metrics

Packet Loss

Column Description
a_lost / b_lost Total RTP packets lost
packet_loss_perc_mult1000 Max loss percentage × 1000
a_sl1 - a_sl10 Distribution of consecutive loss events

Jitter & Delay

Column Description
a_avgjitter_mult10 Average jitter × 10
a_maxjitter Maximum jitter observed
a_d50 - a_d300 PDV distribution buckets (0-50ms, 50-70ms, etc.)

MOS (Mean Opinion Score)

Column Description
a_mos_f1_mult10 MOS with 50ms fixed buffer × 10
a_mos_f2_mult10 MOS with 200ms fixed buffer × 10
a_mos_adapt_mult10 MOS with adaptive buffer × 10
mos_min_mult10 Lowest MOS between A/B streams × 10

RTCP-Reported Metrics

These come from endpoint RTCP reports, not calculated by the sniffer. See Sniffer vs RTCP Packet Loss.

Column Description
a_rtcp_loss Packets lost (reported by endpoint)
a_rtcp_avgfr_mult10 Average fraction loss × 10
a_rtcp_avgjitter_mult10 Average jitter × 10
a_rtcp_avgrtd_mult10 Roundtrip delay (latency) in ms × 10

Related Tables

cdr_next (1:1 with cdr)

Extension table linked via cdr_ID.

Column Description
fbasename Derived from SIP Call-ID; used to link PCAP files. Special characters replaced with underscores (convertchar in config)
match_header Content of custom header for linking call legs (matchheader in config)

cdr_rtp (Per-Stream RTP Stats)

One row per RTP stream. Multiple streams per call.

Column Description
cdr_ID FK to cdr
ssrc RTP Synchronization Source identifier
saddr / daddr Source/destination IP (as integer)
sport / dport Source/destination port
payload Codec type. Video streams: payload ≥ 10000 (offset applied)
received / lost Packet counts

💡 Tip:

cdr_sdp (SDP-Negotiated Ports)

Stores media endpoints declared in SDP. Requires save_sdp_ipport = yes.

Column Description
ip / port Media IP and port from SDP
is_caller 1 = caller side, 0 = callee side

ℹ️ Note: cdr_sdp stores what endpoints agreed to use in SDP; cdr_rtp stores what was actually sent.

cdr_proxy (Proxy Chain)

IP addresses of SIP proxies the call traversed.

cdr_siphistory (SIP Method History)

Stores SIP requests/responses including REFER, BYE, CANCEL, OPTIONS, etc. Requires save_sip_history = all.

cdr_ua / cdr_sip_response (Lookup Tables)

Lookup tables for User-Agent strings and SIP response texts.

Common SQL Queries

Basic CDR Query with IP Conversion

SELECT
    ID, caller, called,
    INET_NTOA(sipcallerip) AS caller_ip,
    INET_NTOA(sipcalledip) AS called_ip,
    duration
FROM cdr
WHERE calldate >= '2024-01-01'
  AND sipcallerip = INET_ATON('192.168.1.50')
LIMIT 10;

Get PCAP Filename (fbasename)

SELECT c.ID, c.caller, c.called, cn.fbasename
FROM cdr c
JOIN cdr_next cn ON c.ID = cn.cdr_ID
WHERE c.ID = 12345;

Get User-Agent Strings

SELECT c.ID, ua_a.ua AS caller_ua, ua_b.ua AS callee_ua
FROM cdr c
LEFT JOIN cdr_ua ua_a ON c.a_ua_id = ua_a.ID
LEFT JOIN cdr_ua ua_b ON c.b_ua_id = ua_b.ID
WHERE c.calldate >= CURDATE() - INTERVAL 1 DAY
LIMIT 10;

Find Calls Through Specific Proxy

SELECT c.ID, c.calldate, c.caller, c.called,
       INET_NTOA(cp.dst) AS proxy_ip
FROM cdr c
JOIN cdr_proxy cp ON c.ID = cp.cdr_ID
WHERE cp.dst = INET_ATON('192.168.1.100')
LIMIT 100;

Query Video Stream Quality

SELECT cdr_ID, payload,
       INET_NTOA(saddr) AS src_ip,
       received, lost,
       (lost / (received + lost)) * 100 AS loss_pct
FROM cdr_rtp
WHERE payload >= 10000  -- Video streams only
  AND cdr_ID = 12345;

Find Calls with Specific SIP Method (REFER)

SELECT cdr_ID, callid, calldate
FROM cdr_siphistory
WHERE method = 'REFER'
  AND calldate > '2024-01-01'
GROUP BY cdr_ID;

Custom Headers

CDR-Level Custom Headers

CDR custom headers are stored in cdr_next_X tables. Use cdr_custom_headers to find the location:

-- Step 1: Find table/column for header
SELECT dynamic_table, dynamic_column
FROM cdr_custom_headers
WHERE header_field = 'X-Customer-ID';
-- Result: dynamic_table=2, dynamic_column=1 → cdr_next_2.custom_header_1

-- Step 2: Query the data
SELECT c.ID, c.calldate, cn.custom_header_1 AS customer_id
FROM cdr c
JOIN cdr_next_2 cn ON c.ID = cn.cdr_ID
WHERE cn.custom_header_1 = '12345';

ℹ️ Note: The GUI's Reports module does not support grouping by custom headers. Use direct SQL queries.

cdr_custom_headers Table Structure

The cdr_custom_headers table stores the configuration for custom header extraction:

Column Description
header_field SIP header name to capture
dynamic_table Target table number (cdr_next_X)
dynamic_column Target column number (custom_header_X)
type Extraction type (first/last/nth)
nth_occurrence (New in 35.x) Specific occurrence number to extract when type is "nth" (e.g., 2 for second occurrence)

ℹ️ Note: The nth_occurrence column uses packet timestamps for accurate ordering, ensuring correct results even when packet time-based reordering is disabled in the sniffer.

SIP Message Custom Headers

Per-message headers are stored in sip_msg_next_X tables. Use sip_msg_custom_headers to find the location.

System Table

The system table stores configuration including manager_key for sensor operations.

SELECT * FROM voipmonitor.`system` WHERE type='manager_key'\G

⚠️ Warning: If sniffer fails with "failed read rsa key", check that mysqlloadconfig is NOT set to no in voipmonitor.conf.

Synchronizing manager_key Between Databases

When multiple GUI instances share sensors (e.g., after database migration), mismatched manager_key values cause PCAP download failures.

Symptoms: "Unable to download PCAP" errors, empty SIP history in one GUI.

Fix:

-- On active database: get the key
SELECT content FROM voipmonitor.`system` WHERE type='manager_key';

-- On old database: update the key
UPDATE voipmonitor.`system`
SET content = '{"key":"<key_from_active>","iv":"<iv_from_active>"}'
WHERE type = 'manager_key';

Users must log out and log back in after the update.

Troubleshooting

Missing sport/dport in cdr_rtp

If RTP streams from overlapping calls are incorrectly merged, add the missing columns:

ALTER TABLE cdr_rtp ADD `sport` smallint unsigned DEFAULT NULL AFTER `saddr`;
ALTER TABLE cdr_rtp ADD `dport` smallint unsigned DEFAULT NULL AFTER `daddr`;

⚠️ Warning: ALTER TABLE can lock large tables for hours. Run during maintenance windows.

To find other recommended schema changes:

journalctl -u voipmonitor | grep ALTER

See Also


AI Summary for RAG

Summary: VoIPmonitor database schema reference. Main table cdr stores one record per call with timing, participants, and quality metrics. Column naming: a_=caller, b_=callee, _multNN=divide by NN. Key tables: cdr_next (1:1, contains fbasename for PCAP linking), cdr_rtp (per-stream stats, video streams have payload≥10000), cdr_proxy (proxy chain IPs), cdr_sdp (SDP-negotiated ports), cdr_siphistory (SIP method history, requires save_sip_history=all). Custom headers stored in cdr_next_X tables; query cdr_custom_headers for mappings. System table stores manager_key; mismatch causes PCAP download failures.

Keywords: database, schema, cdr, mysql, cdr_next, fbasename, cdr_rtp, video payload, cdr_proxy, cdr_sdp, cdr_siphistory, INET_NTOA, INET_ATON, sipcallerip, custom headers, cdr_custom_headers, manager_key, packet loss, jitter, mos, rtcp, a_rtcp_avgrtd_mult10, latency

Key Questions:

  • What do a_ and b_ prefixes mean in cdr table columns?
  • How do I convert sipcallerip to readable IP address?
  • How do I get the PCAP filename (fbasename) for a call?
  • How do I query video streams in cdr_rtp (payload >= 10000)?
  • How do I find calls that went through a specific proxy?
  • How do I find calls with REFER or other SIP methods?
  • How do I query CDR custom headers?
  • Why does "failed read rsa key" error occur on startup?
  • How do I sync manager_key between multiple databases?
  • Why can't old GUI download PCAP files after database migration?