Swap: Difference between revisions

From VoIPmonitor.org
(Add MEMORY IS FULL troubleshooting table with ringbuffer=2000 and max_buffer_mem=8000 recommendations)
(Add MySQL InnoDB buffer pool sizing recommendations and separate storage guidance)
Line 172: Line 172:
* '''If high memory usage/load persists''' after disabling saves: The bottleneck is '''CPU-related''. Focus on reducing processing load (disable audio transcoding, RTP analysis, etc.)
* '''If high memory usage/load persists''' after disabling saves: The bottleneck is '''CPU-related''. Focus on reducing processing load (disable audio transcoding, RTP analysis, etc.)
* '''If memory usage drops significantly and stabilizes''': The bottleneck was caused by packet storage operations. Restore saves after tuning buffer settings.
* '''If memory usage drops significantly and stabilizes''': The bottleneck was caused by packet storage operations. Restore saves after tuning buffer settings.
=== MySQL InnoDB Buffer Pool Sizing ===
For VoIPmonitor deployments, the MySQL/MariaDB InnoDB buffer pool is the most critical memory configuration parameter. This buffer stores database indexes and frequently accessed data in RAM, reducing disk I/O and improving query performance.
{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
|-
! colspan="2" style="background:#ffc107;" | Recommended InnoDB Buffer Pool Sizes
|-
! System RAM !! innodb_buffer_pool_size
|-
| 32GB || '''16GB'''
|-
| 64GB || '''32GB'''
|-
| 128GB || 80-96GB
|-
| 256GB || 180-200GB
|}
'''Key Guidelines:'''
* Allocate approximately 50-75% of total RAM to InnoDB buffer pool for dedicated database servers
* Leave sufficient RAM for the operating system, network buffers, and other MySQL configuration buffers
* For all-in-one deployments (sniffer + database on same host), reduce InnoDB buffer pool to 30-50% of RAM to leave memory for VoIPmonitor's packet capture buffers
* Always use fast storage (SSD/NVMe) for the MySQL data directory when allocating large buffer pools
To configure InnoDB buffer pool size, add or modify the following in <code>/etc/my.cnf</code> or <code>/etc/mysql/mariadb.conf.d/50-server.cnf</code>:
<syntaxhighlight lang="ini">
[mysqld]
innodb_buffer_pool_size = 16G  # Adjust based on system RAM above
</syntaxhighlight>
After changing this setting, restart MySQL:
<syntaxhighlight lang="bash">
systemctl restart mysql
# or
systemctl restart mariadb
</syntaxhighlight>
=== Separate Database and PCAP Storage ===
To maximize performance and prevent I/O contention, store MySQL database data and PCAP files on separate physical storage devices.
{| class="wikitable"
|-
! Data Type !! Recommended Storage !! Location
|-
| '''MySQL Data''' || SSD or NVMe (fast, low latency) || <code>/var/lib/mysql</code>
|-
| '''PCAP Files''' || SATA HDD or SSD (higher capacity) || <code>/var/spool/voipmonitor</code> (spooldir)
|}
'''Why Separate Storage Matters:'''
* MySQL requires high IOPS and low latency for random read/write operations on CDR tables
* PCAP storage involves large sequential writes for packet capture
* Separating them prevents disk I/O contention that causes database queue (SQLq) buildup and packet loss
* This is especially critical for high-traffic deployments (>500 concurrent calls)
Example configuration in <code>/etc/voipmonitor.conf</code>:
<syntaxhighlight lang="ini">
[general]
spooldir = /mnt/capacity_drive/voipmonitor
</syntaxhighlight>
Ensure the MySQL data directory remains on the fast SSD:
<syntaxhighlight lang="bash">
# Check MySQL data directory location
mysql -e "SHOW VARIABLES LIKE 'datadir';"
# Typical output: /var/lib/mysql
</syntaxhighlight>


=== Checking VoIPmonitor Memory Configuration ===
=== Checking VoIPmonitor Memory Configuration ===
Line 235: Line 312:


* [[Scaling|Scaling and Performance Tuning]]
* [[Scaling|Scaling and Performance Tuning]]
* [[Hardware|Hardware Sizing Examples]]
* [[Sniffer_troubleshooting#Check_for_OOM|OOM Troubleshooting]]
* [[Sniffer_troubleshooting#Check_for_OOM|OOM Troubleshooting]]
* [[Sniffer_configuration|Sniffer Configuration]]
* [[Sniffer_configuration|Sniffer Configuration]]
Line 240: Line 318:
== AI Summary for RAG ==
== AI Summary for RAG ==


'''Summary:''' Linux swap configuration for VoIPmonitor servers. Covers swappiness tuning (recommended value 5), disabling swap completely, understanding Linux cache/buffer memory as normal kernel behavior (reclaimable, not consumed), diagnostic steps to identify memory vs CPU bottleneck (temporarily disable savesip, savertp, savertcp, savegraph and monitor if high load persists), and conditional guidance for VoIPmonitor buffer memory settings (ringbuffer, max_buffer_mem): INCREASE max_buffer_mem (e.g., to 10000) when sufficient RAM available (>16GB) to handle high traffic bursts, DECREASE max_buffer_mem (to 500-1000) when RAM limited (<8GB) to prevent OOM crashes.
'''Summary:''' Linux swap configuration for VoIPmonitor servers. Covers swappiness tuning (recommended value 5), disabling swap completely, understanding Linux cache/buffer memory as normal kernel behavior (reclaimable, not consumed), diagnostic steps to identify memory vs CPU bottleneck (temporarily disable savesip, savertp, savertcp, savegraph and monitor if high load persists), conditional guidance for VoIPmonitor buffer memory settings (ringbuffer, max_buffer_mem): INCREASE max_buffer_mem (e.g., to 10000) when sufficient RAM available (>16GB) to handle high traffic bursts, DECREASE max_buffer_mem (to 500-1000) when RAM limited (<8GB) to prevent OOM crashes. CRITICAL: MySQL InnoDB buffer pool sizing recommendations: For 32GB RAM set innodb_buffer_pool_size to 16GB, for 64GB RAM set to 32GB, for 128GB RAM set to 80-96GB. Use SSD/NVMe for fast storage. Store MySQL data and PCAP files on separate physical storage devices to prevent I/O contention.


'''Keywords:''' swap, swappiness, memory, RAM, performance, ringbuffer, max_buffer_mem, OOM, out of memory, packetbuffer, cache, buffer, kernel memory, diagnostic, memory bottleneck, CPU bottleneck, savesip, savertp, savertcp, savegraph, disable packet saving
'''Keywords:''' swap, swappiness, memory, RAM, performance, ringbuffer, max_buffer_mem, OOM, out of memory, packetbuffer, cache, buffer, kernel memory, diagnostic, memory bottleneck, CPU bottleneck, savesip, savertp, savertcp, savegraph, disable packet saving, MySQL InnoDB buffer pool, innodb_buffer_pool_size, 32GB RAM, 64GB RAM, 128GB RAM, 16GB, 32GB, 80-96GB, SSD storage, NVMe, separate storage, I/O contention, spooldir, /var/spool/voipmonitor, /var/lib/mysql


'''Key Questions:'''
'''Key Questions:'''
Line 248: Line 326:
* How to disable swap on Linux?
* How to disable swap on Linux?
* What is swappiness and how to configure it?
* What is swappiness and how to configure it?
* How do I set the MySQL InnoDB buffer pool size for 32GB/64GB/128GB RAM?
* What is the recommended innodb_buffer_pool_size configuration for different system RAM sizes?
* Why should I store MySQL data and PCAP files on separate storage devices?
* How to reduce VoIPmonitor memory usage?
* How to reduce VoIPmonitor memory usage?
* What are ringbuffer and max_buffer_mem settings?
* What are ringbuffer and max_buffer_mem settings?

Revision as of 09:50, 6 January 2026

Category:Installation

Swap Configuration

Swap usage leads to performance degradation on VoIPmonitor servers. For realtime packet processing, we highly recommend configuring swap space properly or disabling it entirely.

Understanding Swappiness

Most Linux distributions default to using swap when RAM usage exceeds 40%. The swappiness value controls this behavior:

# Check current swappiness value
cat /proc/sys/vm/swappiness

A value of 60 means the system starts using swap when less than 60% of RAM is free. This is not optimal for realtime services like VoIPmonitor.

Swappiness Value Behavior
0 Use swap only when RAM is completely exhausted
5 Use swap only when critically low on RAM (recommended)
60 Default - start swapping at 40% free RAM (not recommended)
100 Aggressively use swap (not recommended for VoIPmonitor)

Configuring Swappiness

Temporary Change (Until Reboot)

# Set swappiness to 5 (use swap only when critically low on RAM)
echo '5' > /proc/sys/vm/swappiness

# Apply immediately by clearing swap
swapoff -a && swapon -a

Permanent Change

Add the following line to /etc/sysctl.conf:

vm.swappiness=5

Then apply without reboot:

sysctl -p

Clearing Swap

To move all data from swap back to RAM:

swapoff -a
swapon -a

Disabling Swap Completely

Step 1
Disable swap immediately:
swapoff -a
Step 2
Comment out swap entries in /etc/fstab:
proc            /proc           proc    defaults        0       0
UUID=17e56a9a-0f42-44ca-90e8-570315708def /               xfs     relatime        0       1

# Swap entries commented out:
#UUID=0a403f5e-0505-4055-a219-70217b6b74d1 none            swap    sw              0       0
#UUID=4cf42564-4339-42ed-b7ec-29644a3085f1 none            swap    sw              0       0
Step 3
Monitor for OOM issues:
# Check for out-of-memory events
dmesg -T | grep -i "out of memory\|killed process"

Understanding Linux Cache and Buffer Memory

It is important to understand that high system cache and buffer usage is normal kernel behavior and should not be confused with actual memory consumption. The Linux kernel uses available RAM to cache disk I/O and buffer network packets, but this memory is considered free and can be immediately reclaimed when needed by applications.

# Check memory with cache/buffers separated
free -h

# Example output interpretation:
# total        used        free      shared  buff/cache   available
# Mem:           31Gi       10Gi       2.0Gi       1.0Gi        19Gi        18Gi
#
# The "available" column (18Gi) is the usable memory for applications.
# The 19Gi in "buff/cache" is kernel-managed but reclaimable.

When diagnosing memory issues on VoIPmonitor servers, focus on the available column rather than total memory usage. High cache/buffer values indicate the kernel is efficiently using spare RAM, not that the system is running out of memory.

However, if the sniffer's internal 'working memory buffer/cache is filling up and causing crashes, you may need to adjust max_buffer_mem depending on your server's RAM capacity:

  • Insufficient RAM: Decrease max_buffer_mem to prevent OOM crashes
  • Sufficient RAM Available: Increase max_buffer_mem (e.g., to 10000) to handle high traffic bursts

Diagnostic Step: Identify Memory vs CPU Bottleneck

Before adjusting buffer settings, isolate whether the bottleneck is memory-related or CPU-related. Follow this diagnostic process:

Step 1
Temporarily disable all packet saving:
# In /etc/voipmonitor.conf, temporarily set all saves to no:
savesip = no
savertp = no
savertcp = no
savegraph = no
Step 2
Restart the sniffer:
systemctl restart voipmonitor
Step 3
Monitor the system:
# Watch memory usage
watch -n 1 free -h

# Check if the "buffer filling up" condition persists
journalctl -u voipmonitor -f
Step 4
Analyze the result:
  • If high memory usage/load persists' after disabling saves: The bottleneck is CPU-related. Focus on reducing processing load (disable audio transcoding, RTP analysis, etc.)
  • If memory usage drops significantly and stabilizes: The bottleneck was caused by packet storage operations. Restore saves after tuning buffer settings.

MySQL InnoDB Buffer Pool Sizing

For VoIPmonitor deployments, the MySQL/MariaDB InnoDB buffer pool is the most critical memory configuration parameter. This buffer stores database indexes and frequently accessed data in RAM, reducing disk I/O and improving query performance.

Recommended InnoDB Buffer Pool Sizes
System RAM innodb_buffer_pool_size
32GB 16GB
64GB 32GB
128GB 80-96GB
256GB 180-200GB

Key Guidelines:

  • Allocate approximately 50-75% of total RAM to InnoDB buffer pool for dedicated database servers
  • Leave sufficient RAM for the operating system, network buffers, and other MySQL configuration buffers
  • For all-in-one deployments (sniffer + database on same host), reduce InnoDB buffer pool to 30-50% of RAM to leave memory for VoIPmonitor's packet capture buffers
  • Always use fast storage (SSD/NVMe) for the MySQL data directory when allocating large buffer pools

To configure InnoDB buffer pool size, add or modify the following in /etc/my.cnf or /etc/mysql/mariadb.conf.d/50-server.cnf:

[mysqld]
innodb_buffer_pool_size = 16G  # Adjust based on system RAM above

After changing this setting, restart MySQL:

systemctl restart mysql
# or
systemctl restart mariadb

Separate Database and PCAP Storage

To maximize performance and prevent I/O contention, store MySQL database data and PCAP files on separate physical storage devices.

Data Type Recommended Storage Location
MySQL Data SSD or NVMe (fast, low latency) /var/lib/mysql
PCAP Files SATA HDD or SSD (higher capacity) /var/spool/voipmonitor (spooldir)

Why Separate Storage Matters:

  • MySQL requires high IOPS and low latency for random read/write operations on CDR tables
  • PCAP storage involves large sequential writes for packet capture
  • Separating them prevents disk I/O contention that causes database queue (SQLq) buildup and packet loss
  • This is especially critical for high-traffic deployments (>500 concurrent calls)

Example configuration in /etc/voipmonitor.conf:

[general]
spooldir = /mnt/capacity_drive/voipmonitor

Ensure the MySQL data directory remains on the fast SSD:

# Check MySQL data directory location
mysql -e "SHOW VARIABLES LIKE 'datadir';"

# Typical output: /var/lib/mysql

Checking VoIPmonitor Memory Configuration

After completing the diagnostic above, review and adjust VoIPmonitor's internal memory configuration based on your findings.

# Check current memory-related settings
grep -E '^(max_buffer_mem|ringbuffer|packetbuffer)' /etc/voipmonitor.conf
Parameter Default Description When to Increase When to Decrease
ringbuffer 50 MB Ringbuffer size per interface. Recommended >= 500 for >100 Mbit traffic. Max 2000. High traffic bursts (>200Mbps), "MEMORY IS FULL" errors, packet loss observed RAM limited, packet loss not observed
max_buffer_mem 2000 MB Maximum buffer memory for packet buffers. Sufficient RAM available (>16GB), buffer filling up on high traffic, "MEMORY IS FULL" errors RAM limited (<8GB), OOM crashes occurring
packetbuffer_enable yes Enable packet buffer cache. Generally keep enabled for performance Disable if causing excessive memory pressure
packetbuffer_compress no Compress packet buffer (saves RAM, uses CPU). RAM constrained, CPU available CPU constrained, RAM available

Buffer Tuning Guidelines:

When Experiencing "packetbuffer: MEMORY IS FULL" Errors
Scenario ringbuffer max_buffer_mem
Very high traffic (>200Mbps), severe packet loss 2000 (maximum) 8000
High traffic (100-200Mbps), moderate packet loss 500-1000 4000-6000
Moderate traffic (<100Mbps), occasional issues 500 2000-4000
  • Increasing Buffers (when RAM is available):
    • For "MEMORY IS FULL" errors on high traffic servers: ringbuffer = 2000 and max_buffer_mem = 8000 (requires 32GB+ RAM)
    • For traffic spikes on servers with 32GB+ RAM: max_buffer_mem = 10000 or higher
  • Decreasing Buffers (when RAM is limited):** Reduce ringbuffer to 20-50 and max_buffer_mem to 500-1000 on servers with 8GB or less RAM to prevent OOM crashes.

See Emergency_procedures#Emergency:_Service_Restart_Loop_with_"packetbuffer:_MEMORY_IS_FULL" Emergency Procedures for detailed troubleshooting of "MEMORY IS FULL" errors.

See Sniffer Configuration for complete memory and buffer documentation.

Checking Current Memory and Swap Usage

# Check memory and swap status
free -m

# Monitor memory usage over time
watch -n 1 free -m

# Check which processes use most memory
ps aux --sort=-%mem | head -10

See Also

AI Summary for RAG

Summary: Linux swap configuration for VoIPmonitor servers. Covers swappiness tuning (recommended value 5), disabling swap completely, understanding Linux cache/buffer memory as normal kernel behavior (reclaimable, not consumed), diagnostic steps to identify memory vs CPU bottleneck (temporarily disable savesip, savertp, savertcp, savegraph and monitor if high load persists), conditional guidance for VoIPmonitor buffer memory settings (ringbuffer, max_buffer_mem): INCREASE max_buffer_mem (e.g., to 10000) when sufficient RAM available (>16GB) to handle high traffic bursts, DECREASE max_buffer_mem (to 500-1000) when RAM limited (<8GB) to prevent OOM crashes. CRITICAL: MySQL InnoDB buffer pool sizing recommendations: For 32GB RAM set innodb_buffer_pool_size to 16GB, for 64GB RAM set to 32GB, for 128GB RAM set to 80-96GB. Use SSD/NVMe for fast storage. Store MySQL data and PCAP files on separate physical storage devices to prevent I/O contention.

Keywords: swap, swappiness, memory, RAM, performance, ringbuffer, max_buffer_mem, OOM, out of memory, packetbuffer, cache, buffer, kernel memory, diagnostic, memory bottleneck, CPU bottleneck, savesip, savertp, savertcp, savegraph, disable packet saving, MySQL InnoDB buffer pool, innodb_buffer_pool_size, 32GB RAM, 64GB RAM, 128GB RAM, 16GB, 32GB, 80-96GB, SSD storage, NVMe, separate storage, I/O contention, spooldir, /var/spool/voipmonitor, /var/lib/mysql

Key Questions:

  • How to configure swap for VoIPmonitor?
  • How to disable swap on Linux?
  • What is swappiness and how to configure it?
  • How do I set the MySQL InnoDB buffer pool size for 32GB/64GB/128GB RAM?
  • What is the recommended innodb_buffer_pool_size configuration for different system RAM sizes?
  • Why should I store MySQL data and PCAP files on separate storage devices?
  • How to reduce VoIPmonitor memory usage?
  • What are ringbuffer and max_buffer_mem settings?
  • Should I increase or decrease max_buffer_mem for my server?
  • How do I know if memory issues are caused by VoIPmonitor buffers or CPU bottlenecks?
  • How do I diagnose memory vs CPU bottleneck in VoIPmonitor?
  • What does high cache/buffer usage mean in Linux?
  • Why does free command show high memory usage but system is fine?