Manage SEM data storage

This topic addresses SEM database management.

About the three SEM data stores

By default, the SEM database is allowed 230 GB of the 250 GB allocated to the SEM virtual appliance. This partition consists of three data stores:

  • Syslog store
  • Events store
  • Original or raw log data store (optional)

The syslog store consists of all syslog or SNMP log data sent to the SEM VM. SEM reads and processes the data in real time, and then sends it to the event store for long-term storage. SEM stores the original data for 50 days in its original format (in case you need to review it). The data in the syslog store is compressed and rotated daily to maintain a consistent 50-days' worth of data. The amount of data stored here should level off at around the 50-day mark.

The event store (the second store) contains all normalized events generated by the SEM Manager and SEM Agents. Data in this store is compressed at ratios of 40:1 to 60:1, which equates to an average compression rate of 95–98 percent. Both nDepth and the SEM reports application query the event store for event data when they run.

The original log store (the third store) is an optional store for original or raw log messages that can be searched using Log Message queries in nDepth. The data in this store can come from SEM Agents or other devices logging to the SEM appliance. You can configure if data is sent to this store at the connector level, so not all devices have to store raw log messages in this manner.

For more information about storing original log messages, see Configure SEM to store original log messages (nDepth log retention).

Strategies for managing your SEM data storage needs

Depending on the needs of your environment, you can use one or more of the alternate storage methods listed below.

  • Back up the SEM VM on a regular basis. This will provide offline storage for your SEM data stores.
  • Decrease the number of days that syslog and SNMP data is stored in SEM.
  • Deploy another SEM VM to be used as a syslog server.
  • Deploy another SEM VM to be used as a database server.
  • Increase the space allocated to your SEM VM.

To get help with any of these methods, submit a ticket to Customer Support:

https://customerportal.solarwinds.com/support/submit-a-ticket

View SEM database usage numbers

There are three locations to find metrics that indicate how the SEM database is used:

  • Disk Usage summary in the CMC
  • Database maintenance report
  • Log storage maintenance report

View the Disk usage summary

When you use the command line to log in to SEM, SEM automatically generates a Disk Usage summary. You can also generate an ad hoc disk usage summary by running the diskusage command from the cmc >appliance prompt. The two lines to note here are Logs/Data and Logs.

  • The Logs/Data figure represents the total space being utilized by the SEM database. This value is presented in the percent % (usedG/allocatedG) format, where percent is the percent of the allocated space currently being used, and allocated is the total amount of space currently allocated to the SEM database.
  • The Logs figure represents the amount of space used by the syslog store. This figure is included in the used figure noted above. To figure out how much space is currently being used by the Event store, subtract the Logs value from the used value. If you are storing original log messages in the SEM database, the above calculation shows the combined space utilized by both your Event and original log stores.

View the Database maintenance report

Run the Database Maintenance Report in SEM reports to view a snapshot of your current database usage. The report includes the following values:

  • Disk Usage Summary – provides disk usage values in terms of the percentage of space allocated to the SEM database
  • Disk Usage Details – provides disk usage values in terms of physical file size
  • Database Time Span (days) – shows how many days' worth of live event data is currently stored in the SEM database
  • Other Files – represents the amount of space used by the syslog store

For more information, see the following KB article in the Customer Success Center:
Use the SEM Database Maintenance Report to See Retention and Volume of Traffic

View the Log storage maintenance report

Run the log storage maintenance report in SEM reports to get detailed information about the original log store. If you have not enabled SEM to store original log messages, this report will be blank.

For more information, see the following KB article in the Customer Success Center: Live Data Storage Retention in SEM.

Create a disk usage alert in SEM to warn you when a disk reaches a set limit

You can create a disk usage alert from the CMC command line to warn you when a disk partition reaches a preselected use limit. When the limit is reached, an InternalWarning event displays in the Monitor view.

You can define the disk use limit by the percentage of unavailable disk space (such as 75 percent), or by the amount of free disk space (such as 58G).

To create the disk usage alert:

  1. Open the CMC command line. See Log in to the SEM CMC command line interface for steps.

  2. To access the Appliance menu, at the cmc> prompt, enter appliance.

  3. To view the disk use of each partition, at the cmc::appliance> prompt, enter diskusage. For example:

    cmc::appliance > diskusage
    Checking Disk Usage (this could take a moment)
    ... ....00.00.00.00.00.00.00.
    Partition Disk Usage:
    SEM:             35% (991M/3.0G)
    OS:              45% (1.3G/3.0G)
    Logs/Data:        1% (901M/234G)
    Temp:             2% (252M/5.9G)
    
    Database Queue(s): 4.0K (No alerts queued, 0 alerts waiting in memory)
    Rules Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)
    Console Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)
    DataCenter Queue: 2.1M (0 alerts queued, unknown number of alerts waiting in memory)
    EPIC Rules Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)
    Forensic Database Queue: 2.1M (0 data queued, unknown number of data items waiting in memory)
    Logs: 1.3M
    Tool Profiles Message Queue: 2.1M (0 alerts queued, unknown number of alerts waiting in memory)					
    
    cmc::appliance > 
  4. At the cmc::appliance> prompt, enter diskusageconfig.

    Each partition and corresponding disk use limit displays on your screen. For example:

    cmc::appliance > diskusageconfig
    Current Disk Usage Configuration:
    # | Partition (filesystem) | Configured limit
    ===============================================
    1 |    SEM (/usr/local)    |        90%
    2 |    OS (/)              |        90%
    3 |    Logs/Data (/var/)   |        10G
    4 |    Temp (/tmp)         |        90%
    You can define your disk use limit by the percentage of unavailable disk
    space (such as 75%) or the amount of free disk space (such as 58G). 
    Enter the partition number you want to change (enter 'exit' and press 
    <Enter> to quit): 
  5. Enter the partition number you want to change, and then press Enter.
  6. Enter the disk usage limit value in percentage (such as 75 percent) or size (such as 58G), and then press Enter.

    For example, to change the OS disk partition limit in step 3 from 45 percent to 4 percent, enter 40 percent. To change the OS disk partition limit from 1.3 GB to 2.0 GB, enter 2GB.

    Disk usage limit [90%, sizeK, sizeM, sizeG, sizeT] (default 90%): 40%
    Limit '40%' for the 'OS' partition is set.
    Press <Enter> to set the next partition. Enter 'exit' and press <Enter>
    to quit. 
  7. Press Enter to set the next partition and repeat step 6 (if required).

    See Change the Logs/Data partition setting for additional information.

  8. When you are finished, type exit, and then press Enter to quit.

Change the Logs/Data partition setting

When you set the Logs/Data partition (3), a message prompts you to consider changing the database disk configuration using the dbdiskconfig command. SolarWinds recommends setting the Logs/Data partition and the database disk configuration to the same value.

Change the database disk configuration

  1. Finish configuring your partitions.
  2. At the cmc::appliance> prompt, enter dbdiskconfig.

    The following message displays:

    Current configuration:
    DoNotExceedPercentage = 90%
     
    The Manager will restart and apply your changes. To exit, enter 'exit' and press Enter.
    Enter a new value for DoNotExceedPercentage (default 90):
    Please enter an inter number 0-100 or 'exit'
  3. At the prompt, enter a usage limit value between 0 and 100, and then press Enter.

    If you enter a value less than 25, the partition will be deleted when this value is reached.

    The database disk configuration value is saved, and the appliance restarts the Manager Service.