Configuration – SGBox Next Generation SIEM & SOAR

Regex Pattern

SGBox — Wed, 14 Feb 2024 16:12:27 +0000

1 Definition

A regex is a string of text that lets you create patterns that help match, locate, and manage text.
Regex can be a powerful and very fast method to extract parameters inside a log line to generate special objects, named on SGBox “Events”, to full evaluate a log line message.

2 Regex generation

Concepts

Matching and Capture

When build a regex, two main actions are mainly possible:

✔ Match: you can use word/number o special combination to include to match the line and go on evaluating it
ex. This is my IP. Regex: This .*? my IP.
✂ Capture: in addition to simple match a word or another part of the string, you can also extract some information, by wrapping it by round parenthesis “()”. In this case the engine will separate information inside the round parenthesis from other text. These pieces of text can be turned in “Parameters” inside the “Events” SGBox objects.

Regex engine is very flexible and permit more operations, but only these action can be sufficient to generate a valid SGBox pattern.

Best Practice

Some rules must be keep in mind to improve efficency and decrease possible abnormal behaviors

The text must be as specific as possible in the first part. This because when regex engine start to parse the log line, it stop at first non-match from the beginning.
The match must be less greedier possible.
Eg. not .* but instead .*?

Recommended patterns

Pattern	Match
`.*?`	matching any character
`\s+`	Match any spaces
`(\d+)`	Generic multiple numbers match and capture (Port, numerical id, numerical session id, numerical severity)
`(\d{1,5})`	Match and capture port number (1-65535)
`(\w+)`	Match and capture any single word
`([\d]{1,3}\.[\d]{1,3}\.[\d]{1,3}\.[\d]{1,3})`	Match and capture IP Address (not hostname)
`([\d\.]+)`	Match and capture IP Address (not hostname)
`([0-9a-fA-F:-]+)`	Match and capture any type of MacAddress
`([0-9a-f:]+)`	Match and capture lowercase, `:` separated MacAddress
`([0-9A-F:]+)`	Match and capture uppercase, `:` separated MacAddress
`([0-9a-f-]+)`	Match and capture lowercase, `-` separated MacAddress
`([0-9A-F-]+)`	Match and capture uppercase, `-` separated MacAddress
`(.?@.?\..*?)`	Match and capture generic mail address
`(?:alice\|bob)`	Non-capturing OR match (at least one word must match)
`(?:myparam)?`	Non-capturing optional match (match 0 or 1 time)
`((\w+) .*?)`	Nested match and capture group. Capture first word and then the entire parameter

References

Global Summary Cheat Sheet

Character	What does it do?	Example	Matches
`^`	Matches beginning of line	`^abc`	abc, abcdef.., abc123
`$`	Matches end of line	`abc$`	my:abc, 123abc, theabc
`.`	Match any characters	`a.c`	abc, asg, a2c
`\|`	OR operator	`abc\|xyz`	abc or xyz
`(...)`	Capture anything matched	`(a)b(c)`	Captures ‘a’ and ‘c’
`(?:...)`	Non-capturing group	`(a)b(?:c)`	Captures ‘a’ but only groups ‘c’
`[...]`	Matches anything contained in brackets	`[abc]`	a,b, or c
`[^...]`	Matches anything not contained in brackets	`[^abc]`	xyz, 123, 1de
`[a-z]`	Matches any characters between ‘a’ and ‘z’	`[b-z]`	bc, mind, xyz
`{x}`	The exact ‘x’ amount of times to match	`(abc){2}`	abcabc
`{x,}`	Match ‘x’ amount of times or more	`(abc){2,}`	abcabc, abcabcabc
`{x,y}`	Match between ‘x’ and ‘y’ times.	`(a){2,4}`	aa, aaa, aaaaa
`*`	Greedy match that matches everything in place of the `*`	`ab*c`	abc, abbcc, abcdc
`+`	Matches character before + one or more times	`a+c`	ac, aac, aaac,
`?`	Matches the character before the ? zero or one times. Also, used as a non-greedy match	`ab?c`	ac, abc
`\`	Escape the character after the backslash or create an escape sequence.	`a\sc`	a c

Tools

Some tool can help you to create the right combination of regex

regex101: build, test, and debug regex – An online regex editor. It can
CyberChef (gchq.github.io) – And extended tool to manipulate string and text (helpful to analyse the logline before creating a regex)

3 SGBox Pattern Creation and Add (Advanced)

Object definition

Parameter: a single extracted value that also permit correlation between different pattern/classes.
Pattern / Event Name: Name that identifies a specific event extracted from log.
Pattern / Event: An event/action/information extracted from 1 line of logs. In standard usage for every event correspond only 1 line of log.
Regex definition: the regex syntax that extract information from the log line that match.
Class: a container to group different Events.

Concepts

✍ Parameter name: if possible, always assign a parameter name that is already present in the dropdown menu. For performance reason, avoid to create unnecessary parameter.
💭 Capture always only the relevant information: try to convert log part as parameter only the information that you really need

Make a new pattern

To generate a new pattern you must go on LM -> Configuration -> Pattern and click on “New Pattern” button.

Now in the first part you can search the log you need to parse, by filter out the unnecessary logs and test your regex. In the right pane you can preview the captured group values that will be later transformed in parameters.

Here you must:

Select the Hosts to retreive the logs.
Select a compatible timerange to find the logs you need (try to reduce the timerange if the regex is correct but you cannot find anything, the search is limited to 100.000 lines due to performance limit).
Enter the search or final regex
Press “Search” to match the regex end extract the results
In the right pane you can see the captured group match on the regex

Once you are sure that the correct logs is extracted, you can press the “Create” button and proceed to the Creation Window

Here you must:

Check, fix or complete the regex
Press the “Test” button to start the extract search for the example extracted log in the first box
Once Parameters appear you must associate the Value column with a Parameter Name in the second column. Avoid to create new Parameter name unless absolutely essential (due to performance reason)
Fullfill the Pattern Name and Description to be easily searchable in the pattern view
Select “Create” to finalize the pattern creation

SGBox Agent

SGBox — Thu, 22 Jun 2023 14:31:33 +0000

Installation and Configuration

Requirements

To execeute correctly the Agent, the following software is required:

.NET Framework 4.0
Internet Explorer dll framework
Outgoing open port 443

Antivirus Consideration

We reported that some antivirus can interfere with the normal operation of the Agent (We have especially reported many cases with Sophos). Please be sure to insert an exception

Sophos

Preliminary Information

Agent Buffer: in case of the sgbox is offline, the agent will act as Buffer to store the logs until connection with the appliance is restored. The buffer store depend on the free disk space remaining.
Port used: the port used to communicate is the 443.
Communication type: SGBox Agent will communicate trough the Internet Explorer DCOM API.

Download

To install the agent you must to download the agent from the dedicated download section on SGBox Portal

Note, to download SGAgent, it is required to login or sign up on our portal and go to the Download SGBox Software section.

Installation Configuration

Extract the downloaded archive and run the setup

Click on “Next/Avanti” to continue with the installation

Browse the folder where you want install the agent

Edit the field “Server ip” with the IP or FQDN of your SGBox

You will be asked to confirm the data entered, click “Next/Avanti” to proceed with the installation.

Click on “Yes” to start with the installation

Click on “Close/Chiudi” to finish the installation

If the installation is correctly terminated a new service named “SGBoxTask Service” will be created

Log Retreive Configurations

Capture Logs from Standard Windows Event View

This section explain how to create a new configuration and command. A new command could be added in a same way to an existing configuration.

Click on CLICK HERE TO CREATE NEW CONFIGURATION if you want create a new configuration or click on existing configuration if you want to edit it.

Enter o modify a name for the configuration and select GetEventLog to retrieve new information from Event Viewer.

Enter details of your command:

Name: a descriptive name of your command.
Description: brief description of your command ( not mandatory).
Frequency: how frequent these information will be sent to SGBox.
Log Name: the registry name. If it not present look the this section
Select o specify the Event ID. You can Select All events or -1 to tell the agent to send all events from the specified register.

You can add more commands to your configuration.

Drag & Drop your configuration to the target host and Save Changes.

Capture Logs from Operational (Application) Windows Event View

This section explain how to create a new configuration and command from a custom registry log. We’ll take the Terminal Service Registry as example. Here the details of the logs we want retrieve:

Se the previous section to specify a new command from a basic registry:
https://www.sgbox.it/sgbox/EN/knowledge-base/create-a-new-command/

Fist of all we need to find the exact name of the registry: Right click > Properties

A new command could be added in a same way to an existing configuration.

Click on CLICK HERE TO CREATE NEW CONFIGURATION if you want create a new configuration or click on existing configuration if you want to edit it.

Enter o modify a name for the configuration and select GetEventLog to retrieve new information from Event Viewer.

Enter details of your command:

Name: a descriptive name of your command.
Description: brief description of your command ( not mandatory).
Frequency: how frequent these information will be sent toSGBox.
Log Name: select ADD NEW
New Log Name: the registry name taken before.
Select o specify the Event ID. You can Select All events or -1 to tell the agent to send all events from the specified register.

You can add more commands to your configuration.

Drag & Drop your configuration to the target host and Save Changes.

Capture Logs from File/Folders (TailFolder method)

This section explain how to create a new configuration and the related command in order to retrieve logs from a specific folder.

Requirements

SGBox 5.0.2 or SGBox 4.2.7 is required.
At least SGAgent 3.2.7433.19116 is required .

Click on CLICK HERE TO CREATE NEW CONFIGURATION if you want create a new configuration or click on existing configuration if you want to edit it.
Enter a name and select TailFolder as command.

A new windows appears. Enter details of your command:

Name: a descriptive name of your command.
Description: brief description of your command ( not mandatory).
Frequency: how frequent these information will be sent to SGBox.
Directory Path: where how logs are located
File Name: Logs file name, also star expression could be used.
List Subdirectories: Use this flag if you want to look also logs located in the subdirectories.
Timestamp Pattern: a regex to find the correct timestamp of the logs.
Timestamp Format: Specify logs timestamp format.
Timezone: You can specify if the Timestamp is in Localtime or UTC.

ATTENTION: if the folder you are trying to monitor is inside C:/Windows/System32/ you need to use C:/Windows/sysnative/

Your command has been created. If you want you can add more commands to your configuration.

Click on Save Changes to save your configuration.

Drag & Drop your configuration to the target host and Save Changes.

When everything is configured you can see your logs in historical search

Configure File Integrity Monitoring

File Integrity Monitoring is new feature introduced with the last SGAgent version and it’s used to monitor files and shared folders. Using this feature you can monitor when a specific file is read, modified or deleted.

!Attention: File Integrity Monitoring is not File Auditing, you are not able to see the user that execute the action.

Requirements

SGBox 5.1.3 or higher.
SGAgent 3.4 or higher.

The FIM package can be installed from SCM>Applications>Packages: Click to install to download and install the package, then click on Run and select the hosts you want to monitor.

Go on LM>Configurations>Agents

In our example we create a specific configuration for this feature, but you can also create a new command on a existing configuration and modify it.
Click on “New Configuration” to create a new configuration and select CheckFolder.

A new window will appear to enter the command’s details:

- Name: a descriptive name of your command.
- Description: a short description of your command (not mandatory).
- Frequency: how often this information will be sent to SGBox (60 sec suggested).
- Directory Path: where the files or folders are located.
- File Name: Name of the file (you can also use the star expression).
- Check Subdirectories: Use this flag if you want to look at files located in sub directories as well.
- File Integrity: Select the monitor mode* you want to use
- Exclude files: you can specify some files to exclude for the monitor (not mandatory, regex supported)

Monitor Mode

Monitor Only: check the integrity when the PC and agent are running.
Monitor and store integrity: Store the integrity in a internal DB. Even if some operations on files are performed when the S.O or Agent are not running, the agent can identify them. Store large directories can seriously impact performance.

Click OK to save the command.
Click “Save Changes” to save your configuration.

Drag and drop your configuration to target host and click again on “Save Changes“.

When everything is set up you can see your logs in the historical search or from the “File Integrity Monitoring” dashboards.

FIM is very useful if you want to store critical configurations or backups. It’s no suggested monitor all the C: storage. Here some interested folder to monitor:

C:\inetpub\wwwroot C:\Windows\Boot C:\Windows\System32\drivers\etc

Strict TLS connection with a Personal Certificate

Starting from version 3.7. it is possible configure the SGAgent to check the SGBox/Collector certificate before sending information.

Requirements:

SGAgent version 3.7
SGBox must have a valid certificate. Look this section.

After installed go in the installation directory. Default path is C:Files(x86)Agent** Open the file SGBoxTask.exe.config** as Administrator with a text editor like Notepad.

add the following entry after the connection strings: key=”IgnoreCertificate” value=”False”

Save the configuration and restart the SGBoxTask Service service.

Check the file SGBoxTaskLog.txt to verify that everything is ok.
Here an example of error:

220330 14.51.05 0000008 *** Error The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. System at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context) at System.Net.HttpWebRequest.GetRequestStream() at SGBoxTask.Utils.Internet.GenerateCommandRequest(String uri, String ApplicationId, String login, String password)

Here an example when it works:

220330 14.54.20 0000004 Starting ServiceSGBoxTask 220330 14.54.20 0000006 Starting Main 220330 14.54.20 0000006 Params 0A002700000D https://sgbox192.sgbox.it/sgbox/LM/dataxchange/cmd.php https://sgbox192.sgbox.it/sgbox/LM/dataxchange/send.php 220330 14.54.20 0000006 SleepTime 10 msec 220330 14.54.20 0000006 RandomStartTimer 2 sec RandomMinStartTimer 1 220330 14.54.20 0000006 Enable TLS 1, 1.1, 1.2 220330 14.54.20 0000006 Starting StartSendPacket 220330 14.54.20 0000006 Starting StartGetCommand

Uninstall

Prerequisites

Before Uninstall the Agent be sure that (for all users connected to the server):

All mmc.exe instance are closed
All services panel (services.msc) are closed
The Task Manager (and Process Explorer) is temporary closed
All Event Viewer instance are closed

Automatic procedure (Recommended)

To uninstall the Agent, you must go to “Add/Remove Programs”, then select the “SGBox Agent” and select “Uninstall”.

Reinstall Note: It is recommended, in case of agent reinstall, to full restart the machine before proceeding with the new install.

Manual Full Remove

To Full remove the Agent if anything goes wrong, you must check and remove these items:

Service: Stop and remove the service, you can use this Powershell command:
get-service SGBoxTask | stop-service then on a cmd window sc delete SGBoxTask
Registry: Find and delete this Regsitry Key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SGBoxTask
Folders: Full Remove this Folder: C:\Program Files (x86)\SGBox Agent

Update Agent

To update the Agent you must completely uninstall the old version (we recommend also to reboot the machine if possible), then install the new version with the specific installer.

Silent/Unattended Mode

Install

In order to distribuite SGAgent in silent mode you have to type the follwing command:

SetupSGBox.msi /q ServerIP="192.168.xxx.xxx"

Uninstall

In order to uninstall SGAgent in silent mode you have to type following command:

msiexec /q /x {C09891C0-0E34-4873-A869-F9DC136E67C2}

Troubleshooting

The Agent is composed by:

A service: named SGBoxTask, must be set on automatic start and running
Default Installation folder: C:\Program Files (x86)\SGBox Agent – Main files and folders
SGBoxTask.exe: is the main executable file and service
SGBoxTaskLog.txt: the main log file of agent itself
SGBoxAgent.exe.config: configuration file for the agent
C:\programdata\SGBoxTask\Packet: the folder where packet ready to be send, or cached are stored

How to analyze Agent log

The main log file is: SGBoxTaskLog.txt If you have any sort of problem related to the agent, you can send this file to assistance to check the stream.

Some useful rows to check the correct comunication are:

Row with the command: GetCommand, the agent is checking the command to execute coming from LM -> Configuration -> Agents
Detected OLD Reqest xxx: SGAgent has identified a cached command that is not used and has been marked as inactive. It’s informational
Read Json … : the Json command received from SGBox – Sending File … : a final packet has been sended to SGBox

Check Service

To check the service is running you can from a CMD execute this command:

sc query SGBoxTask

If status equal to Running the service si correctly running, otherwise must be started or check the whole configuration.

Reconfigure IP on change appliance IP

Attention: this procedure is valid only on change IP and not when you are migrating to new appliance istance or a new major version

To change the query IP point for the agent go to the configuration file SGBoxTask.exe.config in the default folder and change these rows:

SGBoxTask.exe.config Definition

: Time interval in seconds between the request of new command configuration coming from SGBox
: Max size of the log file SGBoxTaskLog.txt
: SGagent log verbosity level for SGBoxTaskLog.txt
: Complete Url interrogation for command list coming from SGBox
: Complete Url where log is sended to SGBox
: Max size in bytes of the file sended to SGBox each time
: Milliseconds of delay before send the file to SGBox
: Max size of the whole log waiting to be sended (or cached)

Network Connectivity Checklist

If the agent is unable to communicate with the Appliance/Collector, please check these actions to be sure that the communication over network is correct:

Check that machine firewall does not block the requests
Check that Antivirus installed does not block the requests
Check that network device between machine and main gateway does not block or drop requests
Check that no GPO configuration can collide with the agent requests
Check that the machine is enabled with communication with at least SSL 1.3

log decryption test

SGBox — Wed, 24 May 2023 13:41:54 +0000

Log decryption test

This article explain where encrypted logs are stored in SGBox and how to perform some decryption test.

First of all you need to know that after SGBox receives the logs it store the them in the Online Database in order to allows to do some searchs with Historical Search tool (LM > Analysis > Historical Seach).
Meanwhile SGBox also analyze the logs in order to produce the events you can see in Class/Pattern Analysis, Templates, Dashboards, Reports, ecc..
Here you can find some more information on how logs are stored and their retention: data retention

The raw logs are also stored on the filesystem in encrypted format using GPG. You can see them in LM > Configuration > Encryption
In this page you are also able to download a specific log file and check it can’t be read without the SGBox GPG keys

In order to read it you need to download the GPG keys and store the in a file (read this article to know how to do it: Export GPG key)

WINDOWS

Download & Install a GPG program like GPG4WIN (https://www.gpg4win.org/).Run the progrma and choose Import botton. Select the previuosly exported GPG keys file.

Choose Decrypt/Verify and select your file.

Click on Save All to save the unencrypted file

LINUX

Import your keys:
gpg --import < sgbox_pub.key
gpg --import < sgbox_priv.key

Run following command:
gpg -d -q data_20200202050000_20200202055959_757.log.gpg

Directory Import

SGBox — Tue, 26 Jul 2022 14:40:43 +0000

Directory Import

This feature is used to upload file to SGBox using SFTP or SMB protocol.

Requirements:

SGBox version 5.1.0

First of all check your connection with log user. Credential has been set during the first wizard.
From a terminal like software (PuTTY). Move from the local folder to the “import” directory and create the application folder. Then assign the right privileges.

From LM > Configuration > Directory Import select “New Import“. The folder you created must be visible in the list.

You can upload files on SGBox using SFTP o SMB protocols.
!Note: The SMB protocol is not active by default. You need to active it using the CLI command.

In our example we upload a sample file in SFTP.

From the web interface you can see the uploaded file.

We suggest to set up:

the IP and Name of the machine. After imported the logs will be associated to the specified host.
Put Active to ON in order to start the import.
Select Generic filter from the filter list. This is used to identify the timestamp in your logs.

If everything worked fine you will see the logs appear in the next window.

SGBox will check for new logs every minutes.

You can see the logs in LM > Analysis > Historical Search.

You can also download from here the sample file:
[button color=”accent-color” hover_text_color_override=”#fff” size=”small” url=”http://www.sgbox.it/download/7662/” text=”Download sample file” color_override=””]

Parameter translation

SGBox — Mon, 07 Jun 2021 10:57:55 +0000

Parameter translation in a SGBox pattern

This article explains how to configure the Translate parameter feature in SGBox.
When events are submitted, it is possible to display some parameters through their ‘aliases’. In this section you can specify the parameters and the corresponding aliases in a table and then associate it with a parameter defined in the event (pattern).
For example, you can convert the logon type parameter of the Windows EventID 4624, connection to the Windows server.

10	Remote Desktop
2	Interactive

0xc0000234	User logon with account locked
0xc000006e	Unknown user name or bad password

It is also possible to upload files containing parameter > alias associations.
Note: the files must be text files containing for each line two strings separated by where the first string represents the parameter read by the events and the second the alias that will be displayed. For example

eth0Internal network
eth1WAN

The menu item for parameter translations can be found under SGBox>LM>Configuration>Pattern>Translate Parameters. To translate the parameters you will have to:
Open a new file by typing in the field “values” LogonType, “translate” the meaning of the code, after that save the file.

Once you have translated the values you will have to go to the modification of the pattern [SGAgent] (4624) Logon OK.

Under the item translate parameters click on the item “choose from the list” assigning the file, save the modification of the pattern.

Once you have done the above, the parameter “Logon type” with subsequent logs from the Classes/Patterns Analysis will no longer be displayed as logon type “10” but translated as “Remote Desktop”.

Automatic Vendor Recognition

SGBox — Wed, 25 Nov 2020 16:37:32 +0000

Profiles and Vendors (logs auto recognition)

With version 5.1.0 a new concept has been introduced: logs auto recognition and categorization.
SGBox already recognizes many different log sources, and up to v5.0.7 user needed to associate the collected logs to the desired patterns to extract events.

Pre-defined classes are now associated to known vendors, and by selecting one or more profiles user defines which events should be extracted and should be available for reference.Profile selection automatically associates a recognized host to a set of pre defined classes. This is the simplest way to configure SGBox.
As always, the user can customize these configurations to tailor the SGBox layout and information according to his needs.

This operating mode must first be enabled: in the SCM>Advanced options menu, select “Automatic mode” to activate the automatic log recognition.

SGBox starts to recognise and categorise logs and you’ll be able to gradually see new assets in the SCM>Network>Assets menu.
At last you should apply the profile you prefer, by selecting the LM>Configuration>Profiles menu and check the desired profile(s). Selections are immediately activated (or de-activated) at your click.

Now SGBox will automatically recognize the known vendor’s logs and will gradually extract only the information corresponding to the selected profiles. This process can take some time, depending on the logs frequency.

If you decide to stop extracting events for a specific profile, simply uncheck it. Remember that by unchecking a profile, the corresponding events won’t be deleted. They will be saved as historical data and hidden in the interface. If you re-enable a profile, the association will be re-established and you will be able to view both historical and new data.

In any case, all the raw data is always collected, regardless the configuration you select. You can always make new configurations and re-apply them to your historical raw data to obtain a new set of events. This feature has nothing to do with data collection and you will not break in any way the raw data collection.

Please note: SGBox already has a similar mechanism used to auto configure the remote agents on Windows hosts, so that they can collect a standard set of events (SCM > Advanced options > Initial configuration). These two configuration can live together, but if you first select a profile that involves data coming from Windows agents and then de-select it, SGBox will stop extract those events. As usual you can re-enable the profile or manually configure your hosts and again, no raw data will be lost.

Multi-tenant mode. In the Multi-tenant version of SGBox, administrator can centrally assign specific profiles to the associated tenants. In SCM > Multitenant > Manager > Profiles a matrix will be presented to the administrator. Using that matrix it will be possible to assign one or more profiles to each tenant, in a single place.

Automatic log recognition is an high performance operating mode that has minimal impact to SGBox performances. Anyway, when your setup looks complete, you may consider to disable the log automatic recognition (SCM>Advanced options, uncheck “Automatic mode”). This will have no impact on the configurations, it will simply disable the automatic recognition.